Learning from Milk Bar about Spectacles, a new community tool for Looker integration and DevOps buffs

Profile picture for user jreed By Jon Reed February 6, 2020
Why should we geek out over DevOps? How about serving business users better? How about automated data testing? That was the goal when two engineers launched Spectacles, an open source Looker tool.

milkbar NYC
Milk Bar in midtown Manhattan

Ever since Looker JOIN 2020 in November, I've had Milk Bar on my mind. That's a cream-frosted dilemma for me, as Milk Bar's decadently-crafted dessert menu is not exactly on my diet.

But after I spoke with the Milk Bar team about their Looker project at JOIN, I've been waging an ongoing battle with culinary temptation.

First, I stumbled onto a Milk Bar at the Cosmopolitan in Vegas. Then, at the NRF retail show, my hotel was next to an always-crowded Milk Bar, tempting me with sticky-sweet aromas (pictured right).

Fortunately, at Looker JOIN, Milk Bar analyst and data engineer Josh Temple was kind enough not to bring any food samples to torment tempt me with (Temple is now an analytics engineer at Spotify). At the time of our interview, Temple and Looker consultant Dylan Baker were collaborating on a community project, code-named Spectacles.

Fast forward to February, and Spectacles is entering a new phase of public consumption. Baker, who bills himself as "analytics engineering consultant," passed this update along to me:

We’re really excited to be joining the Looker partner network, and to be launching the first hosted version of Spectacles later in Q1. We think it will be a really great addition to the tech stack of Looker users. If anyone would be interested in taking part in the beta launch, they should sign up on our website, spectacles.dev.

Spectacles - how a community DevOps tool was born

So what is Spectacles, you ask? Well, officially, it's lowercase, as in spectacles, and it's intended as a "continuous integration tool for Looker and LookML." Yep, that's something for the Looker DevOps crowd to dig into for sure. As you might have guessed, LookML is a language for describing dimensions, aggregates, calculations and data relationships in a SQL database. It also enables LookML projects, which is a collection of model, view, and dashboard files that are generally version-controlled via a Git repository.

I'm all for productizing community work for broader consumption, but why is Spectacles needed? What difference can it make for Looker customers? Baker told me that it all started via Looker community meetups in New York City:

Josh and I met through Looker, and liked the community. We decided that we saw an opportunity to build an open source tool that provides testing around Looker, and so we have collaborated on that.

Looker JOIN photo
Temple (left) and Baker (right) at Looker JOIN

When I spoke to Baker, they were onboarding beta testers for the open source tool. Their presentation at JOIN brought them a ton of feedback, and it was about time to test the tool internally. Temple added:

I think it was something that was just a need for us personally. And in some ways, today was a confirmation that it's a need for everyone... There's a movement in data toward DevOps types approaches for testing, monitoring and continuous integration - all these technologies. How can we get on that bandwagon? How can we provide that for [Looker users]?

A moment of frustration gave birth to Spectacle. At an NYC Looker meetup organized by Namely's Alexander Jia, who I wrote about in Namely's Alexander Jia shares the new rules of modern BI projects, Baker vented spleen: 

I was speaking at a meetup in New York in March, and I made a flippant comment at the end, which was along the lines of, "I really just want something in Looker. Someone go build this thing in Looker, that does this type of testing. I'll pay someone to do it." Josh grabbed me at the bar afterwards, and he said, "We should go do that thing."

So what can you do with spectacles? A bunch of nifty DevOps things. The tool allows you to perform a range of tests on your Looker branch, so you can "merge LookML with confidence." As in: 

  • SQL validation - tests for database errors or invalid SQL
  • Assert validation - runs LookML/data tests
  • Content validation (coming soon)
  • Linting (coming soon)

As I see it, this is all about helping Looker engineers and analysts spot dirty data, data anomalies, and dirty data sooner, as part of a more automated workflow. That matters to business users and analysts. If you can't deliver quality data, forget about keeping their trust. Or, as the Spectacle team puts it:

Add Spectacles to your continuous integration pipeline to check for invalid SQL, bad assumptions, broken content, and more.

"You need to build up that trust with the end user"

Looker is a tool that is so flexible it can be hard to wrap your head around, for better and perhaps sometimes for worse (I struggled to capture the range of use cases in Looker CPO - the data nerds won. But if we don't empower users with data, we lose). That's really due to Looker's mission to empower users to run their own queries, and fulfill their own analytics needs. To pull that off, the data stakes are high: As Baker put it:

For us, building on Looker is about building a data product. What you need to do is build up that trust with the end user. If you're offloading the ability to query, if you're providing self-service capabilities, they need to have a high level of trust in what they're doing. Because if they see those errors, you ruin that trust really quickly.

So how does this tool help? Baker:

Basically, it tests that every dimension you build in Looker is a built-in legitimate query. It builds all their sequel queries and tests them and says, "Are there any errors"? And then it reports back to you, and it can be run on pull requests and changes before they get merged in.

More functionality is on the way:

Looker recently announced data test functionality in testers as well. There's a whole bunch of stuff we'd like to add on top, that we see is the next step of the tool.

Anyone who wants to get their hands on Spectacles doesn't have to wait for the hosted version in the Looker partner network. They can use the open source version now. Baker:

We love people to get in touch so we can handhold them through it, and really understand where the pain points are.

Being early adopters of a platform can expose weaknesses, or immature APIs. How did that go with Looker?

I don't think there's any API in the world that is perfect. But the coverage of the Looker API is broad, and it does a lot of what you want it to do.

When you essentially productize a tool - open source or not - you want relationships with the vendor you are connecting to. What about those? Temple:

Something that's cool about Looker is that a lot of the Looker employees, they like to do their own hacking and building. If you just get them excited about what you're doing, you have an internal ally. It's very easy to find an internal champ. Then you can go and bother them about the issue and get something fixed, or get something pushed through. We were lucky to have a couple of people who are really excited about this, and who supported us super well.

The wrap

Part of Looker's charm - and effectiveness - is that type of community spirit. Count me amongst those who sees community as a direct tie to customer value and productivity. That said, I do wonder how all of this will fare now that Looker is owned by Google. Google is no stranger to open source, but when you think "Google," community is not the first word that comes to mind. Time will tell. For now, I'll look forward to an update at next year's JOIN, now that Spectacle is just about ready for wider adoption.

I also talked with Temple about Milk Bar's Looker project, a story I won't go into detail on here. But the neat thing is how Temple was able to serve so many business users within Milk Bar as a self-described "team of one." After a month and a half of implementation, with the Looker team in support, Temple was ready:

What I decided to do was attack a narrow section of our business and build it end to end, and then expand out from there. I was really happy I did that. In the sector I went after first, which was pretty much just revenue reporting, immediately people were like, "This is awesome. I love this. What do you mean it takes 10 seconds, and I don't have to go wait for this awful portal built by a restaurant - legacy software?" It was great right away.

At the time of our interview, Temple was looking ahead to a new feature rollout:

We're just rolling out something right now, which I'm excited about. All of our store managers previously had this performance tracker, and it was a Google sheet, but they were responsible for updating the Google sheets, so they had to go and get the sales numbers and get their labor numbers and their waste numbers and collecting it.

Not exactly a process that gets users excited, is it?

First of all, there's no accountability there. Second of all, it's a pain. The job of a store manager is not to be a data monkey and pull numbers from different places. Their job is to bring context to what happened historically and to just make decisions off of it.

Temple was confident his team would win them over:

What we're doing now is: they get an emailed report from Looker that is a dashboard that I've constructed that gives them all of that data. It's got the date, and it's accurate. And they can look at something like a chart that shows for each hour, "Here's how many people you scheduled, and here's how many orders that came in during that block. So here's the average number of orders each employee was handling during that timeframe." Well, now they can easily see the schedule. Maybe we need more people between 11:00 and 12:00. Feedback cycles can be so much faster.

I'll look for an update from Milk Bar next chance I see them. For now, it's confession time. Yes, I did sample the Milk Bar menu. In "A River Runs Through It," Robert Redford's character was haunted by water. Well, I'm haunted by pink neon.