History at a Glance
When Looker entered the scene, it transformed our data practices, allowing us to employ state-of-the-art engineering principals to a centralized repository of reusable business logic. For data analysts, it became the go-to for crafting complex data transformations and defining metrics, all through the powerful lens of LookML.
Enter dbt™️
Then dbt Labs came along, setting new standards for data transformations right within our warehouse. The rule of thumb was straightforward: lay your foundations with dbt and then use LookML to teach Looker how to navigate it. This translated into crafting transformations in dbt, directing all Looker views to point to dbt models, and favoring a dbt-first approach for calculated fields while reserving Looker for computing metrics. But even with its most recent versions – allowing you to build your entire data model in dbt – the following classic problem persists.
Freedom vs. Governance
A complete data model in dbt is a significant step forward. It finally provides the engineering tools required to centralize business logic in a single place – setting a common ground for the entire organization. With this setup, the recipe seems straightforward – Looker becomes the go-to for visualization and data exploration, leaving the construction of data models, including transformations and metrics, to dbt.
Yet, an important challenge remains:
Business logic doesn’t bloom behind the scenes by data engineers; it’s developed by business analysts on the front lines, as part of their everyday tasks within their native BI environments like Looker. These analysts operate under tight deadlines with a business-centric approach; expecting them to become data engineers overnight is unrealistic.
A New Mindset
The transition to dbt isn’t just about learning new software; it’s about adapting to a fundamentally different mindset of data engineering. It’s more than a technical upgrade — it’s a cultural shift for analysts accustomed to a certain autonomy. Their expertise is in swiftly delivering accurate data analysis within Looker’s flexible interface, and any shift that might slow that process is naturally met with hesitation.
As we bring dbt into our daily practices, we find that inconsistent KPIs across various dashboards remains an issue, and sometimes might become exacerbated by the additional workflow. Striving for an aligned source of truth is hamstrung by the constant race to keep Looker and dbt in sync.
Looking Forward
Perfecting the Looker-dbt interface is key. While we have the engineering tools and solid understanding for modeling within dbt and syncing up with Looker, the reverse flow — capturing the inception of business logic crafted by analysts and embedding it into dbt — remains a territory awaiting innovation.
A solid solution has to provide a frictionless interface equipped with automated model generation tools to convert Looker definitions into dbt models effortlessly along with automation for consistency and quality checks to bridge discrepancies between Looker and dbt, ensuring robust data models.
As data leaders, we continue to search for the right balance between freedom and consistency. The ultimate goal is a data lifecycle that is both dynamic and cultivates trust. We believe this is the only way to reach that goal.
Where Do You Stand?
Data leaders, is it Looker or dbt for your business logic? Have you found the silver lining?
***
Euno it! Euno is officially on the dbt Cloud integrations list. Check us out under Data Catalogs.
