In today’s digital world, data doesn’t just set the rhythm — it’s the heartbeat of businesses. But as with any dance, the challenge lies in syncing the structured steps with the improvisational moves. How can we find the perfect balance between flexibility and control, between agility and structure?
The Modern Data Stack
Every modern business places data at the core of its decision making process, to provide trusted outcomes and quick responses to ever changing needs. In the modern data stack, this requires a system where the business logic remains robust and updated, and can be implemented transparently in the data layer, not trapped in siloed BI tools like Looker or Tableau.
Enter dbt™️
This vision becomes a reality with the emergence of dbt (data build tool): finally an industry standard for codifying business logic at the data layer, within the warehouse. But the benefits of dbt go beyond crafting a fancy data structure. It brings the best of software development - version control, documentation, standards - into the data world.
And with the arrival of dbt 1.6 comes the capability to define metrics in dbt, providing the entire toolkit to centralize business logic and establish a unified foundation for the whole organization. But how will this toolkit adapt to the ever-evolving business logic within our dynamic corporate landscapes? Will the data models be able to keep up with the fast dance of the ever-evolving business needs?
And herein lies the challenge: balancing the freedom of business analysts with the governance of data models.
Flexibility and Control
Some prioritize analysts’ agility, skipping some important central checks to meet business deadlines. But there are no free lunches, and over time this results in fragmented logic.
We have seen this take the form of duplicates in the dbt DAG, logic locked in BI tools, rogue scheduled queries in the warehouse and, in some cases, even cron jobs triggered Python scripts running off of individual analysts laptops.
Others might opt for extreme consistency, but this creates overhead and bottlenecks. Waiting for a centralized team to review every change can frustrate the business and bog down the central data engineering team with countless small tasks.
Ultimately, it’s about achieving harmony in the dance of data: freedom with structure, quick insights with accuracy, intuition with consistency.
Dynamic Data Modeling
Having studied over 200 data teams, I believe the solution lies in dynamic data modeling. In this approach, three key elements are required to achieve harmony:
Meet business analysts where they are
Provide a simple & frictionless interface for analysts to introduce to dbt new business logic created in their native environment.
Automate for integrity
Use automation to detect and prevent data model conflicts and duplications, ensuring quality without sacrificing speed.
Enable collaborative dynamics
Equip data teams with workflows that facilitate coordination between the different roles, varying from analytics engineers to business analysts.
A Reverse Flow
Business logic doesn’t bloom behind the scenes by data engineers. It’s developed by business analysts on the front lines, as part of their everyday tasks within their native BI environments. The switch to dbt is a significant technical upgrade, but it shouldn’t change the valuable role of analysts, who drive their power by being so close to the business.
As we bring dbt into our daily practices, we need to leverage technology so that whatever logic analysts develop in existing native environments could be coded under the hood in dbt using a powerful reverse flow.
This way, we free-up business analysts to do what they do best, and power-up analytics engineers to keep consistent data models at any scale, without compromising analyst autonomy. By implementing these focused strategies, we ensure the dance of data is both fluid and precise — setting the tempo for success.
So, shall we dance?
***
Euno it! Euno is officially on the dbt Cloud integrations list. Check us out under Data Catalogs.
