Open-sourcing dbt-score: lint model metadata with ease!
With the rise of tools like dbt, it has become much easier to document, test and create data models in a data warehouse. In a previous blog post we highlighted how most of this can be achieved through a simple declarative approach. With that configuration, it is possible to define custom metadata properties. This results in an increase of metadata related to data models, which is great! Metadata includes documentation, tests, column types, constraints, and more. At scale it becomes increasingly difficult to manage all of it. That’s where dbt-score comes in!
As an organisation grows, data teams grow and the number of people working on (and with) data models increases. This means that developing data models becomes democratised and people from various teams and backgrounds can collaborate on them. They usually work together on SQL and metadata, building upon each other’s work. This is a good thing, as people closer to the business can create their own data models, leading to faster development cycles.
So, the amount of metadata associated with data models is increasing as is the number of people developing them. This is a recipe for disaster because chaos can ensue. How will we ensure that data models across teams have high-quality metadata? Defining standards is easy, ensuring the standards are met is a different story!
