Intro: How to build your organization’s first dbt project

Welcome to How to build your organization’s first dbt project

If you work in data, there’s a decent chance you have heard of dbt (if not, skip to this section). The open source python-based CLI has been around for a decade now, and it has become the tool of choice for transformation pipelines in data warehouses. But if you work somewhere with low risk tolerance and even lower IT budgets, there is also a decent chance that you do not yet have an active dbt project at your organization. And if you know about dbt - and the many benefits that can come from shifting data transformations over to a dbt project - but your organization has yet to adopt dbt, then you probably feel that gap very keenly.

If that’s you, welcome! I’m Jenna Jordan, and this is the first post in How to build your organization’s first dbt project - a new series about bringing dbt into organizations where new tech usually moves slowly and carefully, like government, academia, and nonprofits.

You may be a data team of one, losing time each week to the manual effort of dragging data around and wrangling it into shape across dozens of spreadsheets and even more tabs. You may be on a data team putting out fires every day from any number of fragile, custom pipelines written by team members who left long ago. You may be a newly promoted data manager, in charge of making sure key reports are refreshed on time, with no direction on how to accomplish it. If you work in government, academia, or nonprofits you are probably working with what Leana Critchell describes in this blog post as “human-generated small data — numbers that exist because a person sat down and entered them” and the standard industry toolkits likely feel like a poor fit. Whatever fresh new data hell you are trapped in, you might have an inkling that dbt can help get you out of it - or at least tame the chaos into something manageable.

This series will help you figure out:

• Where to start and how to create your first dbt project
• What goes into a dbt project and how to organize it
• The best way to get from raw data chaos to clean data product and what the migration actually looks like
• How to sell your dbt project to your boss (and everyone else)
• How to scale your project and team

We will release one post a week so stay tuned for the rest of the series!

Let me introduce myself

Why am I qualified to be your guide on this multi-week series? Because I’ve been in your shoes. I was on a city government’s data team and I knew dbt could substantially improve our data infrastructure, so I embarked on a year-long journey to pitch, design, lead, and implement the migration to dbt. You can learn more about that journey by watching this Coalesce talk or reading this blog post. I caught the dbt bug, and I’ve been building dbt projects - of varying scale and complexity - ever since. Now, I build dbt projects for public health clients at Ratio PBC (notably the New York State Department of Health).

If this is the first blog post of mine that you are reading, then you probably don’t know that I started my career in data with my Master’s in Library & Information Science. While in library school, I got involved with The Carpentries, an organization that publishes open-source collaboratively-built lessons and trains instructors in their evidence-backed ‘code-along’ pedagogical approach. You see, I intended to become a data librarian, and one of the things data librarians do is teach academic researchers how to use tools like git, bash, sql, python, R, etc for computational, data-intensive research (which also enables reproducible research). I got certified as a Carpentries Instructor, and I’ve taught Carpentries workshops at pretty much every job I’ve had since. That background gives me strong opinions on how to teach technical tools to newcomers - and it means I can’t just focus on the technology - I see everything through the framework of people, process & technology.

Before we start, a quick primer on dbt

In the unlikely event you have read this far without knowing what in the world I’m talking about… I’m honored, really, but let’s get our bases covered. As you might imagine, there is a lot already written about dbt, a tool that has been around for a decade, became extremely popular in the data industry, and is the foundation for a billion-dollar company. So I’m going to point you to some of my favorite blog posts about what dbt is and pull out the most pertinent quotes, and then leave you to do your research in peace. Hopefully then you can be prepared for the next post in this series!

The elevator pitch is: “dbt is a command line tool that enables data analysts and engineers to transform data in their warehouses more effectively.” That is from the ultimate authority - CEO & founder of dbt Labs Tristan Handy - who explains what, exactly, is dbt in a blog post that was first written in 2017 and has since been periodically updated. It comes with a quick caveat though: “if you want to go deeper than the elevator pitch, you have to go pretty deep. If you want to understand what dbt is, where it fits into the ecosystem, and how you should think about using it…” then you should go read that blog post.

Also worth reading through is the dbt Viewpoint, which is basically a set of guiding principles for both the software and the analytics engineers who use it. The 3 principles discussed in the Viewpoint are: “Analytics is collaborative”, “Analytics code is an asset”, and “Analytics workflows require automated tools”.

With the rise of dbt came a new job title: the analytics engineer. Jason Ganz, Director of DevEx + AI at dbt Labs, wrote this blog post 5 years ago on why in 5 years every organization will have an Analytics Engineering team. Jason does a great job of articulating the value of an analytics engineer to their organization, from “being able to help build a true data driven culture in ways that would be challenging for a team of people using legacy tools” to “build[ing] complex data models using tools like dbt and hav[ing] analysis ready data tables built on any schedule.”

Ganz also elaborated on the dbt Viewpoint in this blog post, in which he set 3 missions for the dbt community, including “Spreading analytics engineering everywhere - despite the rapid growth and adoption of dbt, the simple fact of the matter is that most people who will ever use dbt have not started yet. To these people, the ideas and values expressed are going to be just as fresh and new as they ever were.”

Growth alone doesn’t explain why dbt sticks once teams adopt this seemingly simple stitcher of sql scripts. dbt changes day to day analytics practices. brittany bennet’s blog post “There's a better way: The case for dbt for progressive data professionals”, kicks off with as strong of an endorsement as you can get: “I swear by dbt. In fact, I refuse to work without it. I believe that dbt does more than just transform data; it can transform entire political data teams.” brittany makes her case through concrete examples drawn from her experience leading data teams in the progressive politics world, illustrating key dbt features along the way. On the magic of dbt: “There is a reason this tool has the cult-like fan base it's famous for. dbt has saved me and my team countless hours of work, freeing up our time to focus on delivering results to our organizers.”

There is no better summary of the case for dbt than the conclusion of Stephen Whitworth’s post “Why is dbt so important?” from 2020: “If you choose not to use dbt, you’ll probably waste time building a less-fully featured, buggy implementation of it yourself. Give it a serious look.”

What to expect next

So, if you’re ready to move from learning about dbt to starting a project in your organization, this series is for you. Over the next five weeks this series will walk through:

• Where to start and how to create your first dbt project
• What goes into a dbt project and how to organize it
• The best way to get from raw data chaos to clean data product and what the migration actually looks like
• How to sell your dbt project to your boss (and everyone else)
• How to scale your project and team

Next week, we'll dig into that first bullet - where to start and what you need in place before you write a single line of dbt code. Stay tuned. Grab a coffee, bookmark this page, and get ready - we've got a lot of ground to cover together.