Part 1: Prerequisites for starting a dbt project

This is Part 1 of a series of posts about bringing dbt into organizations where new tech usually moves slowly and carefully, like government, academia, and nonprofits. The series walks through how to get started, how to organize your work, what good looks like, how to sell it to your boss, and how to scale your project and team.

So you’ve decided to start your organization’s very first dbt project. Congratulations! You are in for some exciting times, and probably also a little bit overwhelmed. Starting a new project using a new technology can be scary, because there are a lot of decisions you need to make while still in a world of uncertainty. You don’t know which decisions are going to be consequential (and thus result in a lot of rework later), or which need to be made first in order to actually progress.

Today we’re going to talk about all of the things you need to have ready before building your dbt project. I’m going to lay out exactly what decisions you need to make at this early stage, my recommendations, and some alternatives (aka backup plans). We’re not going to start building yet (and all of the decisions that come with that stage). We’ll cover building your first dbt models and data products in next week’s post!

Think about baking a new cake recipe. A lot of planning happens before you ever start baking - especially for new recipes! You don’t just start tossing ingredients in a bowl. First, you read the recipe. Then, you gather your ingredients. You may need to go shopping or research substitutes. You may need to preheat your oven. For more complicated recipes, you may need to plan out your steps with a timetable. And to extend the analogy, you may want to experiment with a new recipe first, making smaller versions of the cake to taste-test before you bake the final cake worthy of the Great British Bake Off’s judge’s table (or at least the birthday party).

So, what do we need in order to bake our dbt cake? I’m going to divide the prerequisites into 3 categories:

• Things you need to want (motivations matter!)
• Things you need to have (on your computer)
• Things you need to know

3 Things You Need to Want

A dbt project is likely a good fit for you if you:

1. Want to transform the same data repeatedly
2. Are willing to invest upfront
3. Want to build resilience for yourself and your organization

Before you start baking, let’s take a moment to pause and consider why you are starting this dbt project and what you are hoping to get out of it. You are about to invest a lot of time (and therefore money) into this initiative, and we want it to be a successful one. At the end of the day, if your motivations are not aligned with what dbt is designed to do, it won’t matter how well you build out the project - it won’t be serving your needs and it will eventually be consigned to the project graveyard.

You want to transform the same data repeatedly

Generally, whenever someone is manually doing the same thing over and over again, with little need for creativity or experimentation, it is probably a good candidate for automation. So if you have a dataset, and you need to summarize it in order to produce some metrics, and you need to do this every month because new data has been added, that is a perfect candidate for dbt. But if you work with a different dataset for every project, and you never reuse that data, and you have zero interest in preserving what you did with those past datasets, dbt probably has very little to offer you.

You're willing to invest upfront

Starting a dbt project will mean upfront work in order to drastically reduce the manual work needed for every repetition that comes later. Sometimes all of your work hours are consumed by fire fighting and urgent requests, and you simply don’t have the capacity for a project that cannot show immediate returns. Embarking on a new dbt project takes time - especially if we take into account the learning curve. So, you need to have a buffer - time to build, experiment, learn, and iterate… all before you can show value to leadership. But if you stick with it, you will free up time that you have been spending doing repetitive work manually.

Of course, this also has to be a goal - if you like doing these manual tasks over and over again… or if you consider this to be your job security… then you are not going to appreciate having a dbt project. That may not describe you, but it may describe other people at your organization.

I will also caveat that just how much of a buffer you need to account for will depend on where your organization is currently at on the data maturity scale. We all start at different places, and some of us have a lot further to run in order to reach that first major checkpoint of a functional dbt project. Do you already have a data warehouse? Do you already have data transformations written in SQL? Is your team used to working in git repos and doing code reviews? If so, you may get to speedrun this migration.

You want to build resilience for yourself and your organization

This is about whether you want to encode the data knowledge currently locked away in people’s heads in a way that can outlast any one person. For some folks, this is going to sound very appealing: it means the freedom to go on vacations without stress, less worry that everything will crash and burn if a key team member decides to leave, and generally feeling comfort around socio-technical systems resilience. On the other hand, some people take comfort in feeling needed, and the thought that everything could continue running smoothly without them brings on feelings of panic and insecurity rather than relief and satisfaction. If you take delight in knowing that you are building something that will outlast you, then dbt is probably a good choice.

3 Things You Need To Have (on your computer)

1. A git repository
2. A database
3. A development environment

So you've decided you want to bake; now let's get the ingredients on the counter. First, let’s talk about what a “dbt project” actually is for a minute. A dbt project is essentially just a collection of YAML and SQL files organized into a specific folder structure. The dbt_project.yml file is required and contains all essential configurations, including specifying the folder structure. But no actual data lives in these files (well, except for seeds, but we’ll set that aside for now); the sql files just contain the instructions for how to transform the data. So you also need a place for the actual data to live - a database. You specify the connection details for the database in the profiles.yml file (also required). Finally, you need to be able to actually run dbt commands in order for the transformations specified in the sql files to materialize in your database. In other words, you need a development environment with dbt installed. If you have all 3 of these things, you have a functional dbt project.

However, I will also note that you will most likely need to work with your organization’s IT department to get properly set up with each of these three elements. It is worth some advance investigation to figure out who is the right person to talk to and what forms to fill out to get the resources you need, in order to figure out the most efficient path through your organization’s processes.

A git repository

The very first directive of the first principle in the dbt Viewpoint is that analytics code should be version controlled. The dbt project file structure, therefore, is intentionally designed to be very friendly to git workflows.

So you will need to create a git repository (repo) backed by your platform of choice. You should use the code repository platform that your organization uses - the usual suspects are GitHub, GitLab, or Bitbucket. While you're at it, ask your organization’s DevOps or IT team whether your organization has an account or private instance you should be using.

If your organization does not have a preferred git repo hosting platform (and no plans to adopt one), then I would recommend creating a private GitHub repo (which also requires creating a personal GitHub account if you do not have one). If your organization ends up creating an Organization account later, you can always transfer ownership of this repo. And you can share access to this private repo with specific accounts once more people start contributing.

Even if you don’t (yet) have access to your organization’s git platform, at minimum install git on your computer and initialize a local repo. Your project will still be version controlled (albeit limited to a single computer), and you can connect the local repo to the remote repo on your platform of choice later.

(At this point, you have no excuses for skipping setting up your dbt project in a git repository!)

You will also need to decide on a repo name. This is a major decision point! While you can always change the name later, changing it can be a bit of a painful process - multiplied by however many people have cloned the project. So I would recommend choosing your repo name carefully, as your repo name is also usually an indication of scope.

Is this a proof-of-concept project, something just for you to learn dbt and demonstrate its potential value? Is this repo scoped to transformations for a specific project, or will it house all transformations built by your team? Or is the scope going to be even larger - maybe you envision it being used by the whole department?

If you are new to dbt, my recommendation is to start with creating a project at the proof-of-concept scope, so that you can feel more free to experiment and make mistakes with minimal consequences. Then, create a project at the team scope. While it may be tempting to create a dbt project for every project, you will find that the value of keeping all of your transformations in a single project compounds with each additional model you add… as long as you are also practicing good data modeling.

Back to your repo name - I recommend putting your project scope as the first part (e.g. your team name) and “dbt” or “dbt-project” as the second part, all lowercase, consistently using either “-” or “_” instead of spaces.

I find the easiest flow is to create the repo on your platform (e.g. GitHub) first, and then clone the repo to your local computer - but follow whatever flow you are most comfortable with, or is currently available to you.

A database

The database will be where all of your data is stored, and where you query and access your data. In order to use the database with a dbt project, you need to select a database with a dbt adapter. Adapters contain the DBMS-specific code that connects dbt to your database, and they handle all of the DDL (create, update, delete statements) that dbt abstracts away (you only write select statements). Check this list to make sure your database has a dbt adapter - most databases have one that is officially or a community-maintained. Make sure to read the adapter’s documentation, as each adapter can have custom configurations.

The best option for your dbt project’s database is the database your organization already has for analytical use cases. This may be a dedicated data warehouse like Snowflake, or a managed PostgreSQL database optimized for analytics. You do not want to run analytics workloads on a database that is running transactions for a production application. If you don’t know whether an analytics database is available, ask your organization’s DevOps, Data, or IT team (whichever exists).

If your organization does not already have an analytics database and your organization is not willing to set up the necessary infrastructure in order to support your dbt project (yet), do not worry - you are not out of luck! You can use DuckDB, a free open-source database that exists as a file on your computer (just like SQLite), and is an extremely fast option for analytical workloads. This is a great option for proof-of-concept projects or any project where the team has a size of 1, but it is inconvenient for projects where you need to collaborate with multiple people.

After deciding on your database, you need to make some initial decisions about how your tables and views will be organized inside of your database. Each table & view needs not just one place to live, but two. The first directive of the second principle in the dbt Viewpoint is that analytics requires multiple environments - namely, a development environment and a production environment.

Most databases provide only 1 layer of organization inside of the database, the schema. So you will want separate production and development schemas. A common pattern (which your organization may already follow) is for each developer to have their own dedicated development environment and dedicated schema.

When you are first starting your project, you only strictly need a development environment (schema), but to fully realize the benefits of dbt, you will want to provision a production environment (schema) as well. For your production environment, you will want to create (or request) a new schema, and name it according to your chosen project scope (not “dbt”). I would highly recommend that you constrain this schema to only contain tables and views created by dbt. Your data sources should live in different schemas.

Of course, you can feel free to use more than 1 dev/prod schemas as needed/desired, but I am focusing on the minimum viable dbt project configuration here (using multiple schemas also introduces complications I don’t want to address in this post). And with data warehouses like Snowflake, you have an additional layer of organization available beyond schemas if needed.

To summarize, you will need to choose a database where your data will live, and create dedicated development and production schemas. I recommend naming your development schema to be aligned with your database username, and naming your production schema to be aligned with your project scope.

A development environment

So you have a repo and a database - amazing! You still can’t actually do anything until you can run dbt commands, which means getting dbt installed, preferably in a dedicated environment.

The third principle in the dbt Viewpoint is that analytics workflows require automated tools, and pipelines that move data from multiple sources, test it, and run it in a specific environment, all with a single command. That command is ‘dbt build’. Once you see how much is automated by that single command, there’s really no going back.

Like the previous requirements, this likely involves talking to your IT department.This may even be the biggest hurdle: you need to be able to run python and download python packages on your machine. For many IT departments this can be a scary proposition (if not for yours, count your lucky stars!), but with time and patience (and many promises to the security team) they can be convinced. If you are able to get python on your machine, I highly recommend using uv to install and manage python versions, create virtual environments, manage package dependencies, and everything else that uv does (and does exceptionally well). If, however, you are not able to get python on your machine (yet!), you still have some alternative options.

First, as an individual developer you can use the online dbt Platform IDE for free. You can create a free account, connect to your git repository, connect to your data warehouse, and immediately start running dbt commands from the browser-based IDE with no need to download anything. There are limitations to the free tier, but this can be a great way to get started if you are waiting on IT for the ability to run python locally.

Second, you also now have the option of installing dbt v2 as a software binary. However, this option comes with some pretty big caveats, which means we do need to talk about the differences between dbt Core (now dbt v1) vs dbt Fusion (now dbt v2).

A quick bit of background: dbt was first released in 2016 as a python package, reaching v1.0 in December 2021, the same year Fishtown Analytics rebranded as dbt Labs. In early 2025, dbt Labs acquired SDF Labs and started working on the next generation engine for dbt - a fusion of dbt and SDF that would give analytics engineers superpowers through SQL comprehension and Rust-powered speed.

dbt v2 is much newer technology (it was just officially released for General Availability this week!). Bugs are still being found and ironed out at a much more rapid pace than for dbt v1, fewer databases are supported, and the features are less stable. In both cases, you interact with dbt through the CLI (command line interface), and for an enhanced experience you can use a VS Code extension - the official one or a community developed one.

As of right now, for the intended audience of this series (those working in government, nonprofit, academia, etc), I recommend using the latest version of dbt v1 (v1.12 was recently released). Stability, reliability, wider support, larger userbase, longer history on forums so when you have an issue you can find somebody else who has already solved it… I place a higher value on all of these things than having access to the latest and greatest features or the fastest project compilation times.

However, if the fact that dbt v1 is a python package is a blocker, dbt v2’s installation may be the more accessible option. It is a precompiled binary (available in two distribution types), so if your database of choice is supported, then by all means go straight for the future! I certainly don’t recommend against dbt v2, I just have a relatively low risk tolerance and would consider myself part of the “early majority”, so it’s not my top choice for others like me yet. I plan to re-evaluate my recommendation in about 6 months, but if you are an early adopter and your organization has a higher risk tolerance, you will probably find the benefits of dbt v2 outweigh the risks.

In conclusion, my recommendation is to get the latest version of dbt v1 on your local machine and to manage your python environment with uv. A good backup plan if you are still working on IT is to use the dbt Platform IDE’s free Developer tier. If you would like to try the latest and greatest version of dbt (bugs and all), you can install dbt v2 (aka dbt Fusion) without needing python at all.

3 Things You Need To Know (and 1 more you really should learn)

1. git (for version control)
2. SQL (for data transformation)
3. dbt (at least the basics)
4. Bonus: Data modeling fundamentals

As we covered earlier, a dbt project is a collection of SQL and YAML files in a version controlled repo, where each SQL file contains a single query that describes how the data should be transformed in the database in a modular fashion. To write these queries, you need to know SQL. To manage the version history of the repo, you need to know git. And to know what goes in the YAML files and how to execute commands, you need to know dbt.

Both SQL and git are highly transferable skills used across the data & software industries. If you know SQL and git, then you are ready to start learning dbt - a separate skillset you can learn as you build your first project.

There is one additional skill that will be enormously helpful in figuring out just how to break a dbt project’s SQL code into modular, organized components: data modeling. However, data modeling is also a much more difficult skill to learn - there are many different approaches, and it will take a long time to master. I personally consider data modeling to be more of a philosophical practice than a technical skill… just with important technical applications. For these reasons, I do not count it as a “prerequisite” thing you need to know before building a dbt project, but I do believe that knowing the basics can help you steward a healthy one.

I’m going to quickly review some resources that you can use to learn (or just get a refresher) on each of these skills. And if you have any further resources to add to the list, please let me know!

Git

• dbt Learn offers a course on git fundamentals
• The Carpentries offers a 4-hour workshop on git for novices; there is a Carpentries Incubator version that incorporates branches and PRs; and I adapted both of these into a condensed 90 minute “git speedrun” workshop
• The Carpentries also offers a 4-hour workshop on the Unix Shell, if you would like to start by getting familiar with the command line interface (CLI)

SQL

• dbt Learn offers an intro course on SQL
• The Carpentries offers a 4-hour workshop on Databases and SQL
• The book SQL for Data Scientists is a great introduction with companion exercises

dbt

• The very first course anyone learning dbt should take is dbt Fundamentals, which also now has a version geared for those who prefer working in VS Code.
• After completing the fundamentals course, there are a lot of other topics to learn to become a competent Analytics Engineer. dbt Learn has helpfully compiled all of these together into a dbt certified developer learning pathway
• For any other topic within dbt, the dbt Learn catalog is probably the best place to start. Check if there is a course available - and if not, the team is very open to feedback on courses that would be useful to add to the catalog!

Data modeling

• Joe Reis is publishing a book about data modeling called “Mixed Model Arts”, which is going to be one of the best ways to get up to speed on all of the different data modeling approaches out there. He’s also been publishing the book incrementally on his blog, Practical Data Modeling.
• The best way to learn about Kimball Dimensional modeling is to read the book: The Data Warehouse Toolkit
• Decomplexify has a great explainer video and an ebook about database normalization
• dbt has some blog posts you can check out on topics like Kimball dimensional modeling, data vault, medallion architecture, and normalization vs denormalization

What to expect next

Now that you know what you need to get started, the rest of the series will get you using those knowledge and tools on your first project. The series intro already gave a quick primer on dbt. Over the next couple months this series will walk through:

• What goes into a dbt project and how to organize it
• The best way to get from raw data chaos to clean data product and what the migration actually looks like
• How to sell your dbt project to your boss (and everyone else)
• How to scale your project and team

Next, we'll dig into what goes into a dbt project and how to organize it. Buckle up, this ride is really getting started now!