dbt Project Configuration

Project Architecture

Dependency Management

Environment Configuration

Detailed Explanation

dbt projects are the fundamental organizational unit in dbt. They contain all your models, tests, macros, and configurations.

What is Project Configuration?

The dbt_project.yml file is the central configuration file for your dbt project:

• Project metadata: Name, version, profile
• Model paths: Where to find models, seeds, tests
• Model configurations: Default materializations, schemas, tags
• Variables: Custom variables for dynamic configuration
• Clean targets: Files to remove during cleanup

What is Package Management?

Packages are reusable collections of macros, models, and tests:

What is Profile Configuration?

Profiles define how dbt connects to your data warehouse:

• Target environments: Dev, staging, production
• Connection details: Account, user, password
• Warehouse settings: Size, cluster, timeouts
• Schema configuration: Dynamic schemas per user

What are Environment Variables?

Environment variables allow dynamic configuration:

• Secrets: Passwords, tokens (never commit to Git)
• Environment-specific: Different settings per target
• User-specific: Per-developer configurations
• CI/CD: Pipeline-specific settings

What are the Project Best Practices?

1. Use version control for all configuration files
2. Never commit secrets - use environment variables
3. Separate environments - dev, staging, production
4. Document configurations - add comments and descriptions
5. Test configurations - validate before deployment
6. Use packages - leverage community code
7. Version constraints - specify version ranges for packages
8. Clean regularly - remove unused targets and packages

Key Takeaway: Proper project configuration ensures maintainability, security, and scalability of your dbt data transformation pipeline.

Code Examples

dbt_project.yml

# dbt_project.yml
name: 'my_analytics_project'
version: '1.0.0'
config-version: 2

profile: 'analytics'

model-paths: ["models"]
analysis-paths: ["analysis"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
docs-paths: ["docs"]

clean-targets:
  - "target"
  - "dbt_packages"
  - "dbt_modules"

models:
  my_analytics_project:
    staging:
      +materialized: view
      +schema: staging
      +tags: ['staging']
    
    intermediate:
      +materialized: ephemeral
      +tags: ['intermediate']
    
    marts:
      +materialized: incremental
      +schema: analytics
      +tags: ['mart', 'production']
      
      finance:
        +cluster_by: ['date', 'account_id']
      
      marketing:
        +partition_by: {
          "field": "event_date",
          "data_type": "date"
        }

vars:
  start_date: '2020-01-01'
  enable_audit: true
  default_currency: 'USD'

query-comment:
  comment: "dbt: {{ node.unique_id }} | {{ node.description }}"
  append: true

packages.yml

# packages.yml
packages:
  - package: dbt-labs/dbt_utils
    version: [">=1.0.0", "<2.0.0"]
  
  - package: calogica/dbt_expectations
    version: [">=0.10.0", "<1.0.0"]
  
  - package: dbt-labs/codegen
    version: [">=0.12.0"]
  
  - package: elementary-data/elementary
    version: [">=0.14.0"]
  
  - git: "https://github.com/my-org/custom_dbt_package.git"
    revision: main
  
  - registry: dbt-labs/metrics
    version: [">=0.3.0"]

profiles.yml

# profiles.yml
my_profile:
  target: dev
  
  outputs:
    dev:
      type: snowflake
      account: "{{ env_var('SNOWFLAKE_ACCOUNT') }}"
      user: "{{ env_var('SNOWFLAKE_USER') }}"
      password: "{{ env_var('SNOWFLAKE_PASSWORD') }}"
      role: TRANSFORMER
      database: ANALYTICS_DEV
      warehouse: COMPUTE_WH
      schema: "dbt_{{ env_var('DBT_USER') }}"
      client_session_keep_alive: false
      query_tag: "dbt_dev"
    
    staging:
      type: snowflake
      account: "{{ env_var('SNOWFLAKE_ACCOUNT') }}"
      user: "{{ env_var('SNOWFLAKE_USER') }}"
      password: "{{ env_var('SNOWFLAKE_PASSWORD') }}"
      role: TRANSFORMER
      database: ANALYTICS_STG
      warehouse: ANALYTICS_WH
      schema: public
      client_session_keep_alive: true
      query_tag: "dbt_staging"
    
    prod:
      type: snowflake
      account: "{{ env_var('SNOWFLAKE_ACCOUNT') }}"
      user: "{{ env_var('SNOWFLAKE_USER') }}"
      password: "{{ env_var('SNOWFLAKE_PASSWORD') }}"
      role: TRANSFORMER
      database: ANALYTICS_PROD
      warehouse: ANALYTICS_WH
      schema: public
      client_session_keep_alive: true
      query_tag: "dbt_production"

Model Configuration

# models/marts/fct_orders.yml
version: 2

models:
  - name: fct_orders
    description: "Fact table for orders"
    
    config:
      materialized: incremental
      unique_key: order_id
      incremental_strategy: merge
      partition_by: {
        "field": "order_date",
        "data_type": "date"
      }
      cluster_by: ['customer_id', 'status']
      tags: ['finance', 'core', 'production']
      
      meta:
        owner: data-engineering
        team: analytics
        cost_center: finance
        pii: false
    
    columns:
      - name: order_id
        description: "Unique order identifier"
        data_tests:
          - unique
          - not_null

Performance Metrics

Best Practices

1. Use version control for all configuration
2. Never commit secrets - use environment variables
3. Separate environments - dev, staging, production
4. Document configurations - add comments
5. Test configurations - validate before deployment
6. Use packages - leverage community code
7. Version constraints - specify version ranges
8. Clean regularly - remove unused targets

Interview Q&A

Q1: Why do we use environment variables in dbt instead of hardcoding values?

Answer: Environment variables keep secrets (passwords, API keys) out of version control, allow different configurations per environment (dev/staging/prod), enable per-developer customization, and support CI/CD pipelines. Using env_var() in profiles.yml means the same config file works across all environments without exposing sensitive data.

Q2: Explain the difference between profiles.yml and dbt_project.yml.

Answer: profiles.yml lives outside the project (in ~/.dbt/) and defines connection details—how to connect to your data warehouse (account, credentials, database, schema). dbt_project.yml lives inside the project and defines how to build models—paths, materializations, variables, and model configurations. Profiles are environment-specific; project configs are project-specific.

Q3: How would you set up dbt for multiple environments (dev, staging, prod)?

Answer: Configure separate targets in profiles.yml with different databases/schemas. Use environment variables for credentials. In CI/CD, set --target prod for production runs. Each developer gets their own schema via DBT_USER. Use dbt_project.yml vars for environment-specific settings like date ranges or feature flags.

Q4: What is the purpose of the target field in profiles.yml?

Answer: The target field specifies which output configuration to use when running dbt. It defaults to dev but can be overridden with dbt run --target prod. This allows a single profiles.yml to define multiple environments, and you switch between them without changing the file.

Q5: How do you handle secrets in a dbt CI/CD pipeline?

Answer: Never commit secrets to Git. Use your CI/CD platform's secret store (GitHub Secrets, GitLab CI Variables, AWS Secrets Manager). Set environment variables in the pipeline before running dbt. Use env_var() in profiles.yml to reference them. Rotate secrets regularly and use service accounts with minimal permissions for production.

Q6: What happens if an environment variable is not set when dbt runs?

Answer: dbt will raise an error and fail compilation. You can provide a default value using env_var('VAR_NAME', 'default_value'). This is useful for optional settings. However, critical secrets like passwords should not have defaults—fail loudly if they're missing.

Q7: Explain the schema naming strategy for multi-developer environments.

Answer: Use schema: "dbt_{{ env_var('DBT_USER') }}" so each developer gets their own schema (e.g., dbt_alice, dbt_bob). This prevents conflicts during development. In production, use a fixed schema like public. This pattern keeps dev isolated while maintaining a clean production namespace.

Q8: How do you validate dbt configurations before deployment?

Answer: Run dbt compile to check syntax. Run dbt debug to validate connections. Use dbt ls to verify models are discovered. Run tests with dbt test. In CI/CD, add linting (sqlfluff), schema validation, and dry-run builds. Test against a staging environment before production deployment.

KnowledgeCheck

See Also

• dbt Core Architecture — Manifest, DAG, and compilation pipeline
• The ref() Function — Model reference resolution and dependency management
• Jinja Templating in dbt — Template syntax, macros, and dynamic SQL
• Snowflake Architecture — Snowflake cloud data platform fundamentals
• Data Engineering Fundamentals — Modern data stack overview