dbt Cloud Features
Cloud Architecture
Architecture Diagram
┌─────────────────────────────────────────────────────────────────────────────┐
│ DBT CLOUD ARCHITECTURE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ DBT CLOUD PLATFORM │ │
│ │ │ │
│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────────┐ │ │
│ │ │ IDE │ │ JOBS │ │ MONITORING │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ • Web-based │ │ • Scheduled │ │ • Run history │ │ │
│ │ │ • Git │ │ • On-demand │ │ • Performance │ │ │
│ │ │ • SQL editor │ │ • PR triggers│ │ • Costs │ │ │
│ │ └──────────────┘ └──────────────┘ └──────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ INFRASTRUCTURE │ │
│ │ │ │
│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────────┐ │ │
│ │ │ RUNNERS │ │ SECRETS │ │ ARTIFACTS │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ • Managed │ │ • Encrypted │ │ • Manifest │ │ │
│ │ │ • Scalable │ │ • Rotated │ │ • Run results │ │ │
│ │ │ • Multi-env │ │ • Audited │ │ • Catalog │ │ │
│ │ └──────────────┘ └──────────────┘ └──────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Slim CI Architecture
Architecture Diagram
┌─────────────────────────────────────────────────────────────────────────────┐
│ SLIM CI PIPELINE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ TRIGGER EVENT │ │
│ │ │ │
│ │ ┌──────────────┐ │ │
│ │ │ Git Push / │ │ │
│ │ │ PR Created │ │ │
│ │ └──────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ 1. COMPARE STATEMENTS │ │
│ │ │ │
│ │ dbt run --select state:modified+ --state manifest.json │ │
│ │ │ │
│ │ Modified models: │ │
│ │ ├── fct_orders (changed) │ │
│ │ ├── dim_customers (new column) │ │
│ │ └── stg_payments (unchanged - skipped) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ 2. SELECTIVE EXECUTION │ │
│ │ │ │
│ │ dbt run --select +fct_orders+dim_customers │ │
│ │ dbt test --select fct_orders dim_customers │ │
│ │ │ │
│ │ Execution time: 5 minutes (vs 45 minutes full run) │ │
│ │ Cost savings: 89% │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Job Scheduling
Architecture Diagram
┌─────────────────────────────────────────────────────────────────────────────┐
│ JOB SCHEDULING ARCHITECTURE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ JOB CONFIGURATION │ │
│ │ │ │
│ │ Job: Production Run │ │
│ │ ├── Trigger: Scheduled (Daily 2:00 AM UTC) │ │
│ │ ├── Environment: Production │ │
│ │ ├── Commands: │ │
│ │ │ ├── dbt deps │ │
│ │ │ ├── dbt seed │ │
│ │ │ ├── dbt run --full-refresh │ │
│ │ │ └── dbt test │ │
│ │ ├── Notifications: │ │
│ │ │ ├── Slack: #data-engineering │ │
│ │ │ └── Email: team@company.com │ │
│ │ └── Alert on: failure │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ SCHEDULE PATTERNS │ │
│ │ │ │
│ │ ┌─────────────┬──────────────────────────────────────────────┐ │ │
│ │ │ Pattern │ Configuration │ │ │
│ │ ├─────────────┼──────────────────────────────────────────────┤ │ │
│ │ │ Hourly │ "0 * * * *" │ │ │
│ │ │ Daily │ "0 2 * * *" │ │ │
│ │ │ Weekly │ "0 2 * * 1" │ │ │
│ │ │ Monthly │ "0 2 1 * *" │ │ │
│ │ │ Custom │ "0 2 * * 1-5" (weekdays only) │ │ │
│ │ └─────────────┴──────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Detailed Explanation
dbt Cloud is the enterprise version of dbt that provides a comprehensive platform for data transformation with managed infrastructure, scheduling, and monitoring.
Core Features
1. Web-based IDE
- Write SQL and Jinja in the browser
- Git integration with visual diff
- Auto-completion and syntax highlighting
- Interactive documentation viewer
2. Job Scheduling
- Cron-based scheduling
- Event-driven triggers (Git, API)
- Dependency chains
- Alert notifications
3. Slim CI
- Selective execution of modified models
- State comparison between branches
- Cost optimization for CI/CD
- Fast feedback loops
4. Monitoring and Observability
- Run history and logs
- Performance metrics
- Cost tracking
- Error alerting
Enterprise Features
1. SSO and Authentication
- SAML 2.0 integration
- RBAC (Role-Based Access Control)
- Audit logging
- IP allowlisting
2. Multi-tenant Architecture
- Environment isolation
- Resource quotas
- Cost allocation
- Compliance controls
3. Semantic Layer
- Centralized metric definitions
- Version control for metrics
- API access
- BI tool integration
4. Mesh
- Cross-project references
- Data contracts
- Shared semantic models
- Governance controls
Best Practices for dbt Cloud
- Use Slim CI - Only test modified models
- Set up alerts - Notify on failures
- Monitor costs - Track warehouse usage
- Use environments - Separate dev/staging/prod
- Version control - All configurations in Git
- Document jobs - Clear naming and descriptions
- Test regularly - Automated quality checks
- Review logs - Monitor execution details
Code Examples
Job Configuration (YAML)
# .dbt_cloud/job_config.yml
jobs:
- name: "Production Run"
description: "Daily production run for all models"
environment: "Production"
triggers:
- type: "scheduled"
cron: "0 2 * * *"
- type: "git_push"
branches: ["main"]
steps:
- command: "dbt deps"
- command: "dbt seed"
- command: "dbt run --full-refresh"
- command: "dbt test"
notifications:
- type: "slack"
channel: "#data-engineering"
- type: "email"
recipients:
- "data-eng@company.com"
settings:
warehouse: "ANALYTICS_WH"
schema: "production"
threads: 8
alert_on:
- "failure"
- "warning"
Slim CI Configuration
# .dbt_cloud/ci_config.yml
ci:
enabled: true
state:
compare:
- "manifest.json"
- "run_results.json"
selection:
strategy: "modified"
include:
- "state:modified+"
- "state:new+"
exclude:
- "tag:deprecated"
optimization:
enabled: true
max_run_time: "30m"
cost_threshold: 100
notifications:
on_success:
- type: "slack"
channel: "#ci-results"
on_failure:
- type: "slack"
channel: "#ci-alerts"
- type: "pagerduty"
severity: "critical"
Semantic Layer Configuration
# .dbt_cloud/semantic_layer.yml
semantic_layer:
enabled: true
metrics:
- name: "total_revenue"
type: "simple"
expression: "sum(amount)"
description: "Total revenue from all orders"
- name: "order_count"
type: "simple"
expression: "count(*)"
description: "Total number of orders"
- name: "avg_order_value"
type: "derived"
expression: "total_revenue / order_count"
description: "Average order value"
dimensions:
- name: "order_date"
type: "time"
granularity: "day"
- name: "customer_segment"
type: "categorical"
access:
- type: "bi_tool"
name: "Looker"
permissions: ["read"]
- type: "application"
name: "Feature Store"
permissions: ["read", "query"]
Monitoring Configuration
# .dbt_cloud/monitoring.yml
monitoring:
enabled: true
metrics:
- name: "run_duration"
type: "histogram"
alert_threshold: "30m"
- name: "cost_per_model"
type: "gauge"
alert_threshold: 10
alerts:
- name: "Long Running Job"
condition: "run_duration > 30m"
severity: "warning"
channels:
- "slack:#data-engineering"
- "email:data-eng@company.com"
- name: "High Cost Job"
condition: "total_cost > 500"
severity: "critical"
channels:
- "slack:#data-engineering"
- "pagerduty:data-eng"
dashboards:
- name: "Job Performance"
metrics: ["run_duration", "success_rate", "cost"]
refresh: "1h"
- name: "Cost Tracking"
metrics: ["cost_per_model", "cost_per_job"]
refresh: "1d"
Performance Metrics
| Feature | Description | Impact |
|---|---|---|
| Slim CI | Selective execution | 80-90% faster |
| Caching | Result caching | 50-70% faster |
| Parallelism | Concurrent execution | 2-3x faster |
| Monitoring | Real-time insights | Proactive alerts |
| Semantic Layer | Metric consistency | Improved governance |
Best Practices
- Use Slim CI - Only test modified models
- Set up alerts - Notify on failures
- Monitor costs - Track warehouse usage
- Use environments - Separate dev/staging/prod
- Version control - All configurations in Git
- Document jobs - Clear naming and descriptions
- Test regularly - Automated quality checks
- Review logs - Monitor execution details