Skip to main content

dbt Workflow

Sable uses dbt (Data Build Tool) for all data transformations. This guide covers common dbt operations.

Setup​

cd sable-data/dbt
source ../.venv/bin/activate

# Set database password
export SUPABASE_DB_PASSWORD='<from infisical>'

# Verify connection
dbt debug

Running Models​

Full Pipeline​

# Run all models from silver onwards
dbt run --select pnl_silver+

Specific Models​

# Single model
dbt run --select gold.pnl_daily_dbt

# Model and all downstream
dbt run --select gold.v_nav_daily+

# Model and all upstream
dbt run --select +gold.v_returns_comparison

With Variables​

# Process specific job
dbt run --select pnl_silver+ --vars '{"process_job_id": "uuid-here"}'

Testing​

Run All Tests​

dbt test

Observability Tests​

dbt test --select tag:observability

Specific Test​

dbt test --select obs_01_data_freshness

Model Selection Syntax​

SyntaxMeaning
model_nameJust this model
model_name+Model and all downstream
+model_nameModel and all upstream
+model_name+Model and all connected
tag:observabilityAll models with this tag
path:models/goldAll models in directory

Incremental Models​

Most Sable models are incremental:

{{ config(
    materialized='incremental',
    unique_key=['org_id', 'account_code', 'symbol', 'pnl_date']
) }}

SELECT ...
FROM source_table
{% if is_incremental() %}
WHERE process_job_id = '{{ var("process_job_id") }}'
{% endif %}

Full Refresh​

Force full rebuild (drops and recreates):

dbt run --select model_name --full-refresh

Deploying Functions​

SQL functions are deployed via macros:

# Deploy specific function
dbt run-operation deploy_f_get_nav_daily

# Deploy all functions
dbt run-operation deploy_all_functions

Or via MCPammer:

mcpammer_sql_function_deploy(function_name='f_get_nav_daily')

Troubleshooting​

Connection Issues​

# Test connection
dbt debug

# Check profiles.yml
cat ~/.dbt/profiles.yml

Model Failures​

# See compiled SQL
cat target/compiled/sable/models/gold/model_name.sql

# See run logs
cat logs/dbt.log

Stale Data​

# Check source freshness
dbt source freshness

# Force full refresh
dbt run --select model_name --full-refresh