Skip to main content

dbt Integration

Connect Qarion to your dbt project to import model metadata and lineage automatically.

Prerequisites​

Before connecting, ensure you have a dbt project with defined models, access to the generated artifacts (manifest.json and catalog.json), and for dbt Cloud users, an API token with appropriate read permissions.

Integration Methods​

Option A: dbt Cloud​

You can connect directly to dbt Cloud to enable automatic synchronization of your metadata and lineage.

Option B: dbt Core (Self-Hosted)​

For self-hosted setups, you can upload the artifacts generated by running dbt compile and dbt docs generate to import your data.

dbt Cloud Setup​

Step 1: Add Source System​

Navigate to Source Systems, click Add Source System, and select dbt Cloud as the platform type.

Step 2: API Configuration​

Enter your details: the Account ID, the Project ID you wish to sync, and the Environment (e.g., Production) to read from.

Step 3: Authentication​

Generate an API token in dbt Cloud (under Account Settings → API Tokens) with read-only access to the project. Copy this token into the Qarion configuration.

Step 4: Test Connection​

Click Test Connection to verify that the API token works, the project is visible, and the necessary artifacts are available.

dbt Core Setup​

Step 1: Generate Artifacts​

Run dbt compile and dbt docs generate in your project. This creates target/manifest.json (model definitions) and target/catalog.json (schema info).

Step 2: Upload Artifacts​

In Source Systems, create a dbt Core source system and upload the generated manifest.json and catalog.json files.

Step 3: Schedule Updates​

For ongoing synchronization, integrate the artifact upload into your CI/CD pipeline, using the Qarion API to push updated artifacts after each build.

What Gets Synced​

From manifest.json​

Qarion imports Models (names, descriptions, materialization), Sources, Seeds, Snapshots, Dependencies (refs), and Tests.

From catalog.json​

Qarion imports Columns (names, types, descriptions), Statistics (row counts, sizes), and Database info (schema/table names).

Lineage Mapping​

How Lineage Works​

dbt lineage is derived from ref() and source() calls within your models, as well as explicit model dependencies.

Automatic Product Matching​

Qarion automatically matches dbt models to products by database/schema/table name or by the model's unique_id. Complex cases can be mapped manually.

Manual Mapping​

For custom mappings, open the source system triggers Mappings tab to manually link specific dbt models to Qarion products.

Quality Check Integration​

dbt Tests → Qarion Checks​

You can import dbt tests—including schema tests (unique, not_null), data tests, and generic tests—as quality checks in Qarion.

Test Results Sync​

When configured, test results from your dbt runs are synced to Qarion. Failures in dbt will trigger corresponding alerts in the platform.

Configuration​

Add tags (e.g., qarion-sync) to your dbt models in dbt_project.yml to control which models are synchronized.

Advanced Configuration​

Multi-Project Setup​

For monorepos, create separate source systems for each project, configure project-specific credentials, and use tags to control the scope of each sync.

Environment Filtering​

You can choose to filter syncs by environment: syncing only Production is default, but you can include Staging or exclude Development as needed.

Incremental Sync​

For large projects, consider a weekly full sync combined with incremental syncs on each dbt run, or use delta detection to upload only changed manifests.

CI/CD Integration​

GitHub Actions Example​

Use a workflow to install dbt, generate artifacts, and upload them to Qarion via the API:

name: Sync dbt to Qarion

on:
  push:
    branches: [main]

jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'

      - name: Install dbt
        run: pip install dbt-core dbt-snowflake

      - name: Generate artifacts
        run: |
          dbt deps
          dbt compile
          dbt docs generate

      - name: Upload to Qarion
        run: |
          curl -X POST "$QARION_API/source-systems/$SOURCE_ID/sync" \
            -H "Authorization: Bearer $QARION_TOKEN" \
            -F "manifest=@target/manifest.json" \
            -F "catalog=@target/catalog.json"
        env:
          QARION_API: ${{ secrets.QARION_API }}
          QARION_TOKEN: ${{ secrets.QARION_TOKEN }}
          SOURCE_ID: ${{ secrets.DBT_SOURCE_ID }}

Troubleshooting​

Connection Errors​

"API token invalid" suggests you need to regenerate the token or check its permissions. "Project not found" often means an incorrect Project ID or missing environment.

Sync Issues​

Missing models can happen if the model is disabled or filtered out by tags. regenerate the manifest to fix. Stale lineage usually requires an artifact regeneration or manual sync.

Mapping Issues​

If models not matching products, verify that naming conventions align or use manual mapping to resolve discrepancies.

Best Practices​

Artifact Management​

Version control your dbt project and generate artifacts within your CI/CD pipeline, archiving them for audit trails.

Sync Frequency​

Sync on every production deployment, perform a weekly full refresh, or use webhooks for real-time updates.

Documentation Quality​

Maintain high documentation quality by populating descriptions in dbt models and keeping column descriptions updated in your schema files.