Skip to main content

Qarion ETL Tutorials

Step-by-step tutorials for using Qarion ETL.

Tutorial 1: Building Your First Change Feed​

This tutorial walks you through creating a change feed flow to track changes in customer data.

Step 1: Initialize Project​

qarion-etl init --project-name my_project
cd my_project

Step 2: Create Flow Definition​

Create flows/customers_change_feed.toml:

id = "customers_change_feed"
name = "Customers Change Feed"
flow_type = "change_feed"
namespace = "raw"

[input]
primary_key = ["customer_id"]
columns = ["customer_id", "name", "email", "status", "updated_at"]

[properties.load]
source_path = "data/customers"
file_pattern = "customers_*.csv"

Step 3: Generate Datasets​

qarion-etl generate-docs

This creates dataset definitions in datasets/.

Step 4: Generate Code​

qarion-etl generate-code --format sql --flow customers_change_feed --output-dir output

Step 5: Build and Apply Migrations​

# Generate datasets and migrations
qarion-etl build

# Apply migrations to create tables
qarion-etl apply-migrations

Tutorial 2: Financial Transaction Processing​

Build a delta publishing flow for financial transactions.

Step 1: Create Flow​

Create flows/transactions_delta.toml:

id = "transactions_delta"
name = "Transactions Delta Publishing"
flow_type = "delta_publishing"

[input]
primary_key = ["transaction_id"]
columns = ["transaction_id", "account_id", "amount", "transaction_date", "type"]

[properties]
namespace = "finance"

Step 2: Generate DBT Code​

qarion-etl generate-code --format dbt --flow transactions_delta --output-dir dbt_project --dialect postgres

Step 3: Review Generated Code​

Check the generated DBT models in dbt_project/models/.

Tutorial 3: User Session Analysis​

Create a sessionization flow for web analytics.

Step 1: Create Flow​

Create flows/user_sessions.toml:

id = "user_sessions"
name = "User Sessionization"
flow_type = "sessionization"

[input]
primary_key = ["event_id"]
columns = ["event_id", "user_id", "event_time", "event_type", "page_url"]

[properties]
session_timeout_minutes = 30

Step 2: Generate Code​

qarion-etl generate-code --format dbt --flow user_sessions --output-dir dbt_project --dialect postgres

Step 3: Review Generated Code​

Check the generated DBT models in dbt_project/models/.