Skip to main content

Qarion ETL Examples

Code examples and use cases for Qarion ETL.

Quick Examples​

Change Feed Example​

Track changes to customer data:

# flows/customers_change_feed.toml
id = "customers_change_feed"
name = "Customers Change Feed"
flow_type = "change_feed"

[input]
primary_key = ["customer_id"]
columns = ["customer_id", "name", "email", "status", "updated_at"]

[properties.load]
source_path = "data/customers"
file_pattern = "customers_*.csv"

Delta Publishing Example​

Process financial transactions:

# flows/transactions_delta.toml
id = "transactions_delta"
name = "Transactions Delta Publishing"
flow_type = "delta_publishing"

[input]
primary_key = ["transaction_id"]
columns = ["transaction_id", "account_id", "amount", "transaction_date", "type"]

[properties]
namespace = "finance"

Sessionization Example​

Group user events into sessions:

# flows/user_sessions.toml
id = "user_sessions"
name = "User Sessionization"
flow_type = "sessionization"

[input]
primary_key = ["event_id"]
columns = ["event_id", "user_id", "event_time", "event_type", "page_url"]

[properties]
session_timeout_minutes = 30

Code Generation Examples​

Generate SQL​

qarion-etl generate-code --format sql --flow customers_change_feed --output-dir output

Generate DBT​

qarion-etl generate-code --format dbt --flow transactions_delta --output-dir dbt_project --dialect postgres

Generate Both SQL and Python​

qarion-etl generate-code --format both --flow user_sessions --output-dir output

Complete Tutorials​

See Tutorials for step-by-step guides.