Skip to main content

Configuration Guide

How to configure Qarion ETL for your needs.

Configuration File​

Qarion ETL uses a config.toml file for project configuration:

[app]
name = "my_project"
type = "data_pipeline"

[engine]
type = "sqlite"
config = { path = "data.db" }

[dataset_storage]
type = "local"
config = { dataset_dir = "datasets" }

[flow_storage]
type = "local"
config = { flow_dir = "flows" }

[migration_storage]
type = "local"
config = { migration_dir = "migrations" }

Storage Configuration​

Local Storage​

Store definitions in local files:

[dataset_storage]
type = "local"
config = { dataset_dir = "datasets" }

Database Storage​

Store definitions in database:

[dataset_storage]
type = "database"
config = {
  connection_string = "sqlite:///metadata.db",
  table_name = "datasets"
}

Engine Configuration​

SQLite Engine​

[engine]
type = "sqlite"
config = { path = "data.db" }

Pandas Engine​

[engine]
type = "pandas"
config = { storage_path = "data/" }

DuckDB Engine​

[engine]
type = "duckdb"
config = { path = "data.duckdb" }

Schema Storage​

Local Schema Storage​

[schema_storage]
type = "local"
config = { migration_dir = "migrations" }

Database Schema Storage​

[schema_storage]
type = "database"
config = {
  connection_string = "sqlite:///metadata.db",
  namespace = "xt"
}

Flow Configuration​

Flow Loading​

[flow_loading]
file_patterns = ["*.toml"]
recursive = true

Namespace Configuration​

[metadata]
namespace = "xt"
default_namespace = "public"

Environment Variables​

Some configuration can be overridden with environment variables:

export QARION_ETL_ENGINE_TYPE=sqlite
export QARION_ETL_ENGINE_PATH=data.db

Validation​

Configuration is validated on load. Invalid configuration will raise errors with details about what's wrong.