Skip to main content

CLI Reference

Complete reference for Qarion ETL command-line interface.

Commands​

qarion-etl new-project​

Create a new Qarion ETL project in a new directory.

qarion-etl new-project PROJECT_PATH [OPTIONS]

Required Arguments:

  • PROJECT_PATH: Path to the new project directory (will be created if it doesn't exist)

Options:

  • --project-name: Project name (defaults to folder name)
  • --init-db: Initialize the database after creating the project
  • --with-examples: Include example datasets and flows for each flow type (default: clean project)

Creates:

  • qarion-etl.toml - Project configuration with Fernet key
  • data/ - Data directory
  • datasets/ - Dataset definitions directory
  • flows/ - Flow definitions directory
  • migrations/ - Migration files directory
  • plugins/ - Plugin directory

Examples:

# Create a clean project (default)
qarion-etl new-project my_project

# Create a project with examples
qarion-etl new-project my_project --with-examples

# Create and initialize database
qarion-etl new-project my_project --init-db

# Create with examples and initialize database
qarion-etl new-project my_project --with-examples --init-db

Clean Project (Default): Creates only the directory structure and configuration file. No example files are created. Use this for production projects or when you want to start from scratch.

Example Project (--with-examples): Creates example datasets and flows for each available flow type. Use this to explore Qarion ETL features and learn the structure.

qarion-etl init​

Initialize an existing Qarion ETL project (creates configuration file and directories if they don't exist).

qarion-etl init [--conf CONFIG_PATH]

Options:

  • --conf: Path to the config file (default: qarion-etl.toml)

Creates:

  • qarion-etl.toml - Project configuration (if it doesn't exist)
  • datasets/ - Dataset definitions directory
  • flows/ - Flow definitions directory
  • migrations/ - Migration files directory
  • Initializes the metadata database

qarion-etl generate-code​

Generate code from flows.

qarion-etl generate-code --format FORMAT --flow FLOW_ID --output-dir OUTPUT [OPTIONS]

Required Options:

  • --format: Output format (sql, python, both, dbt, airflow)
  • --flow: Flow ID to generate code for
  • --output-dir: Output directory

Optional Options:

  • --dialect: SQL dialect (for DBT: sqlite, postgres, mysql, snowflake, redshift, bigquery)
  • --overwrite: Overwrite existing files

Examples:

# Generate SQL
qarion-etl generate-code --format sql --flow my_flow --output-dir output

# Generate DBT with PostgreSQL
qarion-etl generate-code --format dbt --flow my_flow --output-dir dbt_project --dialect postgres

# Generate both SQL and Python
qarion-etl generate-code --format both --flow my_flow --output-dir output

# Generate Airflow DAG
qarion-etl generate-code --format airflow --flow my_flow --output-dir dags

qarion-etl generate-docs​

Generate project documentation site.

qarion-etl generate-docs [--output-dir OUTPUT_DIR] [--overwrite] [--multi-page]

Options:

  • --output-dir: Output directory (default: docs_generated)
  • --overwrite: Overwrite existing files
  • --multi-page: Generate multi-page instead of single-page application (SPA)

Example:

qarion-etl generate-docs --output-dir docs_generated

qarion-etl run​

Run Qarion ETL transactions (starts a transaction run).

qarion-etl run [--conf CONFIG_PATH]

Options:

  • --conf: Path to configuration file (default: qarion-etl.toml)

Example:

qarion-etl run
qarion-etl run --conf /path/to/config.toml

qarion-etl build​

Build a project by generating datasets from flows and migrations from datasets.

qarion-etl build [--flow FLOW_ID] [--skip-datasets] [--skip-migrations] [--overwrite]

Options:

  • --flow: Specific flow ID to process (processes all flows if not provided)
  • --skip-datasets: Skip dataset generation step
  • --skip-migrations: Skip migration generation step
  • --overwrite: Automatically overwrite existing files without prompting

Example:

qarion-etl build
qarion-etl build --flow my_flow --overwrite

qarion-etl generate-migrations​

Generate migration files from dataset definitions.

qarion-etl generate-migrations [--conf CONFIG_PATH]

Example:

qarion-etl generate-migrations

qarion-etl apply-migrations​

Apply migration files to the database.

qarion-etl apply-migrations [--dry-run] [--conf CONFIG_PATH]

Options:

  • --dry-run: Validate migrations without executing them

Example:

qarion-etl apply-migrations
qarion-etl apply-migrations --dry-run

qarion-etl new-flow​

Create a new flow definition file.

qarion-etl new-flow --flow-type TYPE --name NAME [--flow-id ID] [--conf CONFIG_PATH]

Options:

  • --flow-type: Flow type (change_feed, delta_publishing, sessionization)
  • --name: Flow name
  • --flow-id: Flow ID (defaults to generated from name)

Example:

qarion-etl new-flow --flow-type change_feed --name "Orders Change Feed"

qarion-etl new-dataset​

Create a new dataset definition file.

qarion-etl new-dataset --name NAME [--namespace NAMESPACE] [--conf CONFIG_PATH]

Options:

  • --name: Dataset name
  • --namespace: Dataset namespace (default: public)

Example:

qarion-etl new-dataset --name orders --namespace raw

qarion-etl list-flows​

List all flows in the project.

qarion-etl list-flows [--format FORMAT] [--conf CONFIG_PATH]

Options:

  • --format: Output format (table, json) (default: table)

Example:

qarion-etl list-flows
qarion-etl list-flows --format json

qarion-etl list-plugins​

List available plugins.

qarion-etl list-plugins [--plugin-type TYPE] [--format FORMAT]

Options:

  • --plugin-type: Filter by plugin type (flow, engine, code_generator, repository)
  • --format: Output format (table, json) (default: table)

Example:

qarion-etl list-plugins
qarion-etl list-plugins --plugin-type flow

qarion-etl credentials​

Manage credentials in the credential store.

qarion-etl credentials store​

Store a credential in the credential store.

qarion-etl credentials store --credential-id ID [OPTIONS]

Required Options:

  • --credential-id: Unique identifier for the credential

Options:

  • --name: Credential name
  • --type: Credential type (aws, database, api_key, oauth, basic_auth, custom)
  • --description: Credential description
  • --file: Path to JSON file containing credential data
  • --json: JSON string containing credential data
  • --interactive / -i: Interactively prompt for credential data

Examples:

# Store from JSON file
qarion-etl credentials store --credential-id aws_prod \
  --name "AWS Production" \
  --type aws \
  --file credentials.json

# Store from JSON string
qarion-etl credentials store --credential-id db_prod \
  --json '{"host":"db.example.com","user":"admin","password":"secret"}'

# Interactive mode
qarion-etl credentials store --credential-id api_key --interactive

qarion-etl credentials get​

Get a credential from the credential store.

qarion-etl credentials get --credential-id ID [OPTIONS]

Required Options:

  • --credential-id: Credential ID to retrieve

Options:

  • --show-value: Show credential value (default: only verify existence)
  • --format: Output format (json, table, env)

Examples:

# Verify credential exists
qarion-etl credentials get --credential-id aws_prod

# Show credential value as JSON
qarion-etl credentials get --credential-id aws_prod --show-value --format json

# Show as environment variables
qarion-etl credentials get --credential-id aws_prod --show-value --format env

qarion-etl credentials list​

List all credentials in the credential store.

qarion-etl credentials list [OPTIONS]

Options:

  • --format: Output format (table, json, csv)

Examples:

# List as table (default)
qarion-etl credentials list

# List as JSON
qarion-etl credentials list --format json

# List as CSV
qarion-etl credentials list --format csv

qarion-etl credentials delete​

Delete a credential from the credential store.

qarion-etl credentials delete --credential-id ID [OPTIONS]

Required Options:

  • --credential-id: Credential ID to delete

Options:

  • --force / -f: Skip confirmation prompt

Examples:

# Delete with confirmation
qarion-etl credentials delete --credential-id old_cred

# Delete without confirmation
qarion-etl credentials delete --credential-id old_cred --force

qarion-etl credentials show​

Show credential definition and metadata (without sensitive data).

qarion-etl credentials show --credential-id ID

Required Options:

  • --credential-id: Credential ID to show

Example:

qarion-etl credentials show --credential-id aws_prod

qarion-etl trigger​

Trigger a flow execution.

qarion-etl trigger --flow-id FLOW_ID [OPTIONS]

Required Options:

  • --flow-id: Flow ID to trigger

Optional Options:

  • --conf: Path to configuration file (default: qarion-etl.toml)
  • --trigger-id: Specific trigger ID (optional)
  • --batch-id: Batch ID for execution (default: 1)
  • --force: Force execution even if conditions not met
  • --var: Variable to pass to flow (format: key=value). Can be used multiple times.

Examples:

# Trigger a flow with default batch ID
qarion-etl trigger --flow-id my_flow

# Trigger with specific batch ID
qarion-etl trigger --flow-id my_flow --batch-id 5

# Trigger with variables
qarion-etl trigger --flow-id my_flow --var env=production --var region=us-east-1

# Force execution
qarion-etl trigger --flow-id my_flow --force

Note: All flow executions are automatically tracked in metadata tables. See Metadata Tracking for details.

qarion-etl validate-config​

Validate configuration files for format and logic correctness.

qarion-etl validate-config [--conf CONFIG_PATH]

Options:

  • --conf: Path to configuration file (default: qarion-etl.toml)

What It Validates:

  1. Schema Validation: Ensures configuration matches expected JSON schema
  2. SQL Parsing: Validates that SQL queries in flow definitions can be parsed
  3. SQL Conversion: Verifies SQL queries can be converted to TransformationInstructions
  4. Non-SQL Engine Compatibility: Checks if queries are compatible with non-SQL engines (pandas, polars, pyspark)

Example:

# Validate default config file
qarion-etl validate-config

# Validate specific config file
qarion-etl validate-config --conf /path/to/config.toml

Output:

  • Reports any schema validation errors
  • Lists SQL queries that cannot be parsed
  • Warns about SQL features that may not work with non-SQL engines
  • Provides detailed error messages for debugging

Use Cases:

  • Validate configuration before deployment
  • Check SQL compatibility before running flows
  • Ensure all SQL queries are parseable
  • Verify non-SQL engine compatibility

qarion-etl validate-dbt​

Validate generated DBT model files.

qarion-etl validate-dbt [PATH] [--incremental] [--check-dependencies] [--check-sources]

Options:

  • PATH: File or directory to validate (default: current directory)
  • --incremental: Validate incremental model logic
  • --check-dependencies: Check model dependencies
  • --check-sources: Check source definitions

Example:

qarion-etl validate-dbt dbt_project/models/ --incremental --check-dependencies

Configuration​

CLI commands use the project's qarion-etl.toml file by default. Specify a different config with:

qarion-etl COMMAND --conf /path/to/config.toml

You can also set the config path via environment variable:

export XTRANSACT_CONFIG_PATH=/path/to/config.toml
qarion-etl COMMAND