Qarion ETL User Documentation

Welcome to the Qarion ETL user documentation! This documentation is for users who want to use Qarion ETL to build data transformation pipelines.

Note: This is user documentation for using Qarion ETL. For developer documentation on extending Qarion ETL, see docs/README.md.

📚 Documentation Structure

Core Concepts - Understanding Qarion ETL fundamentals
Flows Guide - Using flows to build pipelines
Data Ingestion - Loading data from external sources into datasets
Database Ingestion - Ingesting data directly from databases using SQL queries
Data Contracts - Data contract validation and schema enforcement
Quality Check Flow - Quality Check Flow and Quality Check Suites
Data Quality - Data quality validation system
Task System - Understanding task types and the plugin architecture
Triggers - Flow triggers and automation
Migrations Guide - Understanding and working with migrations
Engines and Storage - Engine and storage layer configuration
Data Export - Exporting data from tables to files
Export Flow Reference - Detailed export flow documentation
XCom: Inter-Task Data Exchange - Exchanging data between tasks
Flow Trigger Tasks - Triggering other flows from within a flow
API Call Tasks - Making HTTP API calls with authentication and XCom integration
Credential Management - Managing credentials securely
Metadata Tracking - Tracking and monitoring all operations and executions
Configuration - Configuring Qarion ETL

Examples - Code examples and use cases
Tutorials - Step-by-step tutorials
Complete Examples Reference - Comprehensive TOML configuration examples for all features

For a complete index, see INDEX.md.

Developer Documentation: docs/README.md - For developers extending Qarion ETL
Generated Documentation: Run qarion-etl generate-docs to create project-specific documentation