Orchestration Visibility
Qarion can mirror Airflow pipeline metadata into the platform so data teams can review DAG health, run history, task status, ownership, and annotations without leaving the governance workspace.
Recommended Flow
- Create a source system for the orchestrator.
- Add an Airflow connector with the base URL and credentials required by your deployment.
- Run connector sync to import DAGs, recent runs, task runs, provider metadata, and product relationships where available.
- Assign Qarion teams and criticality to important pipelines.
- Use run and task comments to capture investigation notes and handoffs.
- Trigger supported Airflow pipelines from Qarion for reruns or validation.
Data Model
The orchestration surface separates provider objects from Qarion governance metadata:
| Object | Purpose |
|---|---|
| Pipeline | Provider DAG or equivalent orchestrated job. |
| Pipeline run | One execution of a pipeline. |
| Task run | One task execution within a pipeline run. |
| Connector | The configured Airflow integration used for sync. |
| Comments | Qarion annotations on runs or task runs. |
Provider identifiers remain available for traceability. Qarion-owned metadata such as team assignment, criticality, watched state, and comments lets governance teams layer operational ownership on top of Airflow data.
Operational Notes
- Orchestration endpoints are space scoped.
- Sync and trigger actions are provider backed; v1 supports Airflow.
- Pipeline and run list endpoints return pagination headers.
- Connector sync requires permission to edit the connector.
- Triggering a pipeline returns provider-specific errors when Airflow rejects or cannot complete the request.
Related
- Orchestration API — Endpoint reference
- Airflow Integration — DAG examples and quality-gate patterns
- Connectors API — Connector management