Orchestration Visibility
Qarion can mirror orchestration metadata from Airflow, Luigi, Dagster, and Prefect into the platform so data teams can review pipeline health, run history, task status, ownership, and annotations without leaving the governance workspace.
Recommended Flow
- Add an orchestration connector for Airflow, Luigi, Dagster, or Prefect with the base URL and credentials required by your deployment.
- Run connector sync to import pipelines, recent runs, task runs, provider metadata, and product relationships where available.
- Assign Qarion teams and criticality to important pipelines.
- Use run and task comments to capture investigation notes and handoffs.
- Trigger supported Airflow, Dagster, and Prefect pipelines from Qarion for reruns or validation.
Data Model
The orchestration surface separates provider objects from Qarion governance metadata:
| Object | Purpose |
|---|---|
| Pipeline | Provider DAG, job, task graph, deployment, or equivalent orchestrated job. |
| Pipeline run | One execution of a pipeline. |
| Task run | One task execution within a pipeline run. |
| Connector | The configured provider integration used for sync. |
| Comments | Qarion annotations on runs or task runs. |
Provider identifiers remain available for traceability. Qarion-owned metadata such as team assignment, criticality, watched state, and comments lets governance teams layer operational ownership on top of provider data.
Operational Notes
- Orchestration endpoints are space scoped.
- Sync is supported for Airflow, Luigi, Dagster, and Prefect connectors.
- Manual trigger is supported for Airflow, Dagster, and Prefect pipelines.
- Pipeline and run list endpoints return pagination headers.
- Connector sync requires permission to edit the connector.
- Triggering a pipeline returns provider-specific errors when the orchestrator rejects or cannot complete the request.
Related
- Orchestration API — Endpoint reference
- Airflow Integration — DAG examples and quality-gate patterns
- Connectors API — Connector management