Pipelines and Orchestration
The Pipelines view gives Qarion users visibility into Airflow-backed data orchestration. It connects operational pipeline health with catalog governance, quality checks, ownership, and incident response.
What Pipelines Show
Pipelines represent orchestrated jobs such as Airflow DAGs. A pipeline detail view shows recent runs, task-level status, timing, raw provider metadata, related products, and annotations from team members.
Use Pipelines to answer questions like:
- Which DAG refreshed this product?
- When did the latest run start and finish?
- Which task failed, retried, or ran longer than usual?
- Who owns this pipeline and how critical is it?
- Which runs should be watched by the current user?
Airflow Connector Sync
Create an Airflow connector under a source system to pull pipeline metadata into Qarion. A connector sync imports DAGs, run history, task runs, timing metadata, status, and product relationships where available.
The Pipelines page can filter by provider, connector, status, owner, team, criticality, search text, watched-only state, and recent run window.
Runs and Tasks
Pipeline runs provide a timeline of executions. Open a run to inspect status, start/end time, duration, run type, provider identifiers, and task runs. Task runs help isolate where a pipeline failed or slowed down.
Comments can be added to runs and task runs to record investigation notes, handoffs, incident context, or remediation decisions.
Ownership and Criticality
Assign a team to each pipeline so operational ownership is visible. Criticality classifies the business impact of failures and helps prioritize response:
| Criticality | Use for |
|---|---|
| Low | Informational or low-impact jobs |
| Medium | Useful pipelines with limited downstream impact |
| High | Important production jobs |
| Critical | Business-critical or regulated pipelines |
Manual Triggers
Airflow-backed pipelines can be triggered from Qarion when the connector and provider support it. Manual triggers are useful for reruns, demos, validation, and operator workflows where users should not need to leave Qarion to start a known DAG.
Related Documentation
- Airflow Integration — Developer tutorial for DAG integration
- Source Systems — Manage connector credentials and syncs
- Data Lineage — Understand upstream and downstream dependencies
- Quality Management — Connect pipeline checks to quality monitoring