Pipelines and Orchestration
The Pipelines view gives Qarion users visibility into data orchestration from Airflow, Luigi, Dagster, and Prefect. It connects operational pipeline health with catalog governance, quality checks, ownership, and incident response.
What Pipelines Show
Pipelines represent orchestrated jobs such as Airflow DAGs, Dagster jobs, Luigi tasks, and Prefect deployments. A pipeline detail view shows recent runs, task-level status, timing, raw provider metadata, related products, and annotations from team members.
Use Pipelines to answer questions like:
- Which pipeline refreshed this product?
- When did the latest run start and finish?
- Which task failed, retried, or ran longer than usual?
- Who owns this pipeline and how critical is it?
- Which runs should be watched by the current user?
Connector Sync
Create an orchestration connector to pull pipeline metadata into Qarion. A connector sync imports pipeline definitions, run history, task runs, timing metadata, status, and product relationships where available.
Prefect connectors use the Prefect v3 REST API. Set the API URL to a Prefect
server API root such as http://localhost:4200/api, or to a Prefect Cloud
workspace API URL. Self-hosted Prefect can use no authentication; Prefect Cloud
and API-key deployments use bearer authentication. Optional sync filters can
limit imported deployments and tags, and optional public_base_url enables links
back to the Prefect UI.
The Pipelines page can filter by provider, connector, status, owner, team, criticality, search text, watched-only state, and recent run window.
Runs and Tasks
Pipeline runs provide a timeline of executions. Open a run to inspect status, start/end time, duration, run type, provider identifiers, and task runs. Task runs help isolate where a pipeline failed or slowed down.
Comments can be added to runs and task runs to record investigation notes, handoffs, incident context, or remediation decisions.
Ownership and Criticality
Assign a team to each pipeline so operational ownership is visible. Criticality classifies the business impact of failures and helps prioritize response:
| Criticality | Use for |
|---|---|
| Low | Informational or low-impact jobs |
| Medium | Useful pipelines with limited downstream impact |
| High | Important production jobs |
| Critical | Business-critical or regulated pipelines |
Manual Triggers
Airflow, Dagster, and Prefect pipelines can be triggered from Qarion when the
connector and provider support it. For Prefect, Qarion calls the deployment
create_flow_run API with Qarion labels and an idempotency key, then records the
queued flow run in Pipelines. Manual triggers are useful for reruns, demos,
validation, and operator workflows where users should not need to leave Qarion to
start a known pipeline.
Related Documentation
- Airflow Integration — Developer tutorial for DAG integration
- Source Systems — Manage connector credentials and syncs
- Data Lineage — Understand upstream and downstream dependencies
- Quality Management — Connect pipeline checks to quality monitoring