Skip to main content

Pipeline Authoring

Pipeline Authoring lets teams draft, validate, and publish pipeline definitions from a governed Qarion workspace. It is designed for repository-backed files, AI-assisted code generation, validation, and review before pipeline code is published.

Workspace Setup​

  1. Open Authoring -> Pipeline Authoring.
  2. Create or select a workspace.
  3. Choose the default repository and branch.
  4. Set workspace visibility to Space, Private, or Shared.
  5. Select authoring query connectors when previews or generated code need governed data access.

Shared workspaces can grant users and teams view or manage access.

Authoring Targets​

Pipeline Authoring supports Airflow and Dagster targets. Airflow remains the default for existing drafts and continues to publish generated DAG files under dags/. Dagster drafts generate Definitions, @job, and @op based Python artifacts under orchestrators/dagster/.

Dagster support is compile-and-validate focused in this release. Generic executable nodes such as marker, bash, Python callable, dbt, and SQLMesh nodes are supported. Qarion Airflow provider nodes for quality pushes and contract validation remain Airflow-only until a Dagster or SDK-backed runtime adapter is configured.

Working With Files​

The workspace file navigator supports creating, renaming, moving, searching, and deleting draft files. Python editing includes repository-aware autocomplete when the backend can inspect the current draft and related package context.

Keep generated and edited files reviewable. Pipeline Authoring is most useful when drafts can be validated, audited, and published through the same workspace rather than copied manually into a separate repository.

Governed Agent Flow​

AI-assisted implementation follows a review-first lifecycle:

  1. Qarion asks for missing requirements, such as file format, runtime, connector, dependency, or data residence details.
  2. The agent proposes a bounded decomposition for review.
  3. Approved work seeds a task checklist with required tasks, dependencies, evidence refs, and action refs.
  4. The agent prepares draft files, support files, and tests inside the approved scope.
  5. Validation, generated-code checks, sandbox checks, and diff review run before files are saved.

Required checklist tasks must be completed with evidence before Qarion applies mutating changes. If a task is blocked or missing evidence, the run pauses for a repair, clarification, or revised plan. Resume checks compare the current draft, approved plan, checklist, repo-edit session, and validation tier before the agent continues, so stale runs cannot silently apply older work.

Reviewing Generated Changes​

Plan reviews can ask you to:

  • Implement an approved plan when generated files and checks are ready.
  • Revise the plan with notes when the scope or approach is wrong.
  • Cancel without changing the draft.
  • Save files after failed generated-file validation, selecting only the preview files you want to keep.
  • Approve repository commands when a generated workflow needs an allowed command such as a project-specific validator.

Generated-file reviews are bounded to the previewed paths. Stale reviews are rejected so an older response cannot overwrite newer draft changes.

Generation Readiness​

Plan reviews include a generation-readiness summary when Qarion has enough validation evidence. Treat it as the gate between "draft proposed" and "safe to apply":

StatusMeaning
ReadyGenerated files, support files, support tests, validation, and sandbox/dependency checks passed or were cleanly skipped.
BlockedA required support file, support test, checklist item, validation section, or sandbox section is missing.
Failed validationGenerated files were produced, but validation found issues that need repair or selective save.

When readiness is blocked, use the next action shown in the review before retrying implementation. Do not approve repository commands or save failed files unless the command, path list, and diagnostics match the change you expect.

Validation, Commands, And Publication​

Use validation before publication to catch generated-code and dependency issues. Publication should target the configured repository and preserve the review trail in Qarion.

Repository command execution is disabled by default. When administrators enable local command execution, commands are policy checked before execution and repository-code checks such as tests, package scripts, builds, custom validators, and repository tool profiles require approval before they run. Commands with network intent require the configured authoring network policy, and approvals are bound to the exact request shown in the review.

Generated-code validation can include static checks, generated support tests, dependency smoke checks, generated-code sandbox execution, and database sandbox checks for SQL or database-facing support files. Dedicated code-writer routing, sandbox validation, database sandbox validation, dependency smoke checks, performance profiling, and validation package fetch are rollout controls. Administrators can inspect their current state in AI Ops.

For local Docker demo validation, the demo can preconfigure the Qarion base URL used by validation runners with DEMO_VALIDATION_QARION_BASE_URL.

Troubleshooting​

The plan cannot be implemented means a required checklist item is still open, blocked, or missing evidence. Review the task checklist and answer any pending clarification before retrying.

Generated files failed validation means Qarion could not safely apply the full generated set. Use the failed-file review to save only the preview files you trust, then continue from the updated draft.

Generation readiness is blocked means the review is missing required support files, support tests, sandbox results, dependency evidence, or validation evidence. Follow the review's next action rather than bypassing the gate.

A review became stale means files, checklist state, or the approved plan changed after the review was created. Rerun the agent or validation so the review reflects the current draft.

A repository command needs approval means the agent requested a command outside normal file edits. Approve only commands whose purpose, arguments, and network behavior match the change you expect.

Package or dependency checks are blocked means validation package fetch, private package repository access, public-index access, or the sandbox runtime is not ready. Ask an administrator to review the Pipeline Authoring readiness checks in AI Ops.

Reliability or run trace warnings appear in AI Ops means recent authoring runs may be missing checkpoint, validation-tier, tool-policy, prompt/cache, or memory evidence. Use the linked workflow run before treating the issue as a normal generation failure.