Pipeline Authoring

Pipeline Authoring lets teams draft, validate, and publish pipeline definitions from a governed Qarion workspace. It is designed for repository-backed files, AI-assisted code generation, validation, and review before pipeline code is published.

Workspace Setup

1. Open Authoring -> Pipeline Authoring.
2. Create or select a workspace.
3. Choose the default repository and branch.
4. Set workspace visibility to Space, Private, or Shared.
5. Select authoring query connectors when previews or generated code need governed data access.

Shared workspaces can grant users and teams view or manage access.

Authoring Targets

Pipeline Authoring supports Airflow and Dagster targets. Airflow remains the default for existing drafts and continues to publish generated DAG files under dags/. Dagster drafts generate Definitions, @job, and @op based Python artifacts under orchestrators/dagster/.

Dagster support is compile-and-validate focused in this release. Generic executable nodes such as marker, bash, Python callable, dbt, and SQLMesh nodes are supported. Qarion Airflow provider nodes for quality pushes and contract validation remain Airflow-only until a Dagster or SDK-backed runtime adapter is configured.

Managed API Connections

Create one API connector for a REST, SOAP, or GraphQL service, then attach a compatible Qarion credential. API connectors store only non-secret connection settings, including the protocol, HTTPS base URL, authentication mode, safe headers, timeout and retry policy, pagination, and contract source. Secrets remain in Qarion credentials. Supported authentication modes are no authentication, basic authentication, API key, bearer token, and OAuth 2.0 client credentials.

Each connector owns an immutable normalized contract snapshot. REST contracts can come from OpenAPI, SOAP contracts from WSDL/XSD, and GraphQL contracts from SDL or introspection. A definition can be entered manually, selected from a repository file, or explicitly fetched from an approved remote HTTPS URL. Qarion never performs a hidden contract refresh while authoring. Use Refresh contract or Test connection when you want network access; a failed refresh leaves the last successful snapshot active.

When a pipeline uses an API operation, Pipeline Authoring asks for the connector, compatible credential, operation, request and response mapping, timeout/retry behavior, and pagination. Protocol-specific questions cover REST methods and paths, SOAP service/binding/message details, or GraphQL documents, variables, and response paths. Only connectors and credentials available in the current space are offered.

Review any difference between a connector setting and its credential before generation. Qarion does not silently resolve those conflicts. It also blocks publication when the active contract has changed, the selected operation no longer exists, a credential is missing or incompatible, or a security standard is unresolved. DELETE and operations identified as destructive require a typed confirmation bound to the connector, contract hash, operation, and affected scope; changing any of those values invalidates the confirmation.

Generated Airflow and Dagster code receives managed resource references and a reviewed operation descriptor, never credential values or the complete contract. Qarion's runtime adapter enforces TLS, timeouts, retry and pagination limits, redaction, schema checks, and OAuth token handling. Airflow uses managed Connections and Variables. Dagster publication requires a deployment adapter that verifies environment-backed secret and configuration bindings before the pipeline can run.

Packaged Airflow DAGs

Every successful Airflow publication also creates an immutable ZIP from the exact validated definition version. Qarion discovers the nearest requirements.txt or pyproject.toml by walking from the DAG directory toward the repository root, resolves draft workspace edits and deletions, and vendors the hash-locked wheel closure alongside the DAG and its transitively imported local modules. The ZIP is stored as a deployment artifact; it is not committed to the publication branch or merge request.

The default airflow_package_mode is extracted. Extract this archive into the Airflow DAG runtime so pure-Python and platform-matched native wheels are available as normal imports. zipimport is opt-in and is intended for placing the ZIP directly in DAGS_FOLDER; Qarion rejects that mode when a wheel contains a native extension or other content that Python cannot import directly from a ZIP. The package also contains deterministic .qarion/deployment.json evidence, the selected manifest, local package __init__.py files, and explicitly referenced runtime files. Tests, caches, bytecode, VCS metadata, secrets, unsafe paths, and unrelated repository files are excluded.

Extracted packages preserve repository-relative paths. Direct ZIP-import packages place the compiled DAG at the ZIP root for Airflow discovery while keeping local modules and vendored pure-Python dependencies importable below it.

The Publish inspector shows the selected mode and manifest, dependency-lock identity, checksum, size, compatibility failures, and an authorized download action. A packaging failure occurs before Qarion creates or mutates a Git branch. Dagster publication is unchanged.

Working With Files

The workspace file navigator supports creating, renaming, moving, searching, and deleting draft files. Python editing includes repository-aware autocomplete when the backend can inspect the current draft and related package context.

Keep generated and edited files reviewable. Pipeline Authoring is most useful when drafts can be validated, audited, and published through the same workspace rather than copied manually into a separate repository.

Editing The DAG Graph

The Graph inspector shows the planned execution DAG as stages, nodes, and dependencies. Users with manage access can make direct graph edits from the inspector:

• Select a node to view its canonical payload, upstream tasks, and downstream tasks.
• Add a node from the node catalog examples, then edit the canonical JSON payload before saving.
• Update or remove an existing node. Removing a node also removes dependencies that point into or out of it.
• Add or remove dependencies between existing nodes.
• Load, edit, save, or delete the DAG README. The README is backed by the same draft-file override/deletion mechanism as regular files, but appears as a first-class DAG resource.

Direct graph edits append a new definition version and run full spec validation. They are disabled while reviewing a plan preview so an agent-generated review cannot be mixed with manual edits mid-review. If you need the agent to perform a graph change, ask for the change in chat: the agent uses planner tools that prepare reviewed node, dependency, and README actions rather than writing directly.

README paths resolve from the explicit README path when one is set, otherwise from the DAG or artifact path with an .md suffix, otherwise README.md.

Working With Chat Context

Type @ in the composer to search workspace files and attach an exact file reference to the message. Qarion validates each reference against the current workspace and draft before the message is accepted, expands valid references into file context for the agent, and limits a message to 20 file mentions. Renamed, deleted, inaccessible, malformed, or stale references are rejected so the agent does not silently work from the wrong file.

Historical messages with an older malformed mention remain readable. Qarion leaves that stored text unchanged instead of making the conversation fail to load.

Assistant rationale, plan-review controls, recovery actions, run-summary provenance, and other artifacts are grouped under Show details on a completed assistant message. Expand it when you need the evidence or action controls; collapse it to keep long conversations scannable. The chat keeps your reading position when older content or details expand and follows new activity only when you are already near the latest message.

While Eve works, material milestones appear as Eve messages directly beneath your request. Specialist work remains spoken by Eve and uses role badges such as Security review or Test strategy. Running milestones update in place when they complete, degrade, or become blocked.

Reloaded runs keep a compact transcript with up to three important milestones; choose Show full activity to see the retained history. The separate Agent Activity card contains run status, steps, budgets, validation checks, tools, logs, and diagnostics as technical details, rather than repeating the conversation. Clarification, approval, recovery, and final-response cards remain the authoritative controls.

These updates are bounded, redacted summaries—not raw chain-of-thought. Qarion does not expose prompts, private model reasoning, raw tool inputs or outputs, or token-streaming telemetry. Run transcript items are display metadata attached to Eve's final turn and are not added to future chat context or conversation memory.

Governed Agent Flow

AI-assisted implementation follows a review-first lifecycle. Qarion centrally assesses whether each missing fact can be recovered from saved answers, current context, bounded read-only research, or internal repair before it asks you:

1. Qarion resolves system-owned gaps internally and asks only for decisions or facts that you own, such as product intent, data-correctness mappings, mutation scope, security/provider policy, or publication effects.
2. The agent proposes concise business tasks with nested execution units. Each unit identifies its dependencies, allowed action categories, bounded paths or spec targets, acceptance criteria, required evidence, and verification profile.
3. Qarion checks the execution-unit graph before review and again on approval. Duplicate or cyclic units, missing dependencies or acceptance criteria, and compound or unbounded write scopes are rejected.
4. Approved units seed the task checklist. Mutating units require accepted actions and verification evidence; notes alone can complete only advisory or research work.
5. Validation, generated-code checks, sandbox checks, and diff review run before files are saved.

Required checklist tasks must be completed with evidence before Qarion applies mutating changes. If a task is blocked or missing evidence, the run pauses for a repair, clarification, or revised plan. Resume checks compare the current draft, approved plan, checklist, repo-edit session, and validation tier before the agent continues, so stale runs cannot silently apply older work.

Independent units run in dependency order, up to three at a time when their resolved scopes do not overlap. Overlapping or unresolved scopes are serialized, and a failed unit blocks its descendants rather than unrelated siblings. Older approved sessions remain resumable: Qarion derives one compatibility unit from each older plan item, while newly generated or revised plans enforce strict per-unit action attribution.

When an approved plan contains independent work, Qarion can validate and save a dependency-closed execution slice before asking for a remaining decision. The clarification card shows that the draft advanced and which units remain. Stopping after committed progress requires an explicit choice: keep the reduced scope or restore the session's base as a new version. Historical intermediate versions remain available in either case. External dependencies appear as a separate waiting state rather than a clarification form.

Partial execution slices retain accepted unit work, hashes, and evidence in the repository edit session so a retry can continue only failed or stale units. They cannot be published: publication still requires every required unit, the accepted result for the current version, and the final validation gate.

Repair And Recovery

Validation and blocking diff-review diagnostics use the same recovery flow. Qarion routes each diagnostic independently to code repair, plan revision, user input, environment resolution, transient retry, unsafe-scope review, or internal review. This keeps a repairable code finding from being hidden by an unrelated environment issue.

The cockpit shows each repair attempt, its strategy, resolved and new diagnostics, remaining blockers, and the available recovery actions. A repaired candidate is retained only when diagnostics improve without introducing a new critical, security, or policy blocker; otherwise Qarion restores the last known good candidate. Continue starts the offered recovery work. Inspect recovery context opens the recorded evidence without starting another attempt.

Reliability Budgets

By default, each authoring workflow has a 48-operation ceiling split into protected lanes: 36 for planning and implementation, 4 for final diff review, and 8 for repair or replan. Planning cannot consume the final-review or repair reserves; unused planning capacity can roll forward to repair.

The cockpit reports logical operations and provider attempts separately, along with token use, the remaining repair reserve, deadline, and any exhaustion reason. If a run becomes budget-limited, completed unit checkpoints remain available for a later continuation instead of being discarded.

Superadmins can change the per-run lanes, timeouts, tool and retry allowances, parallelism, recovery limits, and optional monthly token or cost caps in Administration -> System Settings -> AI -> Pipeline Authoring. The effective policy is snapshotted when a run starts, so an in-progress run keeps consistent limits even when an administrator changes the policy for later runs.

Reviewing Generated Changes

Plan reviews can ask you to:

• Implement an approved plan when generated files and checks are ready.
• Revise the plan with notes when the scope or approach is wrong.
• Cancel without changing the draft.
• Save files after failed generated-file validation, selecting only the preview files you want to keep.
• Approve repository commands when a generated workflow needs an allowed command such as a project-specific validator.

Generated-file reviews are bounded to the previewed paths. Stale reviews are rejected so an older response cannot overwrite newer draft changes.

Generation Readiness

Plan reviews include a generation-readiness summary when Qarion has enough validation evidence. Treat it as the gate between "draft proposed" and "safe to apply":

Status	Meaning
Ready	The current saved definition version passed every check required for the selected operation. Publication requires all strict validation sections to pass.
Not validated	The current saved version has no validation evidence for the selected operation.
Stale	Validation evidence belongs to an older definition version, target, fingerprint, or policy version.
Unsupported	The selected target does not implement the operation, such as publication for Dagster.
Blocked	A required support file, support test, checklist item, validation section, runtime resource, dependency check, or sandbox section failed, warned, or was skipped.
Failed validation	Generated files were produced, but validation found issues that need repair or selective save.

When readiness is blocked, use the next action shown in the review before retrying implementation. Do not approve repository commands or save failed files unless the command, path list, and diagnostics match the change you expect.

Validation, Commands, And Publication

Use validation before publication to catch generated-code and dependency issues. Publication should target the configured repository and preserve the review trail in Qarion.

Compile, resource sync, and publish actions use readiness for the current saved definition version. Save or revert dirty editor changes first; unresolved editor conflicts also block these actions so the code being executed cannot differ from the code that was validated. Running validation retries the authoritative operation profile rather than reusing an older local result.

Repository command execution is disabled by default. When administrators enable local command execution, commands are policy checked before execution and repository-code checks such as tests, package scripts, builds, custom validators, and repository tool profiles require approval before they run. Commands with network intent require the configured authoring network policy, and approvals are bound to the exact request shown in the review.

Generated-code validation can include static checks, generated support tests, dependency smoke checks, generated-code sandbox execution, and database sandbox checks for SQL or database-facing support files. Dedicated code-writer routing, sandbox validation, database sandbox validation, dependency smoke checks, performance profiling, and validation package fetch are rollout controls. Administrators can inspect their current state in AI Ops.

For publication, compiler, static-code, support-test, dependency, runtime-resource, and sandbox sections must each be exactly Passed. A requested check that was blocked by an earlier failure is reported as Skipped, not passed, and blocks publication. Sandbox-disabled deployments therefore cannot publish until an administrator configures the hardened runner. Dagster remains compile-only until target-native sandbox validation is available.

Publication and managed-resource synchronization continue as durable operations after you leave or reload the page. The Publish inspector restores active work, shows validation, Git, merge-request, and per-resource checkpoints, and offers retry or cancellation only when the current phase permits it. A publication can be cancelled only before its first Git mutation; resource sync stops between items and retains evidence for items already completed. If the draft advances while an older immutable version finishes publishing, Qarion reports the result as stale and does not mark the newer draft as published.

Agent-authored versions must also have a complete execution-slice record for the exact current version and fresh validation fingerprints. Partial slices, unresolved publication decisions, incomplete units, pending interactions, and a draft that differs from the accepted result all block publication.

For local Docker demo validation, the demo can preconfigure the Qarion base URL used by validation runners with DEMO_VALIDATION_QARION_BASE_URL.

Troubleshooting

The plan cannot be implemented means a required checklist item is still open, blocked, or missing evidence. Review the task checklist and answer any pending clarification before retrying.

Generated files failed validation means Qarion could not safely apply the full generated set. Use the failed-file review to save only the preview files you trust, then continue from the updated draft.

Generation readiness is blocked means the review is missing required support files, support tests, sandbox results, dependency evidence, or validation evidence. Follow the review's next action rather than bypassing the gate.

A review became stale means files, checklist state, or the approved plan changed after the review was created. Rerun the agent or validation so the review reflects the current draft.

A repository command needs approval means the agent requested a command outside normal file edits. Approve only commands whose purpose, arguments, and network behavior match the change you expect.

Package or dependency checks are blocked means validation package fetch, private package repository access, public-index access, or the sandbox runtime is not ready. Ask an administrator to review the Pipeline Authoring readiness checks in AI Ops.

Reliability or run trace warnings appear in AI Ops means recent authoring runs may be missing checkpoint, validation-tier, tool-policy, prompt/cache, or memory evidence. Use the linked workflow run before treating the issue as a normal generation failure.

Related Guides

• Pipelines and Orchestration for observing provider-backed pipeline runs after publication.
• Artifact Repositories for package and OCI dependencies used by authored pipelines.
• Pipeline Authoring API for automating workspaces, definitions, draft files, validation, publication, and audit inspection.
• AI Ops for Pipeline Authoring readiness and failure signals.