Pipeline Authoring

Pipeline Authoring lets teams draft, validate, and publish pipeline definitions from a governed Qarion workspace. It is designed for repository-backed files, AI-assisted code generation, validation, and review before pipeline code is published.

For the complete trust-boundary, permission, approval, network, sandbox, and audit model, see Pipeline Authoring Security.

Workspace Setup

1. Open Authoring -> Pipeline Authoring.
2. Create or select a workspace.
3. Choose the default repository and branch.
4. Set workspace visibility to Space, Private, or Shared.
5. Select authoring query connectors when previews or generated code need governed data access.

Shared workspaces can grant users and teams view or manage access.

Authoring Targets

Pipeline Authoring supports Airflow and Dagster targets. Airflow remains the default for existing drafts and continues to publish generated DAG files under dags/. Dagster drafts generate Definitions, @job, and @op based Python artifacts under orchestrators/dagster/.

Dagster support is compile-and-validate focused in this release. Generic executable nodes such as marker, bash, Python callable, dbt, and SQLMesh nodes are supported. Qarion Airflow provider nodes for quality pushes and contract validation remain Airflow-only until a Dagster or SDK-backed runtime adapter is configured.

Managed API Connections

Create one API connector for a REST, SOAP, or GraphQL service, then attach a compatible Qarion credential. API connectors store only non-secret connection settings, including the protocol, HTTPS base URL, authentication mode, safe headers, timeout and retry policy, pagination, and contract source. Secrets remain in Qarion credentials. Supported authentication modes are no authentication, basic authentication, API key, bearer token, and OAuth 2.0 client credentials.

Each connector owns an immutable normalized contract snapshot. REST contracts can come from OpenAPI, SOAP contracts from WSDL/XSD, and GraphQL contracts from SDL or introspection. A definition can be entered manually, selected from a repository file, or explicitly fetched from an approved remote HTTPS URL. Qarion never performs a hidden contract refresh while authoring. Use Refresh contract or Test connection when you want network access; a failed refresh leaves the last successful snapshot active.

When a pipeline uses an API operation, Pipeline Authoring asks for the connector, compatible credential, operation, request and response mapping, timeout/retry behavior, and pagination. Protocol-specific questions cover REST methods and paths, SOAP service/binding/message details, or GraphQL documents, variables, and response paths. Only connectors and credentials available in the current space are offered.

Review any difference between a connector setting and its credential before generation. Qarion does not silently resolve those conflicts. It also blocks publication when the active contract has changed, the selected operation no longer exists, a credential is missing or incompatible, or a security standard is unresolved. DELETE and operations identified as destructive require a typed confirmation bound to the connector, contract hash, operation, and affected scope; changing any of those values invalidates the confirmation.

Generated Airflow and Dagster code receives managed resource references and a reviewed operation descriptor, never credential values or the complete contract. Qarion's runtime adapter enforces TLS, timeouts, retry and pagination limits, redaction, schema checks, and OAuth token handling. Airflow uses managed Connections and Variables. Dagster publication requires a deployment adapter that verifies environment-backed secret and configuration bindings before the pipeline can run.

Packaged Airflow DAGs

Every successful Airflow publication also creates an immutable ZIP from the exact validated definition version. Qarion discovers the nearest requirements.txt or pyproject.toml by walking from the DAG directory toward the repository root, resolves draft workspace edits and deletions, and vendors the hash-locked wheel closure alongside the DAG and its transitively imported local modules. The ZIP is stored as a deployment artifact; it is not committed to the publication branch or merge request.

The default airflow_package_mode is extracted. Extract this archive into the Airflow DAG runtime so pure-Python and platform-matched native wheels are available as normal imports. zipimport is opt-in and is intended for placing the ZIP directly in DAGS_FOLDER; Qarion rejects that mode when a wheel contains a native extension or other content that Python cannot import directly from a ZIP. The package also contains deterministic .qarion/deployment.json evidence, the selected manifest, local package __init__.py files, and explicitly referenced runtime files. Tests, caches, bytecode, VCS metadata, secrets, unsafe paths, and unrelated repository files are excluded.

Extracted packages preserve repository-relative paths. Direct ZIP-import packages place the compiled DAG at the ZIP root for Airflow discovery while keeping local modules and vendored pure-Python dependencies importable below it.

The Publish inspector shows the selected mode and manifest, dependency-lock identity, checksum, size, compatibility failures, and an authorized download action. A packaging failure occurs before Qarion creates or mutates a Git branch. Dagster publication is unchanged.

Working With Files

The workspace file navigator supports creating, renaming, moving, searching, and deleting draft files. Python editing includes repository-aware autocomplete when the backend can inspect the current draft and related package context.

Keep generated and edited files reviewable. Pipeline Authoring is most useful when drafts can be validated, audited, and published through the same workspace rather than copied manually into a separate repository.

Editing The DAG Graph

The Graph inspector shows the planned execution DAG as stages, nodes, and dependencies. Users with manage access can make direct graph edits from the inspector:

• Select a node to view its canonical payload, upstream tasks, and downstream tasks.
• Add a node from the node catalog examples, then edit the canonical JSON payload before saving.
• Update or remove an existing node. Removing a node also removes dependencies that point into or out of it.
• Add or remove dependencies between existing nodes.
• Load, edit, save, or delete the DAG README. The README is backed by the same draft-file override/deletion mechanism as regular files, but appears as a first-class DAG resource.

Direct graph edits append a new definition version and run full spec validation. They are disabled while reviewing a plan preview so an agent-generated review cannot be mixed with manual edits mid-review. If you need the agent to perform a graph change, ask for the change in chat: the agent uses planner tools that prepare reviewed node, dependency, and README actions rather than writing directly.

README paths resolve from the explicit README path when one is set, otherwise from the DAG or artifact path with an .md suffix, otherwise README.md.

Working With Chat Context

Type @ in the composer to search workspace files and attach an exact file reference to the message. Qarion validates each reference against the current workspace and draft before the message is accepted, expands valid references into file context for the agent, and limits a message to 20 file mentions. Renamed, deleted, inaccessible, malformed, or stale references are rejected so the agent does not silently work from the wrong file.

Historical messages with an older malformed mention remain readable. Qarion leaves that stored text unchanged instead of making the conversation fail to load.

Assistant rationale, plan-review controls, recovery actions, run-summary provenance, and other artifacts are grouped under Show details on a completed assistant message. Expand it when you need the evidence or action controls; collapse it to keep long conversations scannable. The chat keeps your reading position when older content or details expand and follows new activity only when you are already near the latest message.

While Eve works, material milestones appear as Eve messages directly beneath your request. Specialist work remains spoken by Eve and uses role badges such as Security review or Test strategy. Running milestones update in place when they complete, degrade, or become blocked.

Reloaded runs keep a compact transcript with up to three important milestones; choose Show full activity to see the retained history. The separate Agent Activity card contains run status, steps, budgets, validation checks, tools, logs, and diagnostics as technical details, rather than repeating the conversation. Clarification, approval, recovery, and final-response cards remain the authoritative controls.

These updates are bounded, redacted summaries—not raw chain-of-thought. Qarion does not expose prompts, private model reasoning, raw tool inputs or outputs, or token-streaming telemetry. Run transcript items are display metadata attached to Eve's final turn and are not added to future chat context or conversation memory.

Run status and recovery

The status card uses the same server-owned run projection while a job is live and after its assistant turn is saved. Reloading the page therefore restores the same job identity, elapsed duration, activity, budget, and recovery actions instead of reconstructing a second status from message text.

Status	Meaning
Queued	The job is durably accepted and waiting for its first worker transition.
Running	The current worker or coordinator owns healthy active work.
Recovering	Active work is delayed, being reconciled, or has a stalled liveness signal; Qarion preserves the durable run while it determines the next safe transition.
Needs input	Work reached a settled clarification or external-wait boundary. Use the displayed interaction control.
Needs review	Work reached a settled plan or generated-change review. Use the review bound to that turn.
Succeeded	The turn completed without a remaining interaction or blocking failure.
Failed	The parent run or its canonical result failed, even if an older activity item still appeared to be running.
Cancelled	Cancellation reached a settled boundary.

Liveness is a secondary signal—healthy, delayed, recovering, or stalled—and does not override a settled failure, cancellation, review, or input boundary. Once a turn settles, no activity row remains marked as running: successful rows complete, the active failure is marked failed, and other unfinished rows are shown as interrupted. Use the card's recovery action when one is offered rather than starting an unrelated message that would lose the recorded recovery scope.

Governed Agent Flow

AI-assisted implementation follows a review-first lifecycle. Qarion centrally assesses whether each missing fact can be recovered from saved answers, current context, bounded read-only research, or internal repair before it asks you:

1. Qarion resolves system-owned gaps internally and asks only for decisions or facts that you own, such as product intent, data-correctness mappings, mutation scope, security/provider policy, or publication effects.
2. The agent proposes concise business tasks with nested execution units. Each unit identifies its dependencies, allowed action categories, bounded paths or spec targets, acceptance criteria, required evidence, and verification profile.
3. Qarion checks the execution-unit graph before review and again on approval. Duplicate or cyclic units, missing dependencies or acceptance criteria, and compound or unbounded write scopes are rejected.
4. For executable units, review the machine-checkable acceptance details as part of the plan. They identify exact spec or file effects, callable signatures, runtime resources, support-test boundaries, and any data-only behavior cases with their expected return or error. These details are the validation contract, not generated implementation code.
5. Approved units seed the task checklist. Mutating units require accepted actions and verification evidence; notes alone can complete only advisory or research work.
6. Validation, generated-code checks, sandbox checks, and diff review run before files are saved.

Required checklist tasks must be completed with evidence before Qarion applies mutating changes. If a task is blocked or missing evidence, the run pauses for a repair, clarification, or revised plan. Resume checks compare the current draft, approved plan, checklist, repo-edit session, and validation tier before the agent continues, so stale runs cannot silently apply older work.

When you ask for a narrow revision to named plan units, Qarion keeps the other reviewed units unchanged and revalidates the complete plan. Existing behavior case inputs are not silently dropped from a targeted revision. A change to a unit's scope, runtime binding, inputs, or acceptance details invalidates only the affected checkpoint and requires current review before execution.

Independent units run in dependency order, up to three at a time when their resolved scopes do not overlap. Overlapping or unresolved scopes are serialized, and a failed unit blocks its descendants rather than unrelated siblings. Older approved sessions remain resumable: Qarion derives one compatibility unit from each older plan item, while newly generated or revised plans enforce strict per-unit action attribution.

When an approved plan contains independent work, Qarion can validate and save a dependency-closed execution slice before asking for a remaining decision. The clarification card shows that the draft advanced and which units remain. Stopping after committed progress requires an explicit choice: keep the reduced scope or restore the session's base as a new version. Historical intermediate versions remain available in either case. External dependencies appear as a separate waiting state rather than a clarification form.

Partial execution slices retain accepted unit work, hashes, and evidence in the repository edit session so a retry can continue only failed or stale units. They cannot be published: publication still requires every required unit, the accepted result for the current version, and the final validation gate.

Repair And Recovery

Validation and blocking diff-review diagnostics use the same recovery flow. Qarion routes each diagnostic independently to code repair, plan revision, user input, environment resolution, transient retry, unsafe-scope review, or internal review. This keeps a repairable code finding from being hidden by an unrelated environment issue. Plan-repairable findings enter automatic repair only when their execution unit or affected path is inside the approved plan; unscoped or invalid-plan findings stay in the approval lane.

The cockpit shows each repair attempt, its strategy, resolved and new diagnostics, remaining blockers, and the available recovery actions. A repaired candidate is retained only when diagnostics improve without introducing a new critical, security, or policy blocker; otherwise Qarion restores the last known good candidate. Continue starts the offered recovery work. Inspect recovery context opens the recorded evidence without starting another attempt.

Runs expose a bounded repair ledger in chat, job, activity, replay, and saved session snapshots. Its stable entries cover structured-output, decomposition, generation, validation, and diff-review repairs without retaining prompts, generated code, provider request bodies, or detected secret values. The run timeline updates those same entries while work is in flight, so polling does not duplicate completed attempts.

Reliability Budgets

By default, each authoring workflow has a 48-operation ceiling split into protected lanes: 36 for planning and implementation, 4 for final diff review, and 8 for repair or replan. Planning cannot consume the final-review or repair reserves; unused planning capacity can roll forward to repair.

The cockpit reports logical operations and provider attempts separately, along with token use, the remaining repair reserve, deadline, and any exhaustion reason. If a run becomes budget-limited, completed unit checkpoints remain available for a later continuation instead of being discarded.

Superadmins can change the per-run lanes, timeouts, tool and retry allowances, parallelism, recovery limits, and optional monthly token or cost caps in Administration -> System Settings -> AI -> Pipeline Authoring. The effective policy is snapshotted when a run starts, so an in-progress run keeps consistent limits even when an administrator changes the policy for later runs.

Durable Agentic runs also snapshot an administrator-selected execution topology. Pooled sessions use the shared authoring execution pool, slim- orchestrator sessions keep the agent loop in the control plane while delegating blocking work, and dedicated sessions bind an isolated worker to the runtime thread. This placement is operational: it does not change the review, permission, validation, or publication gates described in this guide. If the selected topology is unavailable, Qarion fails closed before starting a new Agentic root instead of moving it to another mode. Existing runs never move when an administrator changes the setting.

Public Web Research

When current public facts materially affect a plan and Qarion or repository evidence cannot answer the question, Eve can request governed web search while reformulating or decomposing the root task. A planning pass can propose one external research need with a public-safe query and an explanation of why the evidence is required. Research stops when the task plan reaches approval; it is not available during implementation, validation repair, diff repair, or publication.

The first consent card identifies the information need and planning stage, shows the configured provider and exact sanitized query, explains why research is needed, and states that approval covers bounded follow-up searches for this plan. Approval is scoped to the current user, space, pipeline definition, root request, and provider binding. It allows at most six distinct searches with eight results each for up to 24 hours. Later agent-decided searches in the same task do not prompt again. The activity timeline reports research progress without exposing query text. You can decline and continue without research; the affected fact remains explicitly unresolved.

Search results appear as source cards with title, domain, date, bounded snippet, and a safe external link. The final plan includes sources accumulated across all searches. Planning searches add official documentation to the effective sanitized query and place domains derived from trusted Qarion, package, connector, or provider metadata first. Other public results remain fallback evidence and are not labeled as official. Treat every snippet as evidence, not instructions. Web content cannot override your request, Qarion policy, repository evidence, validators, or an approved plan.

Repeated identical searches reuse recorded evidence without consuming another search. If the same information need and query recur after their evidence was provided, Qarion stops instead of searching again or inventing a plan fact. Declined consent, unsafe queries, an expired or revoked grant, a changed provider binding, exhausted quota, malformed provider output, and unavailable providers also fail closed with an unresolved-research diagnostic.

Web research never opens pages, browses interactively, accesses authenticated sites, or gives generated pipeline code network access. Checkpoint recovery restores the exact pending requirement and original root task. Delegated workers and subagents receive normalized evidence only; they do not receive the search tool.

The search backend is deployment-configured rather than hard-coded into the authoring flow. The standard Docker demo uses Qarion's bundled SearXNG-compatible adapter against a local live-search service; production can bind the same adapter or another compatible read-only MCP provider. See AI Ops for rollout and health checks.

Reviewing Generated Changes

Plan reviews can ask you to:

• Implement an approved plan when generated files and checks are ready.
• Revise the plan with notes when the scope or approach is wrong.
• Cancel without changing the draft.
• Save files after failed generated-file validation, selecting only the preview files you want to keep.
• Approve repository commands when a generated workflow needs an allowed command such as a project-specific validator.

While a task-decomposition plan is awaiting review, you can also reply in the chat. Clear approval, whole-plan cancellation, or revision instructions are applied to that exact plan revision. Questions receive a plan-grounded answer without changing the review. Ambiguous messages and unrelated requests leave the plan pending and ask you to clarify or use the review controls. This conversational handling does not apply to implementation authorization or repository-command approval, which still require their explicit controls.

Generated-file reviews are bounded to the previewed paths. Stale reviews are rejected so an older response cannot overwrite newer draft changes.

Readiness States

Qarion exposes two related readiness contracts. Plan-review generation readiness decides whether proposed generated changes can move to implementation:

Status	API value	Meaning
Not applicable	not_applicable	The review does not request executable generation.
Ready	ready	The generated candidate has the evidence required for implementation.
Reviewable with warnings	reviewable_with_warnings	Validation completed with warnings that must be reviewed before implementation.
Environment blocked	environment_blocked	Required execution evidence, commonly sandbox evidence, is unavailable in the current environment.
Blocked	blocked	A required artifact, approval, diff review, or policy gate is unresolved.
Failed validation	failed_validation	Generated files were produced, but validation found issues that need repair or selective save.

Compile, resource-sync, and publish actions use a separate immutable operation-readiness contract for the current saved definition version:

Status	API value	Meaning
Ready	ready	The current version passed every check required for the selected operation.
Blocked	blocked	A required capability or validation check prevents the operation.
Not validated	not_validated	The current version has no validation snapshot for the selected operation.
Stale	stale	Evidence belongs to another definition version, target, spec fingerprint, or validation-policy version.
Unsupported	unsupported	The target does not implement the operation, such as publication for Dagster.

When readiness is blocked, use the next action shown in the review before retrying. Do not approve repository commands or save failed files unless the command, path list, and diagnostics match the change you expect.

Validation, Commands, And Publication

Use validation before publication to catch generated-code and dependency issues. Publication should target the configured repository and preserve the review trail in Qarion.

Compile, resource sync, and publish actions use readiness for the current saved definition version. Save or revert dirty editor changes first; unresolved editor conflicts also block these actions so the code being executed cannot differ from the code that was validated. Running validation retries the authoritative operation profile rather than reusing an older local result.

Repository command execution is disabled by default. When administrators enable local command execution, commands are policy checked before execution and repository-code checks such as tests, package scripts, builds, custom validators, and repository tool profiles require approval before they run. Commands with network intent require the configured authoring network policy, and approvals are bound to the exact request shown in the review. The approval fingerprint covers the command identifier, arguments, purpose, timeout, working directory, bounded paths, repository identity, prepared-edit identifiers, test-runner and profile metadata, workspace fingerprint, runtime image, and network request and policy. Changing any of that material invalidates the approval and requires a new one. An authored Bash task is runtime pipeline content; it does not authorize Qarion to run a shell while authoring. Shell executables are not accepted as registered repository-command profiles, and an authoring-time command must use the separate, approval-gated command path.

Sensitive artifact approval is a separate contract from repository-command approval. Mutations to protected automation and executable configuration—such as CI workflows, CODEOWNERS, hooks, Docker and build configuration, scripts, and shell files—require a server-generated approval. It is bound to the repository and ref, target, base definition version, result spec fingerprint, approved-plan hash, path and operation, exact content SHA (or the deletion tombstone SHA), expected base SHA, and sensitive-artifact policy version. Any change makes the approval stale and requires a new review.

Approval-gated authored Bash nodes use the same sensitive-artifact review flow, but bind approval to the node, exact command SHA, detected capabilities, repository and ref, target, base version, approved plan, result spec fingerprint, and Bash policy version. Command or capability drift invalidates the approval. Bash commands classified as blocked cannot be approved.

Repository-control paths under .git, credential and secrets directories, non-template environment files, credential files, and private-key or certificate material are always blocked rather than approval-gated. Compiler-owned generated paths have an additional, separate structural rule: they cannot be moved or deleted directly, and neither command nor sensitive-artifact approval overrides that boundary. Make changes through the supported draft or owning definition surface, then validate the resulting candidate again.

Generated-code validation can include static checks, generated support tests, dependency smoke checks, generated-code sandbox execution, and database sandbox checks for SQL or database-facing support files. Dedicated code-writer routing, sandbox validation, database sandbox validation, dependency smoke checks, performance profiling, and validation package fetch are rollout controls. Administrators can inspect their current state in AI Ops.

Generated or candidate-supplied support-test output is an observation, not publication authority. Qarion can promote it only after an independent dependency-backed, sandbox, or other platform-controlled check produces trusted passing evidence. If that evidence cannot run, the support-test section remains unavailable or blocked even when the candidate reports success.

The six publication sections are necessary but not sufficient. Every publication also requires a passing trusted-parent identity for the exact candidate and spec. Candidates that contain executable Python require canonical platform-controlled behavioral authorization as well; candidate-authored exit codes, JUnit, or output cannot grant that authority.

Under the strict sandbox policy, compiler, static-code, support-test, dependency, runtime-resource, and sandbox sections must each be exactly Passed. A requested check that was blocked by an earlier failure is reported as Skipped, not passed, and blocks publication. The fast policy has one deliberately narrow non-production exception: when the publication gate is not in block mode, publication may continue only when the sandbox section being skipped is the sole gate failure. It does not excuse any other failed, warned, missing, or skipped section, and it cannot bypass a block-mode gate. Dagster remains compile-only until target-native sandbox validation is available.

Publication and managed-resource synchronization continue as durable operations after you leave or reload the page. The Publish inspector restores active work, shows validation, Git, merge-request, and per-resource checkpoints, and offers retry or cancellation only when the current phase permits it. A publication can be cancelled only before its first Git mutation; resource sync stops between items and retains evidence for items already completed. If the draft advances while an older immutable version finishes publishing, Qarion reports the result as stale and does not mark the newer draft as published.

Agent-authored versions must also have a complete execution-slice record for the exact current version and fresh validation fingerprints. Partial slices, unresolved publication decisions, incomplete units, pending interactions, and a draft that differs from the accepted result all block publication.

For local Docker demo validation, the demo can preconfigure the Qarion base URL used by validation runners with DEMO_VALIDATION_QARION_BASE_URL.

Troubleshooting

The plan cannot be implemented means a required checklist item is still open, blocked, or missing evidence. Review the task checklist and answer any pending clarification before retrying.

Generated files failed validation means Qarion could not safely apply the full generated set. Use the failed-file review to save only the preview files you trust, then continue from the updated draft.

Generation readiness is blocked means the review is missing required support files, support tests, sandbox results, dependency evidence, or validation evidence. Follow the review's next action rather than bypassing the gate.

A review became stale means files, checklist state, or the approved plan changed after the review was created. Rerun the agent or validation so the review reflects the current draft.

A repository command needs approval means the agent requested a command outside normal file edits. Approve only commands whose purpose, arguments, working directory, bounded paths, repository, runtime image, and network behavior match the change you expect. If any of those fields or the workspace fingerprint changes, review the replacement request rather than reusing the old approval.

Package or dependency checks are blocked means validation package fetch, private package repository access, public-index access, or the sandbox runtime is not ready. Ask an administrator to review the Pipeline Authoring readiness checks in AI Ops.

Reliability or run trace warnings appear in AI Ops means recent authoring runs may be missing checkpoint, validation-tier, tool-policy, prompt/cache, or memory evidence. Use the linked workflow run before treating the issue as a normal generation failure.

Related Guides

• Pipeline Authoring Security for access, approval, tool, network, sandbox, audit, and revocation boundaries.
• Pipelines and Orchestration for observing provider-backed pipeline runs after publication.
• Artifact Repositories for package and OCI dependencies used by authored pipelines.
• Pipeline Authoring API for automating workspaces, definitions, draft files, validation, publication, and audit inspection.
• AI Ops for Pipeline Authoring readiness and failure signals.