AI Ops
AI Ops gives administrators a live operational view of Qarion AI usage, failures, guardrails, workflow runs, evaluations, and authoring readiness.
Open Admin -> AI Ops.
Guardrail Audit Visibility
Admin AI Logs shows the guardrail outcome and audit coverage on each
governed call. Filters can narrow by outcome, coverage, phase, stable check ID,
finding code, or product. Expanding a row shows full check totals and bounded
safe detail grouped into input, tool, and output phases, including skip reasons,
transformations, product/rule references, and truncation warnings. Older v1/v2
records are labeled legacy/incomplete; they are not evidence that unrecorded
checks passed.
Product Governance shows the product-authorized evaluation history and the
separate opted-in runtime alert stream. Routine pass evidence stays in immutable
AI logs. A runtime governance event is created only when a triggered finding has
alert_on_trigger enabled, and links back to its source log/check.
During rollout, monitor v3 coverage by entry point, truncated detail, audit write failures, and the ratio of complete, exempt, and legacy records. An exempt record must include the code-owned reason declared by its disabled entry point.
Provider And Embedding Controls
AI Ops works with the provider controls in the admin area:
| Control | Where it is configured | What it affects |
|---|---|---|
| LLM provider and model routing | Admin -> System Settings and instance settings | Copilot, generation, triage, authoring assistance, and other LLM-backed workflows. |
| Embedding configuration | Admin -> System Settings and instance settings | Semantic search for catalog products, documentation, and standards content. |
| External AI tool providers | API-backed superadmin and space-setting endpoints | Read-only external MCP tools that can be exposed to AI workflows in selected spaces. |
Use these controls together. LLM configuration decides which model handles AI reasoning and generation. Embedding configuration decides how semantic search vectors are produced. External tool providers decide which additional read-only tool results AI workflows can use in a space.
Embedding Configuration
Embedding configuration controls semantic search. Superadmins can choose a
provider, model, dimensions, optional base URL, enabled state, and index
version. Supported providers are local, openai, azure_openai, mistral,
and voyage.
Key lifecycle states:
| State | Meaning |
|---|---|
| Active | The selected embedding shape is ready for newly indexed content. |
| Pending reindex | Provider, model, or dimensions changed and existing vectors should be regenerated. |
| Failed | The configuration or runtime needs administrator attention. |
For local embeddings, the runtime badge can show idle, warming, ready, or
failed. Remote providers show an external-provider state because Qarion tests
them through provider calls rather than local model warmup.
After changing provider, model, dimensions, or index version:
- Save the embedding configuration.
- Use Test to verify the effective provider can generate a vector.
- Reindex products so catalog semantic search uses the new vector shape.
- Reindex standards or documentation content when the changed provider should affect those results as well.
- Monitor AI Ops and search behavior for provider failures or missing semantic results.
Instance settings can override the control-plane embedding configuration for a specific tenant instance. When an instance override exists, testing and reindexing should be performed from that instance context.
External AI Tool Providers
External AI tool providers let superadmins register governed MCP stdio servers as read-only tool sources for AI workflows. A provider defines the command arguments, optional safe relative working directory, non-secret environment variables, optional credential reference, and tool metadata overrides.
The rollout lifecycle is:
- Create the provider with a safe argv command and credential reference.
- Discover tools so Qarion records tool names, descriptions, input schemas, namespaced names, and risk metadata.
- Review tool metadata and enable only read-risk tools.
- Enable the provider for specific spaces and optionally allowlist tool names.
- Watch AI logs and workflow runs for tool-call failures or unexpected provider output.
For Pipeline Authoring web research, classify exactly one compatible discovered tool as web search, then select it under the target space's governed web search settings. Qarion advertises the canonical authoring tool only when the instance feature flag is enabled and the binding is healthy. There is no automatic provider fallback.
Enable PIPELINE_AUTHORING_WEB_SEARCH_ENABLED for the governed search path.
Enable PIPELINE_AUTHORING_ITERATIVE_WEB_RESEARCH_ENABLED as well when the root
planning agent may request additional searches while reformulating or
decomposing the task. The iterative flag has no effect by itself. Both flags
must have the same effective value in API, control-plane, and worker runtimes;
search remains unavailable to subagents and after task-plan approval.
Production code and Helm default both flags to false. Set
backend.config.pipelineAuthoringWebSearchEnabled and
backend.config.pipelineAuthoringIterativeWebResearchEnabled together after a
compatible provider binding is healthy. The standard Docker demo defaults both
flags to true and points Qarion's bundled adapter at the demo's local SearXNG
service; values in .env override those demo defaults.
The first planning search pauses durably for consent. Approval covers bounded
follow-ups for that root task, so later distinct needs do not create additional
consent cards. Monitor the low-cardinality followup_proposed,
followup_executed, followup_cache_reused, followup_no_progress, and
followup_quota_exhausted outcomes. These signals intentionally exclude query,
domain, tenant, and repository labels.
The provider executable must already be installed in every control-plane
runtime image that may perform discovery or tool execution. Runtime pip and
npx installation is forbidden. The MCP server continues to run over stdio;
allow its search API destinations through deployment-controlled egress or an
outbound proxy. This does not grant network access to generated Airflow code,
authoring sandboxes, repository commands, or delegated workers.
The canonical provider input is a public-safe query of 3–500 characters and
max_results from 1–8. Results must contain a title, public HTTPS URL, bounded
snippet, and optional publication timestamp. Queries are sanitized before
authorization, credentials are blocked, and ambiguous redaction asks the user
for a safer formulation. Qarion retains the sanitized query fingerprint and
normalized evidence, not the unsafe original or raw provider response.
Bundled SearXNG-compatible adapter
Qarion includes a provider-independent MCP stdio adapter that can query any SearXNG-compatible JSON endpoint. Register it with these settings:
| Provider field | Value |
|---|---|
| Command argv | python -m app.services.platform.external_providers.searxng_web_search |
| Environment | QARION_WEB_SEARCH_ENDPOINT=https://search.example.com/search |
| Optional environment | QARION_WEB_SEARCH_TIMEOUT_SECONDS=15; QARION_WEB_SEARCH_USER_AGENT=Qarion-Web-Search/1.0 |
| Optional credential | A credential-store record with secret_env_var set to QARION_WEB_SEARCH_API_KEY |
The endpoint must be an HTTP(S) URL without embedded credentials or a fragment.
It may include fixed parameters such as a search category; the adapter replaces
q and format on every request, requests JSON, and forwards an optional
credential as a Bearer token. Timeouts are bounded from 1 to 30 seconds, the
response is limited to 1 MB, and only bounded public HTTP(S) result URLs are
returned to Qarion for its normal capability validation and normalization.
After registration, discover the web_search tool, classify it as the
web search capability, enable only that read-risk tool for the intended
spaces, and bind it as each space's primary web-search provider. The adapter is
an integration convenience, not a bypass around consent, quota, source
normalization, or the fail-closed research lifecycle.
V1 external provider enablement is read-risk only. Shell workflows, sensitive environment keys, unsafe working directories, and blocked executable names are rejected. Store secrets in credential records instead of plain environment variables.
The Docker demo starts a pinned local SearXNG service, seeds a credential-free Qarion Web Search provider using the bundled adapter, and binds it only when no Acme Analytics web-search binding exists. Demo queries are live and are not matched to fixtures. Consent, quota, cache, recovery, source-card, and planning-loop behavior still use the same provider catalog and governed application path as production. Production feature flags remain disabled by default.
Instance Scope
Some AI controls can be configured at the control plane or overridden for a specific runtime instance. Use instance-level overrides when a tenant needs a different embedding provider, model dimensions, base URL, or key than the global default.
When debugging instance-specific behavior, check the configuration source badge
before changing settings. If the source is environment, the value comes from
runtime settings. If the source is database, it was saved through the admin
configuration APIs or UI.
Pipeline Authoring Readiness
The Pipeline Authoring readiness section summarizes whether generated-code workflows can run safely in the current instance. Use it before rolling out or debugging AI-assisted implementation. It combines runtime settings, recent workflow evidence, failure signals, and reliability audit findings.
Readiness is an operational signal, not an authorization grant. Space, workspace, repository, connector, runtime-profile, and approval checks still run independently. The generated-code sandbox reported here is also distinct from repository-command isolation, deployed Airflow or Dagster runtimes, the shared Authoring Network proxy, and connector-bound credentialless API egress.
The top status is intentionally conservative:
| Status | Meaning |
|---|---|
| Ready | Required rollout controls are configured and recent evidence is healthy. |
| Attention | The workflow can run, but reliability, coverage, or rollout posture needs review. |
| Blocked | A required control is missing, unsupported, or failing closed. |
| Unknown | Qarion does not yet have enough recent activity or definitions to grade the signal. |
Key runtime chips include:
| Signal | Meaning |
|---|---|
| Code writer | Whether dedicated code-writer routing is enabled for executable Pipeline Authoring changes. |
| Code writer source | Which setting or environment source selected the code-writer state. |
| Sandbox | Whether generated-code sandbox validation is enabled. |
| Backend | The generated-code sandbox backend, such as Docker or Kubernetes. |
| Image | Whether the generated-code sandbox image is configured. |
| Namespace | Whether the Kubernetes sandbox namespace is configured when Kubernetes is used. |
| Timeout | The maximum generated-code sandbox execution time. |
| DB sandbox | Whether database sandbox validation is enabled for generated SQL/database work. |
| DB backend | The database sandbox backend, currently expected to be SQLite when enabled. |
| Dependency smoke | Whether dependency smoke validation is enabled. |
| Package fetch | The validation package-fetch policy. |
| Public index | Whether public package index access is configured. |
| Qarion base URL | Whether validation package fetch can resolve the Qarion package endpoint. |
| Profiler | Whether Pipeline Authoring performance profiling is enabled. |
| Slow module | The module load threshold that creates profiler warnings. |
| CPU spike | The CPU spike threshold that creates profiler warnings. |
| Memory growth | The memory growth threshold that creates profiler warnings. |
Readiness Checks
Readiness checks combine runtime configuration, recent failures, and coding-agent diagnostics. Statuses are intended to be operational:
- Ready means the check has the expected configuration or recent evidence.
- Attention means the workflow can still run, but rollout or reliability needs review.
- Blocked means the feature is missing required configuration or is failing closed.
- Unknown means there is not enough recent evidence for that check.
Common checks include:
| Check | What to review |
|---|---|
| Review integrity | Persisted plan reviews, saved-file records, failed-file reviews, and review drift. |
| Failure signals | Recent blocked or degraded examples from the reliability audit. |
| Active jobs | Pipeline Authoring chat jobs that appear stale or stuck. |
| Code generation specialist | Whether dedicated code-writer routing is intentionally enabled or disabled. |
| Validation runtime | Generated-code sandbox settings and database sandbox settings. |
| Package fetch | Whether validation can install dependencies from Qarion repositories and, when allowed, a public index. |
| Dependency smoke | Whether generated dependencies are smoke-tested before rollout. |
| Performance profiler | CPU, memory, and module timing diagnostics in recent traces. |
| Recent AI activity | Failed, blocked, invalid-output, degraded, or running Pipeline Authoring incidents. |
| Prompt/cache health | Cache-aware prompt layout and stable prompt section evidence. |
| Memory freshness | Bounded conversation, clarification, manual workspace memory, and learned long-term memory provenance. |
| Subagent/tool policy | Read-only subagent evidence, denied tool-call recording, and tool-policy diagnostics. |
| Reliability trace health | Structured contract hashes, validation tiers, checkpoint coverage, repair counters, and terminal status. |
| Run trace health | Normalized workflow run traces for recent Pipeline Authoring activity. |
| Run projection health | Unknown terminal statuses, settled runs with stale running activity, and missing canonical summaries. |
When a check has a link target, use it to jump to the related AI logs, workflow runs, jobs, or system settings. Metadata chips show compact evidence, such as selected backend, enabled state, source setting, or warning counts.
Failure Signals
Failure signals show recent Pipeline Authoring examples that need review. They can include stuck jobs, failed generated-code validation, sandbox failures, guardrail blocks, missing package access, command-approval pauses, or repeated repair loops.
Use the linked workflow, log, or authoring session to inspect the affected workspace. The AI Ops page should provide enough context to triage the failure without exposing raw prompts, secrets, or unsanitized command output.
Failure signals are separate from readiness findings. Findings describe persisted review or saved-file drift. Failure signals describe recent workflow behavior, such as validation failures, stale resume checkpoints, sandbox failures, blocked evidence gates, denied tool calls, or subagent budget caps.
Rollout Checklist
Before enabling broad generated-code rollout:
- Confirm code-writer routing is intentionally enabled or intentionally disabled.
- Enable sandbox validation and verify the backend and image are configured.
- Enable database sandbox validation when generated SQL or database-facing support files are part of the rollout.
- Enable dependency smoke checks when package fetch and sandbox execution are available.
- Set validation package fetch to
qarion_onlyorqarion_plus_publicwhen generated dependencies need package installation. - Confirm private package repositories, public-index access, and the Qarion base URL match the package policy.
- Review prompt/cache health, memory freshness, subagent policy, reliability trace health, and run trace health for warnings.
- Enable the performance profiler during rollout windows when CPU, memory, or module load behavior is part of the risk.
- Investigate recent failure signals before increasing rollout.
Troubleshooting
Code writer is disabled means executable changes use the main planning path. Enable the dedicated code-writer setting only when the specialist path is ready for rollout.
Sandbox validation is disabled means generated code is not being executed in the sandbox before review. Enable sandbox validation before relying on generated code for broader teams.
Sandbox backend is unsupported means the configured backend is not one of the supported runtime backends. Use Docker or Kubernetes and configure the matching image and runtime settings.
Database sandbox backend is unsupported means database sandbox validation is enabled with a backend Qarion cannot run safely. Use the supported SQLite-backed database sandbox or disable database sandbox validation until the backend is ready.
Package fetch is disabled means validation cannot install generated external
dependencies. Use qarion_only for private Qarion package repositories or
qarion_plus_public when public-index access is allowed.
Package fetch is enabled but the Qarion base URL is missing means generated dependency installation cannot resolve private Qarion package endpoints. Set the validation package base URL before relying on private package dependencies.
Dependency smoke is disabled means dependencies may be selected but not import-smoke-tested before review. Enable it once package fetch and the sandbox runtime are ready.
Performance profiler warnings appear means recent traces exceeded module load, CPU, or memory growth thresholds. Treat these as rollout signals and inspect the linked workflow before assuming the model response is the root cause.
Prompt/cache or memory freshness warnings appear means recent authoring runs may be missing stable prompt layout, context freshness, or recovery state. Check the linked workflow runs before treating the issue as a model-quality problem.
Reliability trace warnings appear means recent runs missed structured contract evidence, validation-tier evidence, checkpoint coverage, sandbox results, repair counters, or expected terminal status. Review the run trace and the replay/eval output before increasing rollout.
Embedding config is pending reindex means semantic vectors may still use an older provider, model, or dimension setting. Run product and standards reindex jobs before judging semantic search quality.
Embedding test fails means the selected provider cannot generate a vector from the current configuration. Check the provider, model name, dimensions, base URL, API key, local model runtime, and instance scope.
External tool provider discovery fails means Qarion could not start or query the configured MCP stdio server. Check argv, working directory, allowed executable, credential reference, environment keys, and provider logs.
A tool is not available in a space means the provider may be inactive, the tool was not discovered, the tool is disabled, the space enablement is off, the tool is outside the allowlist, or the tool risk level is not read-only.
Related Guides
- Pipeline Authoring for the user-facing review and validation flow.
- Pipeline Authoring Security for permissions, approvals, network boundaries, sandboxes, and audit behavior.
- Authoring Overview for shared workspace and execution controls.
- Artifact Repositories for private package and OCI dependencies.
- Notebook Workers for dedicated notebook runtime monitoring.
- AI-Assisted Capabilities for the broader AI feature map.