Notebook Workers

Notebook Workers gives administrators a live view of dedicated notebook runtime capacity. Use it to monitor worker startup, health, ownership, backend type, resource class usage, and stale runtime states.

Open Admin -> Notebook Workers.

Summary Metrics

The top summary shows:

Metric	Meaning
Total	Workers returned by the current filters.
Active	Workers that are ready, starting, busy, or stopping.
Failed	Workers that ended in a failed state.
Stale	Workers that appear stuck in starting or busy.

The page refreshes periodically and can also be refreshed manually.

Filters

Use filters to narrow the worker list:

• Status: starting, ready, busy, stopping, stopped, or failed.
• Backend: docker, kubernetes, or external.
• Space ID and User ID: UUID filters for incident investigation.
• Limit: maximum number of workers returned.
• Include terminal workers: include stopped and failed workers.
• Include runtime status: ask the runtime gateway or worker endpoint for live health. This can be slower than database-only status.

Worker Details

Expand a worker row to inspect runtime details, timestamps, repository scope, resource class snapshot, gateway metadata, and sanitized error detail. Runtime health can be:

Health	Meaning
ready	The worker endpoint reports healthy.
starting	The worker is still preparing.
error	The worker endpoint reported an error.
unreachable	Qarion could not reach the worker endpoint.

Resource Classes

Notebook resource classes define the CPU, memory, GPU, workspace, and temporary storage limits offered to users when they start dedicated workers. Qarion stores a resource class snapshot on each worker so later investigations can see the limits used at startup even if the class is changed later.

Administrators can configure resource classes through platform settings or the notebook runtime resource class editor when available. At least one enabled class should be marked as the default.

Resource class settings include:

Setting	Meaning
ID and display name	Stable identifier and user-facing label shown in the worker picker.
Enabled and default	Whether users can select the class and which enabled class is preselected.
CPU and memory requests/limits	Scheduler and runtime limits for the worker process.
GPU count and resource	Optional GPU request and qualified resource name.
Workspace and temp storage	Per-worker storage limits for checked-out files and temporary execution output.

If a resource class is changed, new workers use the new values. Existing worker rows continue to show the snapshot used when they started.

Repository Snapshot Cache

The notebook runtime gateway owns a persistent, commit-exact repository cache. It resolves the requested branch through Git on every launch. A successful resolution either validates an existing snapshot for that commit or hydrates a new one; Git resolution failures never fall back to an older branch snapshot. Workers mount only their assigned snapshot read-only, copy its verified regular files into their private UUID workspace, and receive no Git credentials. Set NOTEBOOK_REPOSITORY_CACHE_ENABLED=false to retain direct per-worker Git hydration. When caching is enabled, unavailable storage fails startup instead of silently changing hydration modes.

The main limits are NOTEBOOK_REPOSITORY_CACHE_MAX_ENTRIES and NOTEBOOK_REPOSITORY_CACHE_MAX_BYTES. LRU eviction enforces both limits and does not remove snapshots pinned during worker startup. Hydration reserves room for the incoming temporary snapshot and atomic publication; a repository that cannot fit fails with a workspace or storage-size error. Old incomplete temporary directories and corrupt snapshots are removed during reconciliation.

Kubernetes storage

Configure notebookRuntime.repositoryCache in Helm. Production defaults create a 20 GiB ReadWriteMany PVC with a 16 GiB logical cache limit and 64 retained snapshots. RWX is the default because the gateway and workers may run on different nodes. The gateway mounts the claim read-write; dynamically created workers mount only storage_subpath for their assigned snapshot read-only.

Use a chart-managed claim:

notebookRuntime:
  repositoryCache:
    enabled: true
    root: /var/cache/qarion/repositories
    maxEntries: "64"
    maxBytes: "17179869184"
    persistence:
      create: true
      existingClaim: ""
      storageClass: "shared-rwx"
      size: 20Gi
      accessModes:
        - ReadWriteMany

For an operator-managed claim, set persistence.existingClaim; this suppresses PVC creation. Setting persistence.create: false without an existing claim is invalid and fails Helm rendering. Keep the physical volume larger than maxBytes so hydration has room for temporary and atomic writes.

The demo Helm overlay creates a 1 GiB claim and applies an 8-entry, 768 MiB logical limit.

Docker storage

Docker accepts a named volume or a bind source:

NOTEBOOK_REPOSITORY_CACHE_ENABLED=true
NOTEBOOK_REPOSITORY_CACHE_ROOT=/var/cache/qarion/repositories
NOTEBOOK_REPOSITORY_CACHE_MAX_ENTRIES=64
NOTEBOOK_REPOSITORY_CACHE_MAX_BYTES=17179869184
NOTEBOOK_RUNTIME_DOCKER_REPOSITORY_CACHE_TYPE=volume
NOTEBOOK_RUNTIME_DOCKER_REPOSITORY_CACHE_SOURCE=qarion_notebook_repository_cache

For bind mode, use TYPE=bind and set SOURCE to an existing absolute host path mounted read-write into the gateway at NOTEBOOK_REPOSITORY_CACHE_ROOT. Relative paths, missing sources, unsupported types, and unsafe cache roots are rejected. Dynamic workers receive a read-only mount for the selected snapshot subdirectory, not the cache root. The Docker demo defaults to the persistent named volume above and the same 768 MiB logical bound; named Docker volumes have no portable physical quota.

Health and recovery

The gateway /ready endpoint checks cache-root existence, write access, locking, and free-space headroom. /health remains independent so a storage failure does not create a liveness restart loop. /metrics exports low-cardinality cache hit, miss, error, hydration, byte, entry, eviction, and corruption counters without repository, tenant, path, or credential labels.

If readiness fails, verify claim attachment or Docker source existence, ownership for the gateway user, mount permissions, and free space. Corrupt or incomplete snapshots can be removed while no worker startup is using them; the gateway recreates them only after resolving Git again. Do not delete the whole cache while workers are starting.

Troubleshooting

Workers remain starting: Check whether the runtime gateway is ready, can write and lock the repository cache, and can create Docker containers or Kubernetes workloads. Confirm the worker image is the dedicated notebook worker image rather than the backend image.

Runtime status is unreachable: Confirm gateway routing, internal tokens, network policy, and worker service reachability.

Dependency setup fails: Check the worker image, package fetch policy, private package repository access, and dependency cache configuration.

Users hit worker limits: Review NOTEBOOK_RUNTIME_MAX_WORKERS_PER_USER and the active resource classes. Stop stale or unused workers before increasing limits.

A resource class is rejected: Confirm exactly one enabled class is marked as the default, CPU and memory quantities are positive, GPU resources use a qualified resource name, and workspace/temp storage values meet the configured minimums.

Stop fails: Inspect runtime gateway reachability and worker cleanup logs. Qarion leaves the worker out of stopped until the failed stop is resolved or a cleanup task reconciles the runtime state.

Related Operations

• Local Docker demo: run make demo-notebook-runtime-up.
• Cleanup: stale runtime worker cleanup is scheduled as a background worker task.
• Deployment: Kubernetes installs need the notebook runtime gateway, worker image, service account, and optional dependency cache PVC configured together.
• User guide: Notebooks explains worker selection, package access, and cell execution behavior.