Issue Management Technical Reference

Architecture Overview

The Issue Management system (Ticket Service) serves as the "active" layer of the data catalog. Unlike static catalog metadata, Issues are stateful, workflow-driven entities that govern the lifecycle of data remediation.

Key Architectural Decisions

• Loose Coupling: Tickets are linked to Products/columns via ID references, allowing tickets to persist even if the underlying data asset is dropped/recreated (persistent compliance history).
• JSONB Extensibility: Metadata for DevOps integrations (Git commits, PRs) is stored in unstructured JSONB columns to support evolving integrations without schema migrations.

---

Data Model

Entity Relationship Diagram (ERD)

Core Entities

DQTicket

The central unit of work.

Field	Type	Description
id	UUID	Unique identifier.
priority	String	low, medium, high, critical.
status	String	open, in_progress, resolved, wont_fix.
source_alert_id	UUID	(Optional) Link to the SmartAlert that triggered this.
resolution_summary	JSONB	Structured debrief data (root cause, impact).
devops_metadata	JSONB	Stores git_commits array, merge_requests array.

TicketHistory

Compliance interface.

Field	Type	Description
ticket_id	UUID	Parent ticket.
actor_id	UUID	User who made the change.
field_changed	String	Name of the field (e.g., "status").
old_value	String	Serialized previous value.
new_value	String	Serialized new value.

RecordIssue

Record issues track problems on specific rows within a data product. They can be created manually or mastered by an external validation system.

Field	Type	Description
source_kind	String	manual, airflow, error_table, or custom.
external_source_key	String	Stable identifier for the external snapshot source.
external_issue_key	String	Stable issue identity within the external source.
connector_id	UUID	Optional connector that produced the issue.
sync_job_id	UUID	Sync job that last updated the issue.
last_seen_at	DateTime	Last time the issue appeared in an external snapshot.
external_payload	JSONB	Original external payload for traceability.
external_status	String	Latest status supplied by the external source.
status_overridden_at	DateTime	When a user manually overrode the external status.

---

Service Layer

IssueService

Core Methods

create_ticket_from_alert(alert_id: UUID, user: User) -> DQTicket

• Context Injection:
  • Fetches SmartAlert details.
  • Pre-fills Ticket Title with Alert Name.
  • Parses Alert Context to identify and auto-link the affected DataProduct.
  • Sets source_alert_id to maintain lineage.

associate_merge_request(ticket_id: UUID, url: str)

• Validation:
  • Regex checks the URL pattern (supports GitHub/GitLab).
  • (Optional) Calls external provider API to verify MR existence.
• Storage: Appends object {url: str, verified: bool, timestamp: datetime} to devops_metadata['merge_requests'].

resolve_ticket(ticket_id: UUID, debrief: ResolutionDebriefDTO)

• Gatekeeper Logic:
  • If priority is critical or high: Checks if debrief payload is present and valid.
  • If missing, raises ResolutionDebriefRequiredException.
• State Transition: Sets status resolved.

---

DevOps Integration Patterns

The system uses a "Link & Verify" pattern rather than full bidirectional sync.

1. User Input: User pastes a URL.
2. System Verification: System regex-matches the URL provider.
3. Storage: Stored as metadata on the ticket.
4. Navigation: UI renders deep links to the external system.

Future Extension: Webhooks from GitHub/GitLab could call IssueService.add_comment to post updates back to the ticket.

External Record Issue Sync

The external record issue sync endpoint accepts a complete snapshot for one source_kind and source_key. Qarion upserts matching records by external issue identity, updates metadata and last-seen timestamps, and can auto-resolve records that disappear from the latest snapshot.

Use auto_resolve_missing=true for authoritative sources such as warehouse error tables. Use auto_resolve_missing=false when the sender only posts partial batches.