Skip to main content

Connectors

Connectors are the bridge between Qarion and your external data platforms. They handle schema discovery, metadata synchronization, and integration with tools like dbt, Snowflake, and BigQuery.

What is a Connector?​

A Connector is a configured pipeline that connects to a specific external system and synchronizes its metadata into Qarion. Connectors are created within Source Systems and define what data to import, how to filter it, and how often to sync.

While a Source System represents the external platform itself, a Connector defines the specific integration configuration — which schemas to include, which tables to exclude, and the synchronization schedule.

Connector Configuration​

Schema Filtering​

Control exactly which metadata is imported by configuring inclusion and exclusion rules. Include Lists specify which schemas to import (e.g., PROD, ANALYTICS), while Exclude Patterns filter out unwanted schemas using pattern matching (e.g., *_STAGING, TMP_*).

Table Filtering​

Within included schemas, you can further refine the import scope using table-level filters. Regular expression patterns (e.g., ^PROD_.*) allow precise control over which tables appear in your catalog.

Sync Schedule​

Connectors support flexible scheduling options. Automatic Scheduling uses cron expressions to sync on a recurring basis (e.g., every 6 hours or daily at midnight). Manual Only mode disables automatic sync, requiring explicit user action to trigger an import.

Running a Sync​

Manual Sync​

Click Sync Now on any connector to trigger an immediate metadata import. The platform provides real-time feedback during execution, showing the progress and any warnings or errors encountered.

Sync Results​

After each sync, you receive a detailed summary of the operation. This includes the number of tables Added, Updated, and Deleted, along with any Warnings (such as unreadable column comments). Failed syncs provide specific error messages to guide troubleshooting.

Sync History​

The connector maintains a full history of all sync executions. Each entry records the start time, duration, status (success or failure), the trigger source (manual or scheduled), and the changes detected.

Error Handling​

Connectors categorize errors to help you diagnose issues quickly. Connection Errors indicate problems reaching the external host (e.g., wrong hostname or port). Authentication Errors mean the credentials are invalid or expired. Permission Errors occur when the service account cannot read specific schemas or tables. Timeout Errors happen when syncs exceed the configured time limit.

For transient errors (network timeouts, brief outages), connectors support configurable retry policies with exponential backoff.

Concurrency Protection​

Only one sync can run per connector at a time. If a sync is already in progress, subsequent attempts are queued or rejected to prevent data conflicts and resource contention.

Integration with Products​

When a connector discovers tables and views, they can be automatically linked to Data Products in the catalog. This linkage enables automatic lineage detection, where upstream and downstream dependencies are mapped from the source system through to downstream consumers.

Learn More​