Skip to main content

Credential Management

A comprehensive guide to managing credentials securely in Qarion ETL.

Overview​

Qarion ETL provides a credential store system that allows you to:

  • Define credentials once and reuse them across different configurations
  • Store credentials securely in databases, local keystores, or cloud key management services
  • Reference credentials in configuration files without exposing sensitive data
  • Support multiple backends (database, local keystore, AWS SSM Parameter Store)
  • Automatic Fernet encryption - All credentials are encrypted using Fernet with a project-specific key

Fernet Encryption​

Qarion ETL uses Fernet encryption (symmetric authenticated cryptography) to encrypt all stored credentials. A Fernet key is automatically generated when you initialize a new project and stored in your qarion-etl.toml configuration file.

Key Points:

  • Fernet key is generated automatically on project initialization
  • Each project has its own unique Fernet key
  • All credential stores use this key for encryption
  • The key is stored in qarion-etl.toml - never commit this file to version control

Quick Start​

1. Initialize Project (Fernet Key Generation)​

When you create a new project, a Fernet encryption key is automatically generated:

qarion-etl new-project my_project

This creates qarion-etl.toml with a generated Fernet key:

fernet_key = "gAAAAABh..."  # Automatically generated

Important: Never commit the fernet_key to version control. Add qarion-etl.toml to .gitignore or use environment variables for the key.

2. Configure Credential Store​

Add credential store configuration to your qarion-etl.toml:

[credential_store]
type = "local_keystore"
[credential_store.config]
keystore_path = "~/.qarion_etl/credentials.keystore"

The Fernet key from your project configuration is automatically used for encryption.

3. Define Credentials​

Define credential metadata in your configuration:

[[credentials]]
id = "my_aws_creds"
name = "AWS Production Credentials"
credential_type = "aws"
description = "AWS credentials for production S3 access"

4. Store Credentials​

Store the actual credential data using the Python API:

from qarion_etl.credentials import get_credential_store

store = get_credential_store()
store.store_credential(
    credential_id="my_aws_creds",
    credential_data={
        "aws_access_key_id": "AKIA...",
        "aws_secret_access_key": "...",
        "region_name": "us-east-1"
    }
)

5. Use Credentials​

Reference credentials in your configuration:

[properties.input_ingestion]
path = "s3://my-bucket/data/"
credentials = "${credential:my_aws_creds}"

Credential Store Types​

Stores credentials in a local encrypted file.

Configuration:

[credential_store]
type = "local_keystore"
[credential_store.config]
keystore_path = "~/.qarion_etl/credentials.keystore"  # Optional

Features:

  • Simple setup, no external dependencies
  • File permissions automatically set to 600 (owner read/write only)
  • Default location: ~/.qarion_etl/credentials.keystore
  • Good for development and single-user environments
  • Fernet encryption: Credentials are encrypted using the project's Fernet key

Example:

[credential_store]
type = "local_keystore"

Stores credentials in a database table with optional encryption.

Configuration:

[credential_store]
type = "database"
[credential_store.config]
engine = { name = "sqlite", config = { path = "metadata.db" } }
table_name = "xt_credentials"  # Optional

Features:

  • Centralized credential storage
  • Works with any database engine (SQLite, PostgreSQL, etc.)
  • Fernet encryption: Credentials are encrypted using the project's Fernet key
  • Good for team environments and production

Example:

[credential_store]
type = "database"
[credential_store.config]
engine = { name = "postgres", config = { host = "db.example.com", database = "mydb" } }
table_name = "credentials"

Uses AWS Systems Manager Parameter Store for secure credential storage with automatic encryption.

Configuration:

[credential_store]
type = "aws_ssm"
[credential_store.config]
parameter_prefix = "/qarion_etl/credentials/"  # Optional, default: /qarion_etl/credentials/
region_name = "us-east-1"
kms_key_id = "alias/my-credentials-key"  # Optional, uses default SSM key if not provided

Features:

  • Automatic encryption using SecureString parameters
  • Integrated with AWS IAM for access control
  • Supports IAM roles and temporary credentials
  • Highly secure for production environments
  • No additional storage needed (SSM handles persistence)
  • Optional KMS key for custom encryption

Example:

[credential_store]
type = "aws_ssm"
[credential_store.config]
parameter_prefix = "/myapp/credentials/"
region_name = "us-east-1"
kms_key_id = "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012"

IAM Permissions Required:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameter",
        "ssm:PutParameter",
        "ssm:DeleteParameter",
        "ssm:DescribeParameters",
        "ssm:ListTagsForResource",
        "ssm:AddTagsToResource"
      ],
      "Resource": "arn:aws:ssm:*:*:parameter/qarion_etl/credentials/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt",
        "kms:Encrypt"
      ],
      "Resource": "arn:aws:kms:*:*:key/*"
    }
  ]
}

Credential Definitions​

Define credential metadata in your qarion-etl.toml:

[[credentials]]
id = "my_aws_creds"
name = "AWS Production Credentials"
credential_type = "aws"
description = "AWS credentials for production S3 access"
[credentials.metadata]
environment = "production"
region = "us-east-1"

Credential Types:

  • aws: AWS credentials (access key, secret key, region)
  • database: Database credentials (host, user, password, database)
  • api_key: API key credentials
  • oauth: OAuth credentials
  • basic_auth: Basic authentication credentials
  • custom: Custom credential type

Using Credentials​

CLI Commands​

Qarion ETL provides CLI commands for managing credentials:

Store a Credential​

# From JSON file
qarion-etl credentials store --credential-id aws_prod \
  --name "AWS Production" \
  --type aws \
  --file credentials.json

# Interactive mode
qarion-etl credentials store --credential-id api_key --interactive

# From JSON string
qarion-etl credentials store --credential-id db_prod \
  --json '{"host":"db.example.com","user":"admin","password":"secret"}'

List Credentials​

# List all credentials
qarion-etl credentials list

# List as JSON
qarion-etl credentials list --format json

Get a Credential​

# Verify credential exists
qarion-etl credentials get --credential-id aws_prod

# Show credential value
qarion-etl credentials get --credential-id aws_prod --show-value

# Show as environment variables
qarion-etl credentials get --credential-id aws_prod --show-value --format env

Delete a Credential​

qarion-etl credentials delete --credential-id old_cred

Show Credential Metadata​

qarion-etl credentials show --credential-id aws_prod

For complete CLI reference, see CLI Reference.

Using Credentials​

Credential Reference Syntax​

Reference credentials in configuration using:

${credential:credential_id}

In Flow Configuration​

[properties.input_ingestion]
path = "s3://my-bucket/data/"
pattern = "*.csv"
credentials = "${credential:my_aws_creds}"

In Storage Backend Configuration​

[storage]
type = "s3"
bucket = "my-bucket"
credentials = "${credential:my_aws_creds}"

In Engine Configuration​

[engine]
name = "postgres"
[engine.config]
host = "db.example.com"
database = "mydb"
user = "myuser"
password = "${credential:db_password}"

In Dictionary Values​

For credentials that return dictionaries, you can reference specific fields:

[engine.config]
host = "${credential:db_creds}.host"
database = "${credential:db_creds}.database"
user = "${credential:db_creds}.user"
password = "${credential:db_creds}.password"

Storing Credentials​

Using Python API​

from qarion_etl.credentials import get_credential_store, CredentialDefinition, CredentialType

# Get credential store
store = get_credential_store()

# Store AWS credentials
store.store_credential(
    credential_id="my_aws_creds",
    credential_data={
        "aws_access_key_id": "AKIA...",
        "aws_secret_access_key": "...",
        "region_name": "us-east-1"
    },
    credential_definition=CredentialDefinition(
        id="my_aws_creds",
        name="AWS Production Credentials",
        credential_type=CredentialType.AWS,
        description="AWS credentials for production"
    )
)

# Store database credentials
store.store_credential(
    credential_id="db_prod",
    credential_data={
        "host": "db.example.com",
        "database": "mydb",
        "user": "myuser",
        "password": "secretpassword"
    }
)

Retrieving Credentials​

from qarion_etl.credentials import get_credential_store

store = get_credential_store()

# Get credential
aws_creds = store.get_credential("my_aws_creds")
print(aws_creds)
# {
#     "aws_access_key_id": "AKIA...",
#     "aws_secret_access_key": "...",
#     "region_name": "us-east-1"
# }

# List all credentials
credential_ids = store.list_credentials()
print(credential_ids)
# ["my_aws_creds", "db_prod"]

# Check if credential exists
exists = store.credential_exists("my_aws_creds")
print(exists)
# True

Deleting Credentials​

from qarion_etl.credentials import get_credential_store

store = get_credential_store()
store.delete_credential("my_aws_creds")

Complete Examples​

Example 1: Local Keystore with AWS Credentials​

Configuration (qarion-etl.toml):

[credential_store]
type = "local_keystore"

[[credentials]]
id = "aws_prod"
name = "AWS Production"
credential_type = "aws"

Store credentials:

from qarion_etl.credentials import get_credential_store

store = get_credential_store()
store.store_credential(
    credential_id="aws_prod",
    credential_data={
        "aws_access_key_id": "AKIA...",
        "aws_secret_access_key": "...",
        "region_name": "us-east-1"
    }
)

Use in flow:

[properties.input_ingestion]
path = "s3://my-bucket/data/"
credentials = "${credential:aws_prod}"

Example 2: Database Store with Multiple Credentials​

Configuration:

[credential_store]
type = "database"
[credential_store.config]
engine = { name = "sqlite", config = { path = "metadata.db" } }

[[credentials]]
id = "aws_prod"
name = "AWS Production"
credential_type = "aws"

[[credentials]]
id = "db_prod"
name = "Database Production"
credential_type = "database"

Store credentials:

from qarion_etl.credentials import get_credential_store

store = get_credential_store()

# Store AWS credentials
store.store_credential(
    credential_id="aws_prod",
    credential_data={
        "aws_access_key_id": "AKIA...",
        "aws_secret_access_key": "..."
    }
)

# Store database credentials
store.store_credential(
    credential_id="db_prod",
    credential_data={
        "host": "db.example.com",
        "database": "mydb",
        "user": "myuser",
        "password": "secretpassword"
    }
)

Use in configurations:

[engine]
name = "postgres"
[engine.config]
host = "${credential:db_prod}.host"
database = "${credential:db_prod}.database"
user = "${credential:db_prod}.user"
password = "${credential:db_prod}.password"

Example 3: AWS SSM Parameter Store​

Configuration:

[credential_store]
type = "aws_ssm"
[credential_store.config]
parameter_prefix = "/qarion_etl/credentials/"
region_name = "us-east-1"
kms_key_id = "alias/my-credentials-key"  # Optional

Store credentials:

from qarion_etl.credentials import get_credential_store

store = get_credential_store()
store.store_credential(
    credential_id="aws_prod",
    credential_data={
        "aws_access_key_id": "AKIA...",
        "aws_secret_access_key": "..."
    }
)

Migration from Inline Credentials​

Before (Inline Credentials)​

[properties.input_ingestion]
path = "s3://my-bucket/data/"
credentials = {
    aws_access_key_id = "AKIA..."
    aws_secret_access_key = "..."
    region_name = "us-east-1"
}

After (Credential Store)​

1. Configure credential store:

[credential_store]
type = "local_keystore"

2. Store credential:

from qarion_etl.credentials import get_credential_store

store = get_credential_store()
store.store_credential(
    credential_id="aws_prod",
    credential_data={
        "aws_access_key_id": "AKIA...",
        "aws_secret_access_key": "...",
        "region_name": "us-east-1"
    }
)

3. Update configuration:

[properties.input_ingestion]
path = "s3://my-bucket/data/"
credentials = "${credential:aws_prod}"

Fernet Key Management​

Automatic Generation​

A Fernet key is automatically generated when you:

  • Run qarion-etl new-project to create a new project
  • Run qarion-etl init to initialize a project

The key is stored in your qarion-etl.toml file:

fernet_key = "gAAAAABh..."  # Automatically generated

Key Security​

Critical Security Notes:

  1. Never commit qarion-etl.toml to version control if it contains a Fernet key
  2. Backup your Fernet key - losing it means you cannot decrypt stored credentials
  3. Use different keys for different environments (dev, staging, production)
  4. Rotate keys periodically - generate a new key and re-encrypt credentials

Manual Key Generation​

If you need to generate a new Fernet key:

from qarion_etl.credentials.encryption import generate_fernet_key

new_key = generate_fernet_key()
print(new_key)
# gAAAAABh...

Then update your qarion-etl.toml:

fernet_key = "gAAAAABh..."  # Your new key

Warning: Changing the Fernet key will make existing encrypted credentials unreadable. You'll need to re-store all credentials with the new key.

Security Best Practices​

  1. Never commit credentials to version control

    • Use credential stores for all sensitive data
    • Never commit qarion-etl.toml with Fernet key - add to .gitignore: 
      # Qarion ETL configuration with sensitive Fernet key
      qarion-etl.toml
    • Use environment variables as fallback only
    • Add credential store files to .gitignore: 
      # Credential store files
      ~/.qarion_etl/credentials.keystore
      *.keystore

  2. Choose the right credential store

    • Development: Local keystore
    • Teams: Database store
    • Production: AWS SSM Parameter Store or database with encryption

  3. Rotate credentials regularly

    • Update credentials in the credential store
    • Old references will automatically use new credentials
    • No need to update configuration files

  4. Limit access to credential stores

    • Use IAM roles for AWS SSM Parameter Store
    • Restrict file permissions for local keystore (automatically set to 600)
    • Use database access controls for database store

  5. Use credential definitions

    • Document what each credential is for
    • Use descriptive names and descriptions
    • Include metadata for tracking and auditing

  6. Environment separation

    • Use different credential stores for dev/staging/prod
    • Use different credential IDs for different environments
    • Never share credentials between environments

Troubleshooting​

Credential Not Found​

Error: Credential 'my_creds' not found in credential store

Solution:

  1. Verify credential store is configured in qarion-etl.toml
  2. Check that credential was stored: store.credential_exists("my_creds")
  3. List all credentials: store.list_credentials()

Credential Store Not Configured​

Error: Credential references are not resolved

Solution:

  1. Add [credential_store] section to qarion-etl.toml
  2. Ensure credential store type is supported
  3. Verify credential store configuration is valid

Permission Errors (Local Keystore)​

Error: Permission denied when accessing keystore

Solution:

  1. Check file permissions: ls -l ~/.qarion_etl/credentials.keystore
  2. Ensure you own the file: chown user:user ~/.qarion_etl/credentials.keystore
  3. Set correct permissions: chmod 600 ~/.qarion_etl/credentials.keystore