Skip to main content

Credential Store System

Overview​

The credential store system provides a unified interface for storing and retrieving credentials from various backends. This allows you to:

  • Define credentials once and reuse them across different configurations
  • Store credentials securely in databases, local keystores, or cloud key management services
  • Reference credentials in configuration files without exposing sensitive data
  • Support multiple credential store backends (database, local keystore, AWS SSM Parameter Store)
  • Automatic Fernet encryption: All credentials are encrypted using Fernet encryption with a project-specific key

Fernet Key​

Qarion ETL uses Fernet encryption (symmetric authenticated cryptography) to encrypt all stored credentials. A Fernet key is automatically generated when you initialize a new project and stored in your qarion-etl.toml configuration file.

Key Features:

  • Automatic Generation: Fernet key is generated automatically on project initialization
  • Project-Specific: Each project has its own unique Fernet key
  • Secure: Uses industry-standard Fernet encryption from the cryptography library
  • Required: All credential stores require a Fernet key for encryption

Important:

  • The Fernet key is stored in qarion-etl.toml - never commit this file to version control
  • Keep your Fernet key secure - losing it means you cannot decrypt stored credentials
  • Each project should have its own unique Fernet key

Architecture​

Components​

  1. CredentialStore: Abstract base class for credential stores
  2. CredentialDefinition: Metadata about credentials (name, type, description)
  3. CredentialStoreFactory: Factory for creating credential store instances
  4. Credential Store Implementations: Database, Local Keystore, AWS SSM Parameter Store

Credential Reference Syntax​

Credentials can be referenced in configuration using the syntax:

${credential:credential_id}

This syntax can be used in:

  • Configuration files (TOML)
  • Flow definitions
  • Storage backend configurations
  • Any configuration value that accepts credentials

Credential Store Types​

1. Database Credential Store​

Stores credentials in a database table with optional encryption.

Configuration:

[credential_store]
type = "database"
[credential_store.config]
engine = { name = "sqlite", config = { path = "data.db" } }
table_name = "xt_credentials"  # Optional, default: xt_credentials
# fernet_key is automatically loaded from project config

Note: The Fernet key is automatically loaded from the project's fernet_key configuration. You don't need to specify it in the credential store config.

Features:

  • Stores credentials in a database table
  • Supports optional encryption
  • Automatic table creation
  • Works with any database engine (SQLite, PostgreSQL, etc.)

Example:

[credential_store]
type = "database"
[credential_store.config]
engine = { name = "sqlite", config = { path = "metadata.db" } }
table_name = "credentials"

2. Local Keystore Credential Store​

Stores credentials in a local encrypted keystore file.

Configuration:

[credential_store]
type = "local_keystore"
[credential_store.config]
keystore_path = "~/.qarion_etl/credentials.keystore"  # Optional, default: ~/.qarion_etl/credentials.keystore
# fernet_key is automatically loaded from project config

Note: The Fernet key is automatically loaded from the project's fernet_key configuration. You don't need to specify it in the credential store config.

Features:

  • Stores credentials in a local file
  • File permissions set to 600 (owner read/write only)
  • Supports optional encryption
  • Default location: ~/.qarion_etl/credentials.keystore

Example:

[credential_store]
type = "local_keystore"
[credential_store.config]
keystore_path = "/secure/path/credentials.keystore"

3. AWS SSM Parameter Store​

Stores credentials in AWS Systems Manager Parameter Store with automatic encryption.

Configuration:

[credential_store]
type = "aws_ssm"
[credential_store.config]
parameter_prefix = "/qarion_etl/credentials/"  # Optional, default: /qarion_etl/credentials/
region_name = "us-east-1"  # Optional
kms_key_id = "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012"  # Optional
[credential_store.config.credentials]
aws_access_key_id = "AKIA..."  # Optional, can use IAM roles
aws_secret_access_key = "..."  # Optional, can use IAM roles

Features:

  • Automatic encryption using SecureString parameters
  • Integrated with AWS IAM for access control
  • No additional storage needed (SSM handles persistence)
  • Supports IAM roles and temporary credentials
  • Highly secure for production environments
  • Optional KMS key for custom encryption

Example:

[credential_store]
type = "aws_ssm"
[credential_store.config]
parameter_prefix = "/myapp/credentials/"
region_name = "us-east-1"
kms_key_id = "alias/my-credentials-key"  # Optional

IAM Permissions Required:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameter",
        "ssm:PutParameter",
        "ssm:DeleteParameter",
        "ssm:DescribeParameters",
        "ssm:ListTagsForResource",
        "ssm:AddTagsToResource"
      ],
      "Resource": "arn:aws:ssm:*:*:parameter/qarion_etl/credentials/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt",
        "kms:Encrypt"
      ],
      "Resource": "arn:aws:kms:*:*:key/*"
    }
  ]
}

Configuration​

Basic Setup​

When you initialize a new project, a Fernet key is automatically generated and added to your qarion-etl.toml:

fernet_key = "gAAAAABh..."  # Automatically generated on project init

Add credential store configuration to your qarion-etl.toml:

[credential_store]
type = "local_keystore"
[credential_store.config]
keystore_path = "~/.qarion_etl/credentials.keystore"

The Fernet key from the project configuration is automatically used for encryption - you don't need to specify it in the credential store config.

Credential Definitions​

Define credentials in your configuration:

[[credentials]]
id = "my_aws_creds"
name = "AWS Production Credentials"
credential_type = "aws"
description = "AWS credentials for production S3 access"
[credentials.metadata]
environment = "production"
region = "us-east-1"

Credential Types:

  • aws: AWS credentials
  • database: Database credentials
  • api_key: API key credentials
  • oauth: OAuth credentials
  • basic_auth: Basic authentication
  • custom: Custom credential type

Usage​

Storing Credentials​

Using Python API​

from qarion_etl.credentials import get_credential_store, CredentialDefinition, CredentialType

# Get credential store
store = get_credential_store()

# Store AWS credentials
store.store_credential(
    credential_id="my_aws_creds",
    credential_data={
        "aws_access_key_id": "AKIA...",
        "aws_secret_access_key": "...",
        "region_name": "us-east-1"
    },
    credential_definition=CredentialDefinition(
        id="my_aws_creds",
        name="AWS Production Credentials",
        credential_type=CredentialType.AWS,
        description="AWS credentials for production"
    )
)

Using CLI (Future)​

qarion-etl credentials store \
    --id my_aws_creds \
    --type aws \
    --data '{"aws_access_key_id": "AKIA...", "aws_secret_access_key": "..."}'

Referencing Credentials​

In Flow Configuration​

[properties.input_ingestion]
path = "s3://my-bucket/data/"
pattern = "*.csv"
credentials = "${credential:my_aws_creds}"

In Storage Backend Configuration​

[storage]
type = "s3"
bucket = "my-bucket"
credentials = "${credential:my_aws_creds}"

In Engine Configuration​

[engine]
name = "postgres"
[engine.config]
host = "db.example.com"
database = "mydb"
user = "myuser"
password = "${credential:db_password}"

Retrieving Credentials Programmatically​

from qarion_etl.credentials import get_credential_store

# Get credential store
store = get_credential_store()

# Retrieve credential
aws_creds = store.get_credential("my_aws_creds")
print(aws_creds)
# {
#     "aws_access_key_id": "AKIA...",
#     "aws_secret_access_key": "...",
#     "region_name": "us-east-1"
# }

Examples​

Example 1: Local Keystore with AWS Credentials​

Configuration:

[credential_store]
type = "local_keystore"

[[credentials]]
id = "aws_prod"
name = "AWS Production"
credential_type = "aws"

Store credentials:

from qarion_etl.credentials import get_credential_store

store = get_credential_store()
store.store_credential(
    credential_id="aws_prod",
    credential_data={
        "aws_access_key_id": "AKIA...",
        "aws_secret_access_key": "...",
        "region_name": "us-east-1"
    }
)

Use in flow:

[properties.input_ingestion]
path = "s3://my-bucket/data/"
credentials = "${credential:aws_prod}"

Example 2: Database Store with Multiple Credentials​

Configuration:

[credential_store]
type = "database"
[credential_store.config]
engine = { name = "sqlite", config = { path = "metadata.db" } }

[[credentials]]
id = "aws_prod"
name = "AWS Production"
credential_type = "aws"

[[credentials]]
id = "db_prod"
name = "Database Production"
credential_type = "database"

Store credentials:

from qarion_etl.credentials import get_credential_store

store = get_credential_store()

# Store AWS credentials
store.store_credential(
    credential_id="aws_prod",
    credential_data={
        "aws_access_key_id": "AKIA...",
        "aws_secret_access_key": "..."
    }
)

# Store database credentials
store.store_credential(
    credential_id="db_prod",
    credential_data={
        "host": "db.example.com",
        "database": "mydb",
        "user": "myuser",
        "password": "secretpassword"
    }
)

Use in configurations:

[engine]
name = "postgres"
[engine.config]
host = "${credential:db_prod}.host"
database = "${credential:db_prod}.database"
user = "${credential:db_prod}.user"
password = "${credential:db_prod}.password"

Example 3: AWS SSM Parameter Store​

Configuration:

[credential_store]
type = "aws_ssm"
[credential_store.config]
parameter_prefix = "/qarion_etl/credentials/"
region_name = "us-east-1"
kms_key_id = "alias/my-credentials-key"  # Optional

Store credentials:

from qarion_etl.credentials import get_credential_store

store = get_credential_store()
store.store_credential(
    credential_id="aws_prod",
    credential_data={
        "aws_access_key_id": "AKIA...",
        "aws_secret_access_key": "..."
    }
)

Security Best Practices​

  1. Never commit credentials to version control

    • Use credential stores for all sensitive data
    • Use environment variables as fallback only

  2. Use appropriate credential store for your environment

    • Development: Local keystore
    • Production: AWS SSM Parameter Store or database with encryption

  3. Rotate credentials regularly

    • Update credentials in the credential store
    • Old references will automatically use new credentials

  4. Limit access to credential stores

    • Use IAM roles for AWS SSM Parameter Store
    • Restrict file permissions for local keystore
    • Use database access controls for database store

  5. Use credential definitions

    • Document what each credential is for
    • Use descriptive names and descriptions
    • Include metadata for tracking

API Reference​

CredentialStore​

class CredentialStore(ABC):
    def store_credential(
        self,
        credential_id: str,
        credential_data: Dict[str, Any],
        credential_definition: Optional[CredentialDefinition] = None
    ) -> bool

    def get_credential(self, credential_id: str) -> Optional[Dict[str, Any]]

    def delete_credential(self, credential_id: str) -> bool

    def list_credentials(self) -> list[str]

    def credential_exists(self, credential_id: str) -> bool

CredentialDefinition​

@dataclass
class CredentialDefinition:
    id: str
    name: str
    credential_type: CredentialType
    description: Optional[str] = None
    metadata: Dict[str, Any] = None

Utility Functions​

from qarion_etl.credentials.utils import (
    resolve_credential_reference,
    get_credentials_for_config,
    is_credential_reference,
    extract_credential_id
)