Skip to main content

Task System Documentation Guide

Quick Navigation​

For Users​

  • Task System Guide - Complete guide to understanding and using tasks
  • Start here if you want to learn about task types, how they work, and how to use them

For Developers​

  • Task Type Plugin Architecture - Technical guide for extending the task system
  • Start here if you want to create custom task type plugins or understand the architecture

Documentation Overview​

The task system documentation is organized into three main areas:

1. User Documentation (docs/user/guides/tasks.md)​

Purpose: Help users understand and work with tasks in Qarion ETL.

Covers:

  • What tasks are and how they work
  • All four built-in task types (Ingestion, Transformation, Quality Check, Export)
  • How to use tasks in flows
  • Task dependencies and patterns
  • Basic guide to creating custom task types
  • Best practices

Audience: Data engineers, pipeline builders, Qarion ETL users

2. Developer Documentation (docs/development/task-plugin-architecture.md)​

Purpose: Technical guide for developers extending the task system.

Covers:

  • Architecture components and design
  • Plugin system internals
  • Complete examples for creating custom task type plugins
  • Migration from old to new architecture
  • Testing and validation
  • Implementation details

Audience: Plugin developers, framework contributors, system architects

3. Reference Documentation​

Plugin Interfaces (docs/api/plugin-interfaces.md):

  • Complete API reference for Task Type Plugin interface
  • Interface specifications
  • Code examples

Plugin Overview (docs/plugins/overview.md):

  • Overview of all plugin types including Task Type Plugins
  • Built-in plugins list
  • Quick links to documentation

Quick Reference (qarion_etl/flows/tasks/plugins/README.md):

  • Quick reference for plugin development
  • Simple examples
  • Plugin discovery information

Documentation Structure​

Task Documentation
│
├── 📘 User Guide (docs/user/guides/tasks.md)
│   └── Comprehensive user-facing guide with examples
│
├── 🔧 Developer Guide (docs/development/task-plugin-architecture.md)
│   └── Technical architecture and plugin development
│
├── 📋 API Reference (docs/api/plugin-interfaces.md)
│   └── Complete interface specifications
│
├── 🌐 Plugin Overview (docs/plugins/overview.md)
│   └── Overview of all plugin types
│
└── 📝 Quick Reference (qarion_etl/flows/tasks/plugins/README.md)
    └── Quick plugin development reference

Key Features Documented​

Task Types​

All four built-in task types are fully documented:

  1. Ingestion Tasks - Load data from external sources
  2. Transformation Tasks - Transform data between datasets
  3. Quality Check Tasks - Validate data quality
  4. Export Tasks - Export data to external destinations

Each task type includes:

  • Purpose and use cases
  • Complete property documentation
  • Working code examples
  • Best practices

Plugin Architecture​

Complete documentation of the plugin system:

  • Task Type Plugin interface
  • Plugin registry system
  • Validation mechanisms
  • Property creation
  • Plugin discovery

Custom Task Types​

Step-by-step guides for:

  • Defining custom properties
  • Creating plugin classes
  • Registering plugins
  • Validation logic

Cross-References​

All documentation is properly cross-referenced:

  • User guide → Developer guide (for advanced topics)
  • Developer guide → User guide (for usage examples)
  • Plugin overview → All task documentation
  • All docs → Related topics (flows, plugins, etc.)

Getting Started​

New to Tasks?​

  1. Read the Task System Guide
  2. Review examples for each task type
  3. Try creating tasks in your flows

Building a Plugin?​

  1. Read the Task Type Plugin Architecture
  2. Review the complete examples
  3. Check the Plugin Interfaces Reference
  4. Follow the plugin development guide

Integrating with Flows?​

  1. Understand how Flows use Tasks
  2. Review task generation in flow plugins
  3. See examples in existing flow plugins

Documentation Quality​

  • ✅ Complete coverage of all features
  • ✅ Clear examples for all task types
  • ✅ Proper separation of user vs developer content
  • ✅ Comprehensive cross-referencing
  • ✅ Updated indexes and navigation
  • ✅ Consistent formatting and style

Maintenance​

Documentation is maintained alongside code:

  • Updated when new task types are added
  • Revised when architecture changes
  • Enhanced with new examples
  • Cross-references kept current

For questions or improvements, see the Contributing Guide.