SQLTasks

SQLTasks is a task manager/orchestrator built using primarily PL/pgSQL procedures. All state is stored directly in the PostgreSQL database of your app.

The code here comes in two parts:

• sqltasks-sql: The standalone SQL files (migrations)
• sqltasks-py: A CLI and Python runner library to run tasks from Python

The overwhelming majority of the logic is in the SQL files, so you can use SQLTasks from any programming language that can connect to PostgreSQL. The table structure is simple and the functions have ample comments. You can and should query the tables directly to get task status, history, etc.

I encourage building your own runner in your language of choice, and even directly copying SQLTasks into your own codebase if you want to avoid external dependencies.

Features:

• Leaderless by default
• Retries
• Priorities
• Task dependencies
• Input, output and metadata as JSONB
• Separation of tasks and attempts
• Specify task to start when a task fails (error handler)
• Task-wide configuration and per-task overrides
• Schedule tasks to run in the future
• Task timeouts
• Persisted task events
  • Durable task execution: This will fully depend on the worker to implement, based on persisted events.
• Configurable garbage collection of old tasks
• Task claiming strategies
  • Poll-based
  • Listen/notify
  • Leader election + assignment by leader
• Task schedules with cron syntax (will require an external scheduler process, probably a part of the sqltasks-py)