Takahē’s background task system is called Stator, and rather than being a transitional task queue, it is instead a reconciliation loop system; the workers look for objects that could have actions taken, try to take them, and update them if successful.
As someone running Takahē, the most important aspects of this are:
You have to run at least one Stator worker to make things like follows, posting, and timelines work.
You can run as many workers as you want; there is a locking system to ensure they can coexist.
You can get away without running any workers for a few minutes; the server will continue to accept posts and follows from other servers, and will process them when a worker comes back up.
There is no separate queue to run, flush or replay; it is all stored in the main database.
If all your workers die, just restart them, and within a few minutes the existing locks will time out and the system will recover itself and process everything that’s pending.
You run a worker via the command
manage.py runstator. It will run forever
until it is killed; send SIGINT (Ctrl-C) to it once to have it enter graceful
shutdown, and a second time to force exiting immediately.
Each object managed by Stator has a set of extra columns:
state, the name of a state in a state machine
state_ready, a boolean saying if it’s ready to have a transition tried
state_changed, when it entered into its current state
state_attempted, when a transition was last attempted
state_locked_until, when the entry is locked by a worker until
They also have an associated state machine which is a subclass of
stator.graph.StateGraph, which will define a series of states, the
possible transitions between them, and handlers that run for each state to see
if a transition is possible.
An object becoming ready for execution happens first:
If it’s just entered into a new state, or just created, it is marked ready.
state_attemptedis far enough in the past (based on the
try_intervalof the current state), a small scheduling loop marks it as ready.
Then, in the main fast loop of the worker, it:
Selects an item with
state_readythat is in a state it can handle (some states are “externally progressed” and will not have handlers run)
Fires up a coroutine for that handler and lets it run
When that coroutine exits, sees if it returned a new state name and if so, transitions the object to that state.
If that coroutine errors or exits with
Noneas a return value, it marks down the attempt and leaves the object to be rescheduled after its