Yasser Ameur

04 / 06FlowOSPython · FastAPI · asyncio

How FlowOS backs a workflow compiler with 1,791 collected backend tests

Workflow automation that compiles the graph you draw into the plan that runs, checkpoints every step

backend tests collected
1,791
Source: python -m pytest --collect-only -q, backend/, 2026-09-04

Repository

The FlowOS editor with the Order gate example open: a node palette on the left, six connected nodes on the canvas with true and false routes out of the gate, and the inspector on the right
The editor's Order gate example, from docs/assets/hero-dark.png in the repo

The problem

Most workflow tools ask you to trust a graph at run time: the engine reads the JSON you drew, node by node, and interprets it as it goes. That means the thing that runs and the thing you see in the editor can drift apart mid execution, and a worker that dies partway through leaves you guessing what actually happened. I wanted a workflow platform where saving a graph produces a fixed, validated plan, where every step leaves a durable mark of its own position, and where a crash is a known state the system recovers from, not an incident you reconstruct from logs.

I built FlowOS as a self hosted, open source engine on Python 3.13: draw a graph in the editor, save it, and the engine compiles it once into the plan it actually runs.

The mechanism

Saving a workflow compiles it into a CompiledExecutionPlan in backend/src/flowforge/application/compiler.py: a validated graph, every node type resolved against the registry, and batches laid out in topological order. The executor in backend/src/flowforge/application/executor.py walks those batches and never reads the workflow JSON again, so what runs is exactly what was compiled, not a live reinterpretation.

Every step writes a checkpoint carrying a dense sequence number. When a worker dies mid run, the job is redelivered and resumes from that last checkpoint, the frontier the executor had reached; a stale worker's late write is refused because its sequence number is behind. Jobs move through a JobQueue port defined in backend/src/flowforge/application/ports.py, with an in memory adapter for development and an ARQ adapter on Redis for production, and each claim carries its own lease so an acknowledgment from the wrong owner is ignored. The four packages under backend/src/flowforge, domain, application, infrastructure, presentation, depend inward only: the domain package imports nothing from FastAPI, SQLAlchemy or Pydantic.

The compile then execute then checkpoint path, from the What runs section of README.md: compiler.py, executor.py, the frontier checkpoint, and the JobQueue port with its in memory and ARQ on Redis adapters.

What it did

The backend collects 1,791 tests under backend/ (python -m pytest --collect-only -q, run from backend/ against the repo's own virtual environment). That is a test count, not a benchmark: it says the suite exercising the compiler, the executor, the checkpoint ordering and the queue adapters is large, not that any particular path is fast. README.md names the guarantees those tests carry: a run resumes from its last checkpoint after a worker crash (test_scheduler.py::test_resume_continues_after_worker_crash), a crash before an acknowledgment redelivers the job while a crash after it does not (test_worker_restart.py), and a resumed run never reuses a sequence number (test_scheduler.py::test_repeated_resume_never_reuses_schedule_sequence). A separate jepsen suite drops tables and kills servers against a disposable PostgreSQL and Redis to check that two workers cannot disagree about a workflow version, and that no acknowledged enqueue is lost when Redis fsyncs and is then killed. The API also exposes 38 route declarations across auth, workflows, runs, executions, schedules, credentials, OAuth2, nodes, plugins, webhooks, audit, health and metrics, and the backend itself is 62,734 source lines of Python, both counted the same day as the test collection.

What I learned

Compiling the graph once, instead of letting the executor reinterpret raw JSON on every step, turned out to change more than performance. It meant the executor's contract with the rest of the system, what a batch is, what a dependency is, what a node's effective retry and timeout policy is, could be typed and tested without a live workflow document in scope at all. The harder lesson was in the checkpoint ordering itself: giving every step a dense sequence number is what makes resume safe, because the executor does not need to reconstruct what happened from timestamps or event order, it only needs to compare a number. That single decision is also what let a stale worker's late write be refused outright instead of needing a separate reconciliation pass, and it is the reason the crash suites can assert exact guarantees, not probabilistic ones.

Sources

  1. README.md, 2026-09-04
  2. DESIGN.md, 2026-09-04
  3. backend/src/flowforge/application/compiler.py, 2026-09-04
  4. backend/src/flowforge/application/executor.py, 2026-09-04
  5. backend/src/flowforge/application/ports.py, 2026-09-04
  6. python -m pytest --collect-only -q, run from backend/, 2026-09-04
  7. grep -rE "@router\.(get|post|put|delete|patch)" --include=*.py backend | wc -l, 2026-09-04
  8. find backend -name "*.py" -not -path "*/.venv*" | xargs wc -l, 2026-09-04