How NEXUS ran 1,000 concurrent research tasks with zero dead letters at 126 tasks/sec

The problem

Agent chat is easy to fake and hard to trust. What I wanted to build instead was knowledge work: a system that plans an investigation, decomposes it into tasks, runs those tasks in parallel, checks the evidence they return, and updates what it knows, all without losing a task or corrupting its own record of what happened. That means two hard problems at once, a knowledge graph that can answer "what do we actually know" and a distributed runtime that can execute many tasks concurrently without silently dropping or double-applying one of them. I built NEXUS to keep those two problems apart so each could be tested on its own terms.

The mechanism

NEXUS splits into two components with one boundary between them. nexus_knowledge ingests sources, builds a knowledge graph, and exposes hybrid retrieval, lexical, vector, entity, and graph search combined through retrieval/hybrid.py, plus GraphRAG evidence extraction in retrieval/graphrag.py. nexus_runtime is provider neutral: it plans, decomposes, and executes tasks, then proposes updates back to the engine. The runtime never opens the engine's database directly, it only calls through the knowledge-service boundary defined in nexus_knowledge/service/engine.py.

Inside the runtime, distributed/coordinator.py owns the task graph and hands ready work to distributed/scheduler.py, which enforces backpressure, concurrency limits, and retry policy. Tasks sit in distributed/queue.py until a distributed/worker.py with a matching capability leases one through distributed/worker_registry.py. Delivery is at least once: a completed idempotency key is never rescheduled, and a lease that expires because a worker died gets requeued by the coordinator rather than left stuck. State is checkpointed to a durable store so a coordinator restart reloads in-flight work instead of losing it.

The benchmark that produced the headline number runs the real Coordinator, Worker, scheduler, TaskQueue, and an atomic in-memory TaskStore, with a deterministic UTC clock so the run reproduces the same counts every time. It submits 1,000 independent agent.execute tasks to 10 workers.

Metric	Result
Tasks	1,000
Workers	10
Scheduling cycles	100
Successful	1,000
Dead-lettered	0
Retries	0
Wall-clock throughput	126.40 tasks/s

Reference run on 8 August 2026, CPython 3.12 on macOS.

What it did

All 1,000 tasks succeeded, none were dead-lettered, and none needed a retry, across 100 scheduling cycles and 10 concurrent workers. Wall-clock throughput came out to 126.40 tasks per second on that reference run. That last number is machine and Python-version dependent, the doc that reports it says so directly, so I treat it as a snapshot of one environment, not a target. What is deterministic and worth trusting is the correctness side: the task count, the zero dead letters, and the scheduling cycles, because the benchmark drives a manual clock rather than wall time for anything but the final throughput figure. The knowledge engine's own evaluation suite, run over a synthetic labeled corpus, is separate and reproducible on every run by design.

What I learned

Keeping the boundary one directional, runtime calls engine, engine never calls back, made both halves independently testable and let me change the scheduler without touching retrieval at all. The harder lesson was about what "at least once" delivery actually costs: it pushes the exactly once guarantee onto whoever consumes the idempotency key, not onto the queue. I had to make that explicit in the worker contract instead of assuming the queue would absorb it. Separating what a benchmark proves (correctness, counts, scheduling behavior) from what it merely measures on one machine (wall-clock throughput) is a distinction I now write into the benchmark doc itself, not just into how I talk about the number afterward.

Sources

1. docs/distributed-benchmark.md, 2026-08-08
2. README.md, 2026-09-04
3. docs/distributed-execution.md, 2026-09-04
4. pyproject.toml, 2026-09-04
5. src/nexus_runtime/distributed/ (coordinator.py, scheduler.py, queue.py, worker.py, worker_registry.py, store.py), 2026-09-04
6. src/nexus_knowledge/ (service/engine.py, retrieval/hybrid.py, graph/graph.py), 2026-09-04