The problem

Most AI tools are single player: one person, one chat, one context window. A team making a real decision, migrate the auth service or not, rewrite the billing module or not, does not work that way. Three or four people need to see the same conversation, branch out specialists to check different angles, and end up with one artifact everyone can point to later and say this is what we decided and why. Chat logs do not survive that. Nobody can prove afterward which output fed which claim, or whether the log was edited after the fact.

I built XYZZY to give a room of humans and AI agents a shared, durable place to have that conversation: one event history, one set of artifacts and tasks, and a decision record that keeps its evidence chain intact.

The mechanism

A room holds an ordered event log. Every action, a message, an agent run, an approval, is appended as an event, never mutated. Agents can branch: several specialists run in parallel on the same prompt, a human includes or excludes each output, and only the included ones become Claims. A Decision links to the Claims it rests on, and each Claim links back to the exact AgentOutput that produced it, so a synthesis is inspectable down to the run behind it. The event log itself is hash-chained: each event's hash depends on the one before it, so an edited or deleted row breaks every hash after it.

The server is a single FastAPI process. A service layer holds the state machines, input validation, and authorization; below it, a RealtimeHub does pub/sub and queued delivery over WebSocket while a NexusAgentBridge drives agent execution, budget limits, and pause and resume. A repository layer of 44 typed repos handles atomic event sequencing against SQLite in WAL mode through aiosqlite. NEXUS, a separate agent runtime, is optional underneath: when it is unavailable the bridge calls the configured model provider directly, and without an API key it emits a clearly labelled simulated output so the collaboration mechanics stay testable without pretending placeholder text is real analysis.

What it did

The seeded run recorded in docs/benchmarks/multiai-seeded-run-2026-08-20.md exercised the served UI against file-backed SQLite: three specialists launched concurrently, three completed outputs persisted, two included and one excluded, and a Decision Brief published with both included claims drilling down to their exact source AgentOutputs. Two consecutive reload and re-entry cycles restored the same three outputs, the same 2 included and 1 excluded review state, and exactly 30 visible events each time. Against that same run, 100 atomic file-backed selection writes came back at 1.412 ms p95, with the reopened log retaining 100 unique, contiguous events.

The honest caveat, stated in that same document: this is a single local run on one developer machine with no concurrent load, and the benchmark's own stated goal, a blind comparison against a real three-person ChatGPT shared Project, has not been won yet. Both halves of that comparison, a signed-in shared Project and a live model-backed XYZZY run, are still pending, and the document counts that as a loss until they exist.

What I learned

Hash-chaining the event log looked like the hard part going in and turned out to be the easy part: it is a function of the previous hash and the new event, checked by one CLI command. The harder design problem was making the simulated path, the one that runs without an API key, honest rather than convenient. It would have been simple to let a placeholder string sit quietly among real outputs. Labelling it conspicuously as simulated everywhere it appears, in the UI and in the evidence chain, forced every other part of the system, review, publishing, drill-down, to work the same way whether the content behind it is a real model call or not. That discipline is also why the benchmark document above admits its own comparison is not yet won instead of rounding up.

The problem

A search engine's real mechanics, an inverted index, BM25 scoring, link based authority, consistent hashing, gossip membership, log replication, are usually hidden behind a database or a managed service before an engineer gets to build them by hand. I wanted to build all of it myself: one process by default, several when I choose, without Lucene, Elasticsearch or Kafka doing the interesting part for me. I built MiniGoogle, a search engine you run yourself: crawl pages into it, query them over HTTP or the bundled web UI, and, when you need it, run several nodes that place documents by consistent hashing, track each other by gossip and replicate a key value store by Raft. Java 21, one jar, one Docker image.

The mechanism

A STANDALONE node does everything in one JVM. RestServer, built on com.sun.net.httpserver with no framework underneath, exposes POST /api/v1/crawl and POST /api/v1/search. A crawl request runs through com.minigoogle.crawler, its CrawlWorker and jsoup based parser, appends the parsed page to a JSONL file and rebuilds com.minigoogle.indexer's inverted index. A search request reaches com.minigoogle.query's lexer, parser and executor, retrieves candidate postings, and hands them to com.minigoogle.ranking, which scores by BM25, blends in PageRank and, with semantic search on by default, merges lexical and vector candidates before a learning to rank model reorders the top results.

Running as CLUSTER adds three subsystems beside that same index and API. com.minigoogle.cluster.ConsistentHashRing decides which nodes own a crawled document. com.minigoogle.cluster.GossipProtocol keeps membership current: a silent peer becomes SUSPECT, then DEAD, and leaves the ring. com.minigoogle.cluster.RaftConsensus replicates the cluster's key value store through leader election, AppendEntries and snapshots, and never reads the ring or membership once its own configuration is committed.

Retrieval quality rows: seed 42, 320 document corpus, 16 judged queries. Latency rows: a separate 3200 document corpus, 500 queries after a 100 iteration warmup, on one otherwise idle developer machine. Both from Benchmark.md, measured 2026-08-15.

What it did

On the 320 document seeded corpus, turning on hybrid retrieval and the learning to rank reranker moved NDCG@10 from 0.6929 to 0.7477 and MAP from 0.2750 to 0.7877 against the same 16 judged queries. On the separate 3200 document corpus used for timing, standalone search latency dropped from a p50 of 36.26 ms to a p50 between 4.98 ms and 9.59 ms, and p99 from a range of 62.70 ms to 63.79 ms down to 14.82 ms to 32.66 ms, after a 100 iteration warmup.

The honest caveat, stated in Benchmark.md itself: this is a seeded 320 document corpus with 16 judged queries on one idle developer machine, not a public BEIR leaderboard number, and the latency rows are described in the same document as bimodal and range based rather than single deterministic values. The document's own README also carries a second, larger relevance run: pure BM25 on BEIR scifact, 5,183 documents and 300 judged queries, reaching NDCG@10 of 0.6746, a separate measurement on a different corpus that I am not blending into the headline number above.

What I learned

The gap between the quality numbers and the latency numbers taught me that a search engine has two nearly independent report cards, and improving one tells you nothing about the other. Hybrid retrieval and reranking are what moved NDCG@10 and MAP; neither touches how fast a single query returns. The latency drop came from a separate pass through the ranking stage, isolated benchmarks show its 5,000 candidate, top 20 case going from a p50 of 1205.35 ms to 10.42 ms, work that would have looked like nothing on a relevance benchmark and everything on a latency one.

Running the CLUSTER mode end to end also taught me where the three distributed pieces actually meet. Gossip and the hash ring both reason about which nodes exist and what they own, so it is tempting to let them share state. Keeping GossipProtocol responsible only for liveness and ConsistentHashRing responsible only for placement, with Raft never consulting either once its own log is committed, is what let me stop a node, watch it declared SUSPECT then DEAD, restart it, and see repair reconverge the ring without touching a single line of the replicated key value store.

The problem

I wanted a message broker I owned from the log format up, not a hosted queue I only configure. Most teams reach for a hosted broker before they know whether they need clustering at all, and pay for guarantees they never exercise. I set out to build the single-node case first: an event broker with a durable, checksummed log, a gRPC API, a CLI and a public Go client, for services that need ordered replay without running a cluster.

The mechanism

Pulse follows a plain layering. A caller drives it through pulse-cli or through pkg/client, the public Go client both the CLI and any external Go program depend on. Both talk gRPC to internal/adapters/grpc, which exposes the Broker and PubSub services and does nothing but translate wire types. The application layer, internal/application/services, holds the actual broker logic: topic management, publish and subscribe, and the compaction and retention sweepers. Underneath, each partition is one durable log in internal/infrastructure/storage/engine/log, backed by append-only segments and a sparse offset index, with every batch framed and checksummed by engine/codec using a CRC-32C from engine/checksum. A torn write on crash is caught by that checksum and truncated back to the last valid batch during recovery, rather than served as a partial record.

I benchmarked Pulse against NATS JetStream with the same paired harness, bench/harness.go, one run on the same machine: same Docker Desktop, both binaries and data on the container filesystem, 3000 publishes per point, 256-byte payloads, sync-mode: every-write (fsync on every batch) on both sides. Hardware is not recorded beyond that.

What it did

At concurrency 32, 3000 publishes of 256-byte messages against Pulse's own segment log completed at 282 messages per second with a p50 latency of 113.6 ms, both fsyncing every batch. That is the row I stand behind, from assets/bench-publish.json, dated 2026-09-02. The same file shows JetStream ahead of Pulse at every concurrency level on this box under this one configuration, which puts the two brokers in the same class here, not Pulse in front of it. It is a single run, not an average: a second run on the same machine landed at 349, 435 and 445 msg/s for Pulse while keeping roughly the same relative gap against JetStream, so the absolute numbers move between runs even though the comparison holds. I show both rows rather than only Pulse's, because a throughput number with no baseline is not proof of anything.

What I learned

Building the checksum and recovery path before the gRPC layer forced the durability guarantees to be real from the start instead of retrofitted: a torn write from a crash mid-fsync gets truncated deterministically because engine/codec rejects any batch whose CRC-32C does not match, not because a later patch added a check. The other lesson was about benchmarking itself. My first instinct was to publish a single Pulse number and call it done; running the same harness against JetStream, and running it twice, is what showed the number moves by 30 percent between runs on the same idle machine. A benchmark without a baseline and without a second run is not a number, it is a guess with three significant figures.

The problem

Most workflow tools ask you to trust a graph at run time: the engine reads the JSON you drew, node by node, and interprets it as it goes. That means the thing that runs and the thing you see in the editor can drift apart mid execution, and a worker that dies partway through leaves you guessing what actually happened. I wanted a workflow platform where saving a graph produces a fixed, validated plan, where every step leaves a durable mark of its own position, and where a crash is a known state the system recovers from, not an incident you reconstruct from logs.

I built FlowOS as a self hosted, open source engine on Python 3.13: draw a graph in the editor, save it, and the engine compiles it once into the plan it actually runs.

The mechanism

Saving a workflow compiles it into a CompiledExecutionPlan in backend/src/flowforge/application/compiler.py: a validated graph, every node type resolved against the registry, and batches laid out in topological order. The executor in backend/src/flowforge/application/executor.py walks those batches and never reads the workflow JSON again, so what runs is exactly what was compiled, not a live reinterpretation.

Every step writes a checkpoint carrying a dense sequence number. When a worker dies mid run, the job is redelivered and resumes from that last checkpoint, the frontier the executor had reached; a stale worker's late write is refused because its sequence number is behind. Jobs move through a JobQueue port defined in backend/src/flowforge/application/ports.py, with an in memory adapter for development and an ARQ adapter on Redis for production, and each claim carries its own lease so an acknowledgment from the wrong owner is ignored. The four packages under backend/src/flowforge, domain, application, infrastructure, presentation, depend inward only: the domain package imports nothing from FastAPI, SQLAlchemy or Pydantic.

What it did

The backend collects 1,791 tests under backend/ (python -m pytest --collect-only -q, run from backend/ against the repo's own virtual environment). That is a test count, not a benchmark: it says the suite exercising the compiler, the executor, the checkpoint ordering and the queue adapters is large, not that any particular path is fast. README.md names the guarantees those tests carry: a run resumes from its last checkpoint after a worker crash (test_scheduler.py::test_resume_continues_after_worker_crash), a crash before an acknowledgment redelivers the job while a crash after it does not (test_worker_restart.py), and a resumed run never reuses a sequence number (test_scheduler.py::test_repeated_resume_never_reuses_schedule_sequence). A separate jepsen suite drops tables and kills servers against a disposable PostgreSQL and Redis to check that two workers cannot disagree about a workflow version, and that no acknowledged enqueue is lost when Redis fsyncs and is then killed. The API also exposes 38 route declarations across auth, workflows, runs, executions, schedules, credentials, OAuth2, nodes, plugins, webhooks, audit, health and metrics, and the backend itself is 62,734 source lines of Python, both counted the same day as the test collection.

What I learned

Compiling the graph once, instead of letting the executor reinterpret raw JSON on every step, turned out to change more than performance. It meant the executor's contract with the rest of the system, what a batch is, what a dependency is, what a node's effective retry and timeout policy is, could be typed and tested without a live workflow document in scope at all. The harder lesson was in the checkpoint ordering itself: giving every step a dense sequence number is what makes resume safe, because the executor does not need to reconstruct what happened from timestamps or event order, it only needs to compare a number. That single decision is also what let a stale worker's late write be refused outright instead of needing a separate reconciliation pass, and it is the reason the crash suites can assert exact guarantees, not probabilistic ones.

The problem

Every product I build ends up needing to notify someone: an email, a Slack message, a webhook into another service. The usual answer is to wire each channel's SDK into the product directly, which means every product re-solves retries, templating and delivery tracking on its own, and a crashed worker can quietly drop a message with no record it ever existed. I wanted a platform that sits between a product and its channels, where the product calls one API and the platform is the one place that has to get delivery right.

The mechanism

NotiFly is laid out as strict clean architecture: presentation, application, domain, infrastructure, with dependencies pointing inward only, so a FastAPI router never contains business logic and the domain layer never imports SQLAlchemy. A request to send a notification goes through presentation/api into application/services/notifications.py, the Notification Engine, which validates variables, renders the per-channel template and writes everything in one database transaction: the notification row, its per-channel deliveries, an idempotency record, an audit log entry and a pending row in outbox_events. That last write is the transactional outbox. Nothing is sent from inside that transaction. A separate relay, application/services/outbox.py, polls the pending outbox rows on an ARQ cron job and enqueues the actual dispatch. If Redis or the worker process is down, the outbox rows just stay pending in Postgres until the relay comes back, so the notification is never lost between "the database says it happened" and "a channel actually sent it". The dispatcher, application/services/dispatcher.py, is the only piece that touches a provider, and it only ever calls the shared Provider interface defined in domain/ports/providers.py, never a channel directly. The six adapters that satisfy that interface live in src/notifly/infrastructure/providers/: email.py, slack.py, discord.py, teams.py, webhook.py and http.py. Adding a seventh channel means writing one adapter file and registering it; the engine, the outbox and the dispatcher stay untouched.

What it did

The platform exposes 24 REST endpoints, from /v1/apps and /v1/templates to the operations routes for inspecting notifications, deliveries and dead letters, and it ships behind those endpoints with six working provider adapters: SMTP email, Slack, Discord, Microsoft Teams, generic webhooks and a plain HTTP provider. Both numbers are structural counts, not a measured throughput or a reliability figure: I did not run a load test against this API, and there is no committed benchmark in the repo. What the outbox buys is a specific, checkable property, that a notification cannot be acknowledged as created and then silently disappear if the worker crashes between enqueuing and sending, because the outbox row stays pending in Postgres until the relay picks it up. That is a design guarantee I can point to in the code, not a number I measured under load.

What I learned

Splitting the Notification Engine from the Dispatcher, so that rendering and persistence happen in one service and only transport happens in the other, turned out to matter more than I expected: it means the worker that talks to Slack or SMTP never needs a template engine or write access to the notification's business state, which makes it small enough to reason about and safe to restart at any point. The other thing that stuck with me is that a provider is genuinely just an adapter behind a capabilities interface, not a feature with its own logic scattered through the app. Every channel-specific quirk, like Slack's block format or SMTP's message headers, stays inside its one file under infrastructure/providers/, and the dispatcher never has to know which channel it is talking to.

The problem

Agent chat is easy to fake and hard to trust. What I wanted to build instead was knowledge work: a system that plans an investigation, decomposes it into tasks, runs those tasks in parallel, checks the evidence they return, and updates what it knows, all without losing a task or corrupting its own record of what happened. That means two hard problems at once, a knowledge graph that can answer "what do we actually know" and a distributed runtime that can execute many tasks concurrently without silently dropping or double-applying one of them. I built NEXUS to keep those two problems apart so each could be tested on its own terms.

The mechanism

NEXUS splits into two components with one boundary between them. nexus_knowledge ingests sources, builds a knowledge graph, and exposes hybrid retrieval, lexical, vector, entity, and graph search combined through retrieval/hybrid.py, plus GraphRAG evidence extraction in retrieval/graphrag.py. nexus_runtime is provider neutral: it plans, decomposes, and executes tasks, then proposes updates back to the engine. The runtime never opens the engine's database directly, it only calls through the knowledge-service boundary defined in nexus_knowledge/service/engine.py.

Inside the runtime, distributed/coordinator.py owns the task graph and hands ready work to distributed/scheduler.py, which enforces backpressure, concurrency limits, and retry policy. Tasks sit in distributed/queue.py until a distributed/worker.py with a matching capability leases one through distributed/worker_registry.py. Delivery is at least once: a completed idempotency key is never rescheduled, and a lease that expires because a worker died gets requeued by the coordinator rather than left stuck. State is checkpointed to a durable store so a coordinator restart reloads in-flight work instead of losing it.

The benchmark that produced the headline number runs the real Coordinator, Worker, scheduler, TaskQueue, and an atomic in-memory TaskStore, with a deterministic UTC clock so the run reproduces the same counts every time. It submits 1,000 independent agent.execute tasks to 10 workers.

Reference run on 8 August 2026, CPython 3.12 on macOS.

What it did

All 1,000 tasks succeeded, none were dead-lettered, and none needed a retry, across 100 scheduling cycles and 10 concurrent workers. Wall-clock throughput came out to 126.40 tasks per second on that reference run. That last number is machine and Python-version dependent, the doc that reports it says so directly, so I treat it as a snapshot of one environment, not a target. What is deterministic and worth trusting is the correctness side: the task count, the zero dead letters, and the scheduling cycles, because the benchmark drives a manual clock rather than wall time for anything but the final throughput figure. The knowledge engine's own evaluation suite, run over a synthetic labeled corpus, is separate and reproducible on every run by design.

What I learned

Keeping the boundary one directional, runtime calls engine, engine never calls back, made both halves independently testable and let me change the scheduler without touching retrieval at all. The harder lesson was about what "at least once" delivery actually costs: it pushes the exactly once guarantee onto whoever consumes the idempotency key, not onto the queue. I had to make that explicit in the worker contract instead of assuming the queue would absorb it. Separating what a benchmark proves (correctness, counts, scheduling behavior) from what it merely measures on one machine (wall-clock throughput) is a distinction I now write into the benchmark doc itself, not just into how I talk about the number afterward.