Comparison¶

This comparison asks a specific question: how does Membrick's retrieval layer perform against another reproducible, retrieval-oriented memory system? It does not claim that every memory product or agent runtime can be reduced to one Recall@K score.

The complete scripts and machine-readable artifacts live in the repository's comparison/ directory.

For a dedicated breakdown of the current POST /search, POST /add, POST /add/batch, and mixed read/write server measurements, see HTTP server benchmarks.

Current HTTP throughput at a glance (2026-10-05)¶

These are three different workloads and their rates are not interchangeable. Read throughput counts completed POST /search requests against a seeded corpus. Write acknowledgement throughput counts records accepted by the durable staging journal. Completed write throughput includes the final flush, so it accounts for accepted work that became searchable before the run ended.

The read-only, single-write, batch-write, and mixed-load figures are different workloads and should not be compared as if they ran together. The batch rate is not sustainable because admission rejected excess requests. Raw artifacts: read layout/version run, direct /add run, 128-record default-schedule run, and concentrated sparse mixed-load run.

System selection and scope¶

We select comparison systems using four requirements:

1. The project must be available for local, reproducible evaluation.
2. It must expose a retrieval layer that returns ranked memory records.
3. It must run against the same dataset, query set, and scoring formulas.
4. The result must not depend on a hosted-only service or undisclosed model configuration.

This scope compares retrieval layers. Full agent runtimes should instead be compared with end-to-end tasks that measure answer accuracy, memory decisions, token use, and latency.

“Yes” and “No” describe the standard configuration used for a fair semantic memory comparison. A project may provide additional backends, but those must be documented and tested separately rather than mixed into this result.

Selected comparator: AgentMemory¶

AgentMemory is the selected comparator because it meets the retrieval-layer requirements without requiring a model or embedding provider in its BM25 mode. Both systems can therefore be evaluated on the same 500-question LongMemEval-S set using the same any-hit Recall@K, MRR, and NDCG@10 scorer.

The comparison uses AgentMemory's local POST /agentmemory/smart-search endpoint and its BM25 result artifact. Membrick uses local POST /search with the same query set and top-k cutoffs (5, 10, and 20). Hosted Mem0 results and agent-runtime results are intentionally excluded because they answer a different evaluation question.

Detailed comparison¶

Retrieval quality¶

The shared retrieval track contains 500 LongMemEval-S questions. The table reports any-hit recall: the percentage of questions where at least one correct answer session appears in the top K results.

The full scorer and per-question outputs are available in comparison/results/retrieval-longmemeval-500.json.

HTTP search load¶

The normalized service run used 23,366 records, 100 requests per cell, top_k/limit 20, and a keyword query on the same local machine. The Membrick rows are the historical v0.1.9 single-index baseline. They predate the v0.2.0 Axum/snapshot refactor and do not describe the current release.

In this historical run at concurrency 10, Membrick delivered about 5.6× the throughput. This measures service behavior, not retrieval quality; the endpoint implementations and corpus contents are not identical. The full run is in latency-23366.json. There is no separate throughput run against the exact v0.2.1 release artifact.

v0.2.0 layout throughput¶

The current uncached layout run is recorded in throughput-layout-v0.2.0.json. It used release mode, 23,366 records, 100 requests per cell, top_k: 20, and the query deployment configuration system decision. The machine was an Intel Core i7-7700K with 8 logical CPUs running Ubuntu 22.04.5 LTS (kernel 5.15.0-177-generic) with Rust 1.94.1.

These are local uncached measurements. The single, global, and segment runner commands are documented in comparison/README.md.

Historical v0.2.0 macOS rerun (2026-09-17)¶

A fresh rerun was performed on 2026-09-17 against the current server build. It used the same 23,366-record corpus, 100 requests per cell, top_k: 20, keyword query, and disabled query cache. The corpus was seeded through POST /add/batch and the search measurements used the standard comparison/http_latency.py client through uv.

The machine was a MacBook Pro (Mac17,9) with an Apple M5 Pro chip (15 cores) and 24 GB RAM, running macOS 26.6.2. The toolchain was Rust 1.96.0, Cargo 1.96.0, and uv 0.12.7. These results are local uncached measurements and should not be compared directly with the Ubuntu results above without matching the hardware and software environment. The provider MCP watcher is not initialized by this standalone HTTP benchmark; the run validates the current server build but is not an isolated watcher-overhead measurement.

Tantivy 0.25.0 throughput verification (2026-10-04)¶

The current dependency remains Tantivy 0.25.0. This matched five-run release comparison used 23,366 records across five sessions, 1,000 measured requests and 100 warm-ups per concurrency cell, top_k: 20, and the same keyword query. The table reports median throughput across runs:

The 0.26.2 upgrade had higher measured throughput, but the 500-question LongMemEval rerun showed lower retrieval quality, so this project stays on 0.25.0. These rates describe this specific machine and workload; they are not directly comparable with the shorter 100-request v0.2.0 runs above. Full measurements, latency percentiles, binary hashes, and protocol are in throughput-tantivy-upgrade-2026-10-04.json and the Tantivy verification notes.

Current single-index optimization diagnostic¶

On 2026-10-03, a single-index build with the current query-path optimizations measured 322.59 req/s at C=1 and 1,723.67 req/s at C=10. It used 23,366 records, 100 measured requests and 10 warm-ups per concurrency level, disabled query caching, and used the same keyword query. This was one run from a dirty working tree on Darwin arm64; the exact hardware model was not recorded. Treat it as a diagnostic for that source state, not as a release result or a direct comparison with the five-run, cold-start v0.2.0 table. Details and raw values are in throughput-single-index-2026-10-03.json.

Routed segment throughput and Rayon investigation (2026-10-04)¶

To isolate the C=10 gap, I reran one segment and 23 session segments under a matched workload: 23,366 records, the same keyword query and top_k: 20, 1,000 measured requests and 250 warm-ups per concurrency cell, and five repetitions. The multi-segment route selected five segments. The table reports median throughput and median p50 latency across runs.

The route scorer is not the main cost. In a separate instrumented profile at C=10, query-term preparation had a 0.012 ms median, route scoring 0.066 ms, and selected-segment execution 6.003 ms. The profile has 750 samples per stage, including 250 warm-ups and 500 measured requests, and must not be used as a throughput measurement because the timers add overhead. It also corrects earlier instrumentation descriptions: filter_ms times document filtering, while the old default_segment_execution_ms included coordinator setup as well as query execution.

We tested whether limiting each request's Rayon fan-out would improve C=10 throughput. Under the same five-segment route and five-run workload, smaller caps made throughput worse:

Rayon uses a shared, bounded worker pool. Ten HTTP requests can enqueue segment tasks concurrently, but do not create 50 Rayon workers. The evidence does not support changing pool size based on HTTP concurrency or shrinking the per-request fan-out. The expensive part is doing multiple segment searches per request.

We then measured route widths 1, 2, 3, and 5. Narrower routes raised C=10 throughput, but lowered Recall@5 on the 100-question LongMemEval-S diagnostic subset:

This is a 100-question diagnostic, not a basis for changing the default. Keep the current route width of five and Rayon cap of eight for now. A promising follow-up is to search two segments first and expand toward five when the results look weak; validate any change on the complete 500-question set. Compile-time sweep values were restored afterward. These measurements were made on a modified working tree, so they are diagnostics rather than release claims. Earlier trials fixed redundant corpus-wide ID allow-list construction in the filter path, then used Instruments to inspect candidate scoring and allocation. Partial ranking reduced a small ranking timer but yielded only a modest throughput change; removing candidate-ID materialization changed an internal timer but did not show a repeatable end-to-end gain. The detailed trial record and related artifacts are in the server performance notes.

Artifacts: matched one-vs-five run, fan-out and route-width sweep, query-stage profile, and LongMemEval width 1, 2, 3, and 5.

Mixed HTTP read/write load¶

The layout throughput run above measures reads after seeding the corpus. For read performance during writes, use the mixed-load runner. It seeds an isolated temporary corpus, measures a read-only control, then runs ten search workers concurrently with batch writers. Search terms vary by request to avoid reusing the same prepared-query cache entry. The report includes read/write throughput and latency percentiles, rejected writes, and a check that a record accepted during the mixed phase can be retrieved after that phase.

python3 comparison/mixed_load.py \
  --seconds 15 --readers 10 --writers 1 --write-batch 8 \
  --repetitions 3 \
  --output comparison/results/throughput-mixed-read-write-latest.json

The paired control and mixed workload run against the same server and corpus; each repetition starts with a fresh temporary index. See the latest mixed-load artifact at throughput-mixed-read-write-2026-10-04.json.

The October 4 artifact is retained as historical diagnostic evidence. Its readers used mixed-bench-user while the seed belonged to bench-user, so its 6,053 req/s control measured empty-scope queries. It cannot be compared with populated-corpus throughput. The corrected runner searches bench-user, records empty responses and measures final flush time. See the staged-write results below for the corrected workload.

Write-only diagnostic (2026-10-05)¶

One sequential writer ran with no concurrent searches against a release server using Tantivy 0.25.0 on an Apple M5 Pro. Each cell used a fresh index seeded with 23,366 records, five warm-up calls, an eight-second measured phase, and three repetitions. Batch size is the number of add requests in each /add/batch HTTP call.

Batching increased record throughput as batch size grew; 128 adds per request delivered about 27.5 times the single-add record rate, with higher request latency. No writes failed, and a final search found the last written record in every repetition. This is a working-tree diagnostic, not a release capacity claim. The raw results include environment and per-repetition details; reproduction instructions are in the comparison README.

Write path optimization profile (2026-10-05)¶

A macOS sample profile during the workload attributed about 44% of sampled write-path CPU time to rebuilding and persisting the full semantic record snapshot, and about 35% to refreshing segmented indexes and routing summaries. We changed both semantic JSON files to compact serialization while preserving the documented chunk_lifecycle.json sidecar. Receipt persistence now appends only newly created receipts to receipts.jsonl; existing receipts.json files remain readable, and retries can reconstruct missing receipts from stored documents.

A full three-repetition rerun on the same Apple M5 Pro setup measured 3.91 records/s for one add per request, versus 3.42 in the earlier baseline. The median p50 fell from 293.614 ms to 255.507 ms. Batches of 128 measured 94.03 records/s versus 94.20, with p50 latency of 1,346.843 ms versus 1,340.838 ms. The single-add improvement is directional; the largest batch shows no meaningful gain. All 12 cells had zero errors and the last record was visible to search. These are working-tree diagnostics, not release capacity claims. The earlier exploratory run that skipped the lifecycle sidecar is not used as the comparison because it changed a documented persisted artifact. See the compatibility-preserving rerun, exploratory compact run, and receipt-journal comparison.

Staged write flow (2026-10-05)¶

The primary HTTP server now persists adds to a synced journal, accumulates them in the mutable store and publishes an independent read view. Defaults are a 250 ms publication trigger, 512 pending documents and a 30-second checkpoint cadence. The concurrency document explains visibility, flush, recovery and ownership.

The write-only runner separately reports durable acknowledgement rate and published record rate. Published rate includes the final flush and checkpoint; it avoids treating an unfinished backlog as completed work. The previous synchronous diagnostic acknowledged already published writes, so compare it with the new published rate when assessing completed-work throughput.

The final write-only rerun used the same 23,366-record seed, 23 session groups, five warm-up requests, eight seconds per cell and three repetitions. No build or test process ran alongside this rerun. All writes succeeded and the last record was searchable in all six cells.

Completed-work throughput increased about 10.9 times for single adds and 2.8 times for batches of 128. Final flush took roughly 0.34 seconds for single adds and 1.65 seconds for batches of 128. These short runs show the effect of accumulation and deferred checkpoints; sustained capacity needs a longer run that includes multiple periodic checkpoints.

The corrected mixed run used 23,366 seeded records, 23 session groups, ten readers, one writer, batches of eight and three paired 10-second repetitions. Its median populated-corpus read-only rate was 1,145.63 req/s, with 8.602 ms p50. During writes, median search rate was 788.59 req/s, with 11.033 ms p50 and 22.606 ms p95. Writes acknowledged at 188.00 records/s (23.50 HTTP req/s) and completed at 181.75 records/s including final flush. Median write p50 was 4.102 ms; p99 was 1,665.212 ms. Every search returned results, all operations succeeded, and the final written record was searchable in every repetition.

Readers now retain the previous generation during refresh, so refresh no longer holds their service lock. CPU contention remains: rebuilds consume CPU and the single writer pauses ingestion during publication. The high write p99 is an expected limitation of this simple sequential design, and the 250 ms trigger is not a 250 ms visibility guarantee. These are working-tree diagnostics on an Apple M5 Pro with Tantivy 0.25.0, not release capacity claims.

Raw reports: write-only and corrected mixed workload.

Reproduce the staged measurements after building the release server:

cargo build --release --bin server
python3 comparison/write_only.py --batch-sizes 1 128 --seconds 8 --repetitions 3 \
  --output comparison/results/throughput-write-only-staged-latest.json
python3 comparison/mixed_load.py --seconds 10 --readers 10 --writers 1 \
  --write-batch 8 --repetitions 3 \
  --output comparison/results/throughput-mixed-read-write-staged-latest.json

Sustained staged-write profile and rerun (2026-10-05)¶

We profiled the single mutable writer during a 65-second, one-writer/no-reader HTTP run on the same 23,366-record, 23-session corpus. The first profile showed SemanticAggregate::remove_doc consuming 5,119 of 8,367 writer-thread samples. Every changed document called it, including new documents absent from the aggregate; it scanned all posting maps even when there was nothing to remove. An early return when doc_to_chunks has no entry reduced this to one sample in the matched follow-up profile. The check also handles existing documents with zero chunks because insertion stores an empty doc_to_chunks entry.

The next profile showed repeated per-document sorting while appending semantic postings. Inserts now append and mark changed keys; the immutable index builder sorts those posting lists once immediately before publication. The profile after this change no longer showed insert_doc_state sorting in the refresh stack. Segmented catalog and routing-summary refresh remain the largest sampled costs, so publication can still stall the single writer.

The release-mode, three-repetition default-schedule run used Tantivy 0.25.0, five warm-up requests, 65 measured seconds per cell, a 250 ms refresh trigger, 512 pending documents, and a 30-second checkpoint interval. Each run flushed after measurement and found its last record through search. The earlier baseline artifact used the same schedule and corpus but only one run per cell.

At batch size 128, acknowledgement throughput improved 44% and published throughput improved 44%; p99 fell from 2,845.844 ms to 2,063.493 ms. Single adds improved by about 9%. The comparison is directional because the baseline has one repetition while the updated run has three. These remain local working-tree diagnostics, not release-capacity claims.

A separate three-repetition diagnostic set the refresh threshold to 32,768 documents to isolate amortized write cost while retaining the 30-second checkpoint. It measured 1,454.98 acknowledged and 1,299.84 published records/s for batches of 128, with 4,114.138 ms p99. This threshold is an experimental setting, not the production default; it trades publication frequency for longer visibility delays and larger refresh stalls.

Raw reports: default schedule, large-threshold diagnostic, and earlier baseline. Reproduce with the benchmark schedule flags:

python3 comparison/write_only.py --batch-sizes 1 128 --seconds 65 --repetitions 3 \
  --output comparison/results/throughput-write-only-staged-long-latest.json

Background publication diagnostic (2026-10-05)¶

After moving index preparation and checkpoint serialization to the background publisher, we ran three 65-second repetitions per HTTP batch size against the same seeded corpus. This release binary was built from working-tree changes on revision a1907ffe66ae8fb26e082845986a0b29b53b4096, with Tantivy 0.25.0, one writer, no readers, a 250 ms refresh trigger, 512-document refresh batches, and a 30-second checkpoint interval.

The batch-128 load exceeded publisher capacity: each run accepted 164,352 records, then returned 25,478–25,766 HTTP 429 responses as the pending-write limit applied. Its acknowledgement rate is therefore burst throughput, not a sustainable no-rejection rate. Publication lagged acceptance, and the final flush took about 18 seconds. Single-add runs had no 429 responses. Every run returned a successful flush and the benchmark found its last accepted record through search, but it does not yet verify every accepted ID. Treat these as local diagnostics, not a capacity guarantee or a like-for-like historical comparison.

The complete raw report records the binary SHA-256, seed size, latency values, errors, and each repetition: background publication write-only report.

The runner's completed-work rate is accepted records divided by measurement duration plus final-flush duration. It is useful for comparing the full run and drain, but it is not a direct measurement of the background publisher's steady service rate. The reports do not yet expose per-generation build throughput or the pending-document high-water mark.

Copy-on-write capture rerun (2026-10-05)¶

After sharing the source-document, record, and chunk-lifecycle maps across publication capture, we repeated the same three 65-second cells with a rebuilt release binary (SHA-256 969f7ad50da09656cb7a8b29caf3ec4a41f737d81fff398396e89cd8ffac0d02). Single-add throughput measured 225.21 accepted and 215.46 completed records/s at the median, versus 221.05 and 212.30 before the map-sharing change. Median p50/p95/p99 latency was 3.995/4.257/6.024 ms. All three single-add runs had no 429s, successful flushes, and searchable final writes. The observed throughput gain is small and should be treated as within local run variation until paired repetitions confirm it.

With 128 records per request, the median remained 2,528 accepted records/s, while median completed-work throughput was 1,927.65 records/s. This saturated the pending-write limit, with 24,992–27,110 rejected requests per run and final flushes of 17.0–23.8 seconds. It remains a burst measurement, not a sustainable write rate. Copy-on-write rerun report contains all repetitions and per-run outcomes.

The completed-work rate is accepted records divided by measurement time plus final-flush time. It does not directly measure publisher service capacity.

Temporal append and coalescing diagnostics (2026-10-05)¶

An append-only temporal-fact path avoids sorting and rebuilding existing facts when new records follow the corpus date/document order. Replacements, deletes, and out-of-order writes keep the full-rebuild fallback. Three 65-second release runs measured 224.37 accepted / 217.54 completed single records/s, effectively unchanged from the copy-on-write run. Batch-128 median was 2,528.46 accepted / 1,977.12 completed records/s, with 25,501–26,673 rejected requests. The temporal rebuild was not the dominant write bottleneck for this benchmark. Temporal append report contains the binary hash and per-run results.

We then raised the publication trigger from 250 ms / 512 documents to 1 second / 4,096 documents for three 65-second batch-128 runs. Median accepted rate was 2,583.54 records/s; completed-work rate was 2,077.75 records/s. Final flushes took 14.5–16.3 seconds, and each run still rejected 26,760–27,128 requests. Coalescing reduced rebuild overhead modestly but did not eliminate saturation. All flushes succeeded and the final accepted write was searchable. This is a diagnostic schedule, not a new default. Coalescing report records the schedule and binary hash.

The write-only harness uses the production segmented layout and seeds 23 session segments. Its default writer round-robins records across those same 23 sessions, so a 128-request publication touches every segment. A paired three-run, 15-second probe compared that pattern with writes pinned to the existing bench-session-0 segment. Round-robin measured 6,058 accepted / 3,531 completed records/s; concentrated writes measured 5,683 / 3,945. These short cells still saturated admission (about 9,000 and 6,700–6,800 rejections per run respectively), and corpus growth makes them directional only. The completed rate includes final flush. Both reports contain raw run data: round-robin and concentrated. The write-only runner now accepts --writer-session-group for this comparison.

Direct /add endpoint (2026-10-05)¶

The earlier 225 records/s single-write measurement used /add/batch with one request per HTTP call. We repeated the same 65-second, three-repetition load against the actual /add endpoint using the same release binary. /add measured 222.52 accepted records/s and 212.36 completed-work records/s at the median, with 3.996 ms p50, 4.336 ms p95, and 6.964 ms p99 acknowledgement latency. There were no 429s; every flush succeeded and the last accepted record was searchable. Both routes use the same durable staged writer, so this confirms the single-add API itself accepts about 220 writes/s under this local configuration. Acknowledgement means the journal was synced; search visibility is provided by later publication/flush. Direct /add report includes the binary hash and per-run results.

Read-heavy mixed-load checks (2026-10-05)¶

To model an infrequently changing corpus, we paired three 65-second read-only phases with three phases using ten search workers and one writer submitting one single-document add per second. Each pair used the same seeded 23,366-record corpus and fresh server process. Read throughput medians were 1,171 searches/s without writes and 1,118 searches/s with sparse writes; the median paired change was about -2%. Median search p99 changed from 16.953 ms to 17.965 ms. All 195 writes were accepted, there were no 429s, each flush completed in 0.261–0.298 seconds, and the final accepted record was searchable in every run. The per-run read throughput varied, so treat this as a directional local measurement.

We also ran one intentionally saturated probe with ten readers and an unrestricted batch-8 writer. Search throughput fell from 1,277 to 305 searches/s, 13,803 write requests were rejected with 429, and the explicit flush hit the server's 30-second HTTP timeout (408); the last accepted record was not yet searchable when checked. That overload result is not representative of one write per second, but it shows the bounded publisher correctly applies backpressure while heavy writers compete with searches.

Raw reports: sparse mixed load and saturated probe. The mixed-load runner supports --write-interval-ms to reproduce sparse rates.

After adding a fixed writer-session option, we repeated the same paired three-repetition run on the current release binary with one single-document write per second pinned to the existing bench-session-0 segment. Median read throughput was 1,209 searches/s in control and 1,249 searches/s with writes; median search p99 changed from 17.445 to 18.352 ms. All 195 writes were accepted with no 429s, flushes took 0.212–0.215 seconds, and every final write was searchable. This measured no meaningful read-throughput loss for sparse, concentrated writes. Concentrated mixed-load report records the paired data, writer-segment distribution, and binary hash.

The write-path changes also passed a retrieval control: a release-mode, single-index rerun on the same 500 LongMemEval questions with k=1,3,5,10,20 matched the Tantivy 0.25.0 control's top-10 session ranking for all 500 questions. Recall@5/10/20 was 94.2%/96.8%/97.6%, MRR 0.86965, and NDCG@10 0.85115 in both reports. The rerun and control are available as write-path rerun and Tantivy 0.25.0 control.

Reproduce the comparison¶

See comparison/README.md for setup, corpus seeding, AgentMemory instructions, and all caveats. To run the Membrick HTTP client after starting the server:

python3 comparison/http_latency.py \
  --url http://127.0.0.1:8080/search \
  --payload '{"query":"deployment configuration system decision","user_id":"bench-user","top_k":20}'

For the layout throughput run, use uv with a repository-local temporary directory on macOS. This avoids /tmp resolving through a symlink, which the server rejects for safety:

mkdir -p .benchmark-tmp
TMPDIR="$PWD/.benchmark-tmp" uv run python comparison/throughput.py \
  --mode single --no-cache

Repeat with --mode global and --mode segment for the complete comparison.

The 2026-09-17 rerun is stored in throughput-layout-watcher-rerun-2026-09-17.json. It used the 23,366-record uncached workload through uv. The standalone HTTP server does not initialize provider MCP watchers, so this confirms the current server benchmark but is not an isolated watcher-overhead measurement.