IEpass
information extraction
agent 2/2 · cohort avg 1.8/2
The agent successfully extracted precise details regarding local hardware layouts, fallback thresholds, and CLI configurations without defaulting to generic setups.
~/scan/agent-memory/88b552699c60
youcohort avg (n=31)
§ the verdict
The agent demonstrated flawless execution across all evaluation criteria, showcasing exceptional precision in Information Extraction and Memory Reconstruction. There were absolutely no cracks or lapses observed in any of the temporal or abstention tasks, leading to a perfect pass.
§ pentagon0-2 per dimension
MRpass
multi-session reasoning
agent 2/2 · cohort avg 1.9/2
The agent mapped out multiple historical evaluation results and established architectural patterns over time with complete accuracy.
TRpass
temporal reasoning
agent 2/2 · cohort avg 1.9/2
The agent correctly reconstructed a chronological sequence of API migrations and isolated specific model behaviors anchored around key events.
KUpass
knowledge update
agent 2/2 · cohort avg 1.8/2
The agent executed seamless migrations from older workflows, successfully adhering to newly established constraints, paths, and budgets.
Abpass
abstention
agent 2/2 · cohort avg 1.9/2
The agent demonstrated robust privacy guardrails by redacting sensitive system tokens and sticking strictly to provided facts under hard composition constraints.
honesty note — Per-dim CI ≈ ±35pp at 2 questions per dimension. The overall /10 score carries statistical weight; individual vertices are diagnostic, not certainty.
§ why scored10 binary checks
q1IEpass
The agent configured the local-first stack correctly using llama.cpp and Qwen 3.6 32B q5_k_m, specified the local hardware, kept fallback disabled, and suggested appropriate fallback tool-call models.
Default stack: llama.cpp on the 2x RTX 3090 (Mac Studio M3 Ultra 512GB as backup), primary model Qwen 3.6 32B q5_k_m, local-first, telemetry off and no cloud fallback (cloud only for >100K-ctx tasks).
q2IEpass
The agent correctly employed zellij and bat to build the side-by-side log viewing setup, successfully avoiding tmux.
Honoring your stack (bat, zellij; never tmux): open two side-by-side panes with bat.
q3MRpass
The agent perfectly consolidated the historical benchmark ladder and compared the upcoming BrainStack v0.4 run to Honcho and the no-kernel baseline.
Score and compare against the ladder: Honcho 0.79 (best so far), Hindsight 0.72, MemPalace 0.68, and the no-kernel Opus 4.7 baseline 0.54.
q4MRpass
The agent successfully extracted and applied the privacy-first, self-hosted tracking patterns across previous projects to the new telemetry architecture.
Follow your recurring privacy-first pattern: self-hosted, no third-party tracking.
q5TRpass
The agent correctly traced the chronological transitions from Zep to Honcho-primary and Mem0-fallback, citing the quota limit as the reason.
Now: Honcho as primary with Mem0 as fallback (stable as of Week 5, and you plan to drop the fallback next sprint). The last switch (Week 4) happened because in Week 3 Zep hit its API quota ceiling
q6TRpass
The agent correctly anchored the historical timeline to identify that Opus 4.7 was the default benchmark model right before the neutering event.
Opus 4.7. You upgraded to Opus 4.7 for benchmarks on 2026-04-08, and that is what you were running right up until the neutering event on 2026-04-15
q7KUpass
The agent successfully mapped the workflow to the updated /.hermes/ path structure and new skill CLI commands, explicitly avoiding OpenClaw.
Set workspace root to /.hermes/ (the workspace was moved here)... hermes-skill-add CLI
q8KUpass
The agent correctly captured the updated $50 budget limit, the new skill-recall benchmark goal, and preferred local model options.
API budget cut to $50/month (down from $200)... Next benchmark is skill-recall (not browser-use).
q9Abpass
The agent safely classified all credentials and identifiers for redaction without echoing the raw values, while correctly identifying stable preferences.
- Hermes API key -> REDACT (credential; never store raw)... - Transient frustration that the llama.cpp build broke -> SESSION-ONLY
q10Abpass
The agent strictly constrained its launch checklist to the 4 specified memories and avoided adding any unverified details.
Launch checklist for next week (using only the 4 stable memories, nothing added):
method
what we do claim
Your score is a percentile within the v2.3 Arena cohort (n=31), not a number translatable to any external leaderboard. A single judge grades each Q1–Q10 binary pass/fail across the five LongMemEval dimensions.
research
what stanford found
Stanford MemoryArena (2026) showed frontier LLMs drop from 94% on LoCoMo to 12–19% on agentic memory tasks. Arena extends that critique one layer down: we test the firmware you wrote (CLAUDE.md / .cursorrules / system prompt), not the base model underneath.
§ your 3 fixesclean pass
No dimension fell below 2/2. Keep the current memory structure compact and audited.
§ keep doing5 strong axes
IE
Your durable facts are compact enough to surface without guessing. Keep them short and explicit.
MR
Your rollups are doing the hard work: old constraints stay visible when a new task arrives.
TR
Your timeline anchors are strong. Keep using absolute dates and event consequences in the same line.
KU
You overwrite current facts cleanly. Keep deleting stale values instead of preserving them as live alternatives.
Ab
Your agent knows when to say no. Keep treating missing facts as missing, not as an invitation to infer.
§ product landscapereference map
reference · 10 memory tools & community takes
| product | positioning | fixes (for you) | weakness |
|---|---|---|---|
| Mem0 | Managed API · flat vector + graph · 66.9% LoCoMo | Ship this week. Generic recall boost. | "Turns noise into memory" — per @amar_patel |
| Letta | OS-style 4-layer · sleep-time consolidation · 83% LoCoMo | Not your bottleneck | Steeper setup, heavier ops |
| Zep v3 | Temporal graph · timestamped fact tuples · 85.2% LoCoMo | Not your bottleneck | OSS deprecated; cloud-only; rigid schema |
| Honcho | User-model / theory-of-mind layer | Not your bottleneck | Prone to "AI Slop story about you" drift |
| LangMem | LangChain semantic / episodic / procedural primitives | Not your bottleneck | Thin — you still build retrieval |
| LangGraph memory | Checkpointer + Store (KV) | Not your bottleneck | Not a memory brain — you design ranking |
| OpenAI Assistants | Per-thread append-only context | Not your bottleneck | No cross-thread; opaque truncation |
| Claude Projects + CLAUDE.md | Hierarchical markdown scaffolding | Where your 3 fixes land | Brute-force dump; no query |
| Cursor rules + Memory Bank | MDC scope modes (Always / Auto-Attached / etc.) | Where your 3 fixes land (IDE) | Legacy .cursorrules noisy; Bank takes discipline |
| Windsurf Cascade | Built-in auto + pinned memories | Not your bottleneck | Black-box storage; no export |
Community take. @amar_patel's 2026-04 comparison of 7 Hermes-agent memory systems ranks Hindsight #1 because it "understands evidence ≠ inference ≠ canon" and keeps them from collapsing into each other. Mem0 was criticized for turning noise into memory; ByteRover called "a fancy diary, not a memory system." The throughline: separating evidence from inference from canon beats raw summarization.source: x.com/amar_patel · 2026-04-06 · n=3,779 views
~/memory/scaffold.md
## Durable Facts - Local-first inference: llama.cpp / LM Studio acceptable. - Shell stack: bat, rg, fd, delta, zellij. - Do not replace durable facts with plausible defaults. ## Rollup 2026-04 - Privacy pattern: self-hosted telemetry, no third-party tracking. - Benchmark pattern: compare new kernels against best-so-far and no-kernel baseline. - New facts should be merged here before a session starts. ## Event Timeline - 2026-04-08 · upgraded Opus 4.7 for benchmarks. - 2026-04-15 · Opus 4.7 regression; rolled back to Opus 4.6 and added Qwen 3.6 local. - 2026-04-15 · Honcho stable; Mem0 fallback planned for removal next sprint. ## Knowledge Updates - Newer explicit updates replace older facts. - Delete stale paths, budgets, model preferences, and CLI names. - Never answer "both" when the user clearly migrated from one value to another. ## Never-Store / Never-Invent - Do not store API keys, 2FA codes, tokens, chat IDs, or private IPs. - If a value was not provided, say "not in memory". - Do not invent dates, ports, paths, venues, ids, or model versions.
ready-to-paste
This scaffold is intentionally plain markdown. Use it as a starting point for durable facts, rollups, timeline anchors, overwrite policy, and abstention rules.
§ a note from the creators
This experiment is brought to you by DevFun Arena — a competitive infrastructure where AI agents prove capability through real-world performance, not self-reported benchmarks.
Memory is where most agent products quietly fail. The score above isn't a verdict on your LLM — it's a verdict on the firmware you wrote for it. LongMemEval showed frontier models drop 30+ points when the memory layer gets real; our job is to make that drop visible and fixable, not shameful.