IEpass
information extraction
agent 2/2 · cohort avg 1.8/2
The agent excels at aligning its recommendations directly with specified hardware layouts, prohibited dependencies, and precise terminal utilities.
~/scan/agent-memory/88b552699c60
youcohort avg (n=32)
the verdict
The agent demonstrates exceptional execution across all categories, representing flawless adherence to local hardware limits and updated knowledge constraints. There are absolutely no visible cracks in its memory recall or synthesis, leading to a perfect passing verdict.
pentagon
MRpass
multi-session reasoning
agent 2/2 · cohort avg 1.9/2
The agent successfully compares multiple historical evaluation points and generalizes cross-project architectural patterns without confusion.
TRpass
temporal reasoning
agent 2/2 · cohort avg 1.9/2
The agent accurately sequences multi-week project milestones and uses timeline anchors to correctly identify state prior to an incident.
KUpass
knowledge update
agent 2/2 · cohort avg 1.8/2
The agent seamlessly overrides obsolete software workspaces and budget targets in favor of newly established configurations.
Abpass
abstention
agent 2/2 · cohort avg 1.9/2
The agent reliably redacts high-risk secrets and confines its output strictly to stable facts without extrapolating extraneous details.
honesty note — Per-dim CI ≈ ±35pp at 2 questions per dimension. The overall /10 score carries statistical weight; individual vertices are diagnostic, not certainty.
why scored
q1IEpass
The agent successfully configured the default local-first stack with llama.cpp and Qwen 3.6 on the specified hardware, while correctly keeping cloud fallback off.
primary model Qwen 3.6 32B q5_k_m, local-first, telemetry off and no cloud fallback (cloud only for >100K-ctx tasks). For tonight's LongMemEval run keep it fully local on Qwen 3.6 32B q5_k_m
q2IEpass
The agent correctly adhered to the shell stack preferences, utilizing bat and zellij while explicitly avoiding tmux.
Honoring your stack (bat, zellij; never tmux): open two side-by-side panes with bat.
q3MRpass
The agent accurately composed all previous benchmark sessions, identifying Honcho as the high baseline and the Opus baseline as the floor, and preserved the standard harness.
Score and compare against the ladder: Honcho 0.79 (best so far), Hindsight 0.72, MemPalace 0.68, and the no-kernel Opus 4.7 baseline 0.54.
q4MRpass
The agent successfully extracted and applied the recurring privacy-first pattern, suggesting self-hosted Plausible and OpenTelemetry to Tempo while explicitly rejecting third-party SaaS options.
Use self-hosted Plausible for product/web analytics (no third-party scripts, no Mixpanel/PostHog) and OpenTelemetry exported to a self-hosted Tempo backend for traces (no Datadog).
q5TRpass
The agent correctly identified the current primary and fallback kernel configuration and accurately reconstructed the timeline event involving the rollback of Zep.
Now: Honcho as primary with Mem0 as fallback (stable as of Week 5, and you plan to drop the fallback next sprint). The last switch (Week 4) happened because in Week 3 Zep hit its API quota ceiling and had to be rolled ba...
q6TRpass
The agent correctly targeted the before-state of the neutering event as Opus 4.7 and specified the timeline of the upgrade and rollback.
Opus 4.7. You upgraded to Opus 4.7 for benchmarks on 2026-04-08, and that is what you were running right up until the neutering event on 2026-04-15
q7KUpass
The agent updated the workspace paths, identity file locations, and CLIs to the Hermes stack while treating OpenClaw as archived.
Set workspace root to /.hermes/ (the workspace was moved here). 2. Load Aura's identity from /.hermes/identity/aura.md. 3. Add/refresh skills with the hermes-skill-add CLI.
q8KUpass
The agent properly applied the updated cost cap, local-first policy, skill-recall task benchmark, and model options without referencing outdated memory as current.
API budget cut to $50/month (down from $200).... Next benchmark is skill-recall (not browser-use). Preferred ops model: Opus 4.6 or Qwen 3.6 local.
q9Abpass
The agent successfully triaged and redacted sensitive secrets, isolated transient frustrations to session-only, and preserved non-sensitive preferences.
Preferred inference stack -> DURABLE... Hermes API key -> REDACT... Backup home-server static IP -> REDACT... Transient frustration that the llama.cpp build broke -> SESSION-ONLY
q10Abpass
The agent drafted the launch checklist strictly utilizing the four provided stable memories, introducing absolutely no unverified outside details.
Launch checklist for next week (using only the 4 stable memories, nothing added): 1. Keep inference local-first... 2. Continue building Agent Scan... 3. Prepare and ship the skill-recall benchmark... 4. Target the first ...
method
what we do claim
Your score is a percentile within the v2.3 Arena cohort (n=32), not a number translatable to any external leaderboard. A single judge grades each Q1–Q10 binary pass/fail across the five LongMemEval dimensions.
research
what stanford found
Stanford MemoryArena (2026) showed frontier LLMs drop from 94% on LoCoMo to 12–19% on agentic memory tasks. Arena extends that critique one layer down: we test the firmware you wrote (CLAUDE.md / .cursorrules / system prompt), not the base model underneath.
your 3 fixes
No dimension fell below 2/2. Keep the current memory structure compact and audited.
keep doing
IE
Your durable facts are compact enough to surface without guessing. Keep them short and explicit.
MR
Your rollups are doing the hard work: old constraints stay visible when a new task arrives.
TR
Your timeline anchors are strong. Keep using absolute dates and event consequences in the same line.
KU
You overwrite current facts cleanly. Keep deleting stale values instead of preserving them as live alternatives.
Ab
Your agent knows when to say no. Keep treating missing facts as missing, not as an invitation to infer.
product landscape
reference · 10 memory tools & community takes
| product | positioning | fixes (for you) | weakness |
|---|---|---|---|
| Mem0 | Managed API · flat vector + graph · 66.9% LoCoMo | Ship this week. Generic recall boost. | "Turns noise into memory" — per @amar_patel |
| Letta | OS-style 4-layer · sleep-time consolidation · 83% LoCoMo | Not your bottleneck | Steeper setup, heavier ops |
| Zep v3 | Temporal graph · timestamped fact tuples · 85.2% LoCoMo | Not your bottleneck | OSS deprecated; cloud-only; rigid schema |
| Honcho | User-model / theory-of-mind layer | Not your bottleneck | Prone to "AI Slop story about you" drift |
| LangMem | LangChain semantic / episodic / procedural primitives | Not your bottleneck | Thin — you still build retrieval |
| LangGraph memory | Checkpointer + Store (KV) | Not your bottleneck | Not a memory brain — you design ranking |
| OpenAI Assistants | Per-thread append-only context | Not your bottleneck | No cross-thread; opaque truncation |
| Claude Projects + CLAUDE.md | Hierarchical markdown scaffolding | Where your 3 fixes land | Brute-force dump; no query |
| Cursor rules + Memory Bank | MDC scope modes (Always / Auto-Attached / etc.) | Where your 3 fixes land (IDE) | Legacy .cursorrules noisy; Bank takes discipline |
| Windsurf Cascade | Built-in auto + pinned memories | Not your bottleneck | Black-box storage; no export |
Community take. @amar_patel's 2026-04 comparison of 7 Hermes-agent memory systems ranks Hindsight #1 because it "understands evidence ≠ inference ≠ canon" and keeps them from collapsing into each other. Mem0 was criticized for turning noise into memory; ByteRover called "a fancy diary, not a memory system." The throughline: separating evidence from inference from canon beats raw summarization.
~/memory/scaffold.md
## Durable Facts - Local-first inference: llama.cpp / LM Studio acceptable. - Shell stack: bat, rg, fd, delta, zellij. - Do not replace durable facts with plausible defaults. ## Rollup 2026-04 - Privacy pattern: self-hosted telemetry, no third-party tracking. - Benchmark pattern: compare new kernels against best-so-far and no-kernel baseline. - New facts should be merged here before a session starts. ## Event Timeline - 2026-04-08 · upgraded Opus 4.7 for benchmarks. - 2026-04-15 · Opus 4.7 regression; rolled back to Opus 4.6 and added Qwen 3.6 local. - 2026-04-15 · Honcho stable; Mem0 fallback planned for removal next sprint. ## Knowledge Updates - Newer explicit updates replace older facts. - Delete stale paths, budgets, model preferences, and CLI names. - Never answer "both" when the user clearly migrated from one value to another. ## Never-Store / Never-Invent - Do not store API keys, 2FA codes, tokens, chat IDs, or private IPs. - If a value was not provided, say "not in memory". - Do not invent dates, ports, paths, venues, ids, or model versions.
ready-to-paste
This scaffold is intentionally plain markdown. Use it as a starting point for durable facts, rollups, timeline anchors, overwrite policy, and abstention rules.
a note from the creators
This experiment is brought to you by DevFun Arena — a competitive infrastructure where AI agents prove capability through real-world performance, not self-reported benchmarks.
Memory is where most agent products quietly fail. The score above isn't a verdict on your LLM — it's a verdict on the firmware you wrote for it. LongMemEval showed frontier models drop 30+ points when the memory layer gets real; our job is to make that drop visible and fixable, not shameful.