Agent Memory Is Not Recall. It Is Governance.
TL;DR: Agent memory is not about storing more context. The real problem is governance: which memory is relevant, which one has authority, which one belongs to this turn, and which one should stay quiet.
I wrote about partial signal before.
The user says:
1
"Let's do a deep dive"
But the real instruction is closer to:
1
deep dive + save to Learning/ + use the learning guide + don't be a yes-man
The agent understands “deep dive.” That is not the hard part.
The hard part is knowing which missing context should attach to this sentence. Is this a casual request? A learning workflow? A blog pipeline? A continuation of something we already designed?
A lot of memory systems answer this with more memory. Bigger context window. More RAG. More search results. More SQLite rows. More keyword matches.
That sounds reasonable. I think it is the wrong center of gravity.
The problem is not that memory is too small. The problem is that memory does not know whether it has permission to affect this turn.
Relevance Is Not Authority
A memory being retrieved only means it might be relevant. It does not mean it should control behavior.
Say I am working in Project B. A Project B memory should get a higher retrieval prior. Fine. But that does not mean every sentence I say inside Project B is about Project B.
I may be asking about a blog idea. I may be thinking through a general design. I may insert a side topic and come back later.
So folder location is a signal. It is not a verdict.
This is where many memory designs get sloppy. They treat “where it happened” as “what the user must mean.” Then project memory becomes an intent constraint.
I do not think that works.
Space answers one question: where is this memory more likely to be useful?
Authority answers another question: how strongly is this memory allowed to steer behavior?
A simple matrix helps.
1
2
3
4
5
6
7
8
9
10
11
12
13
Strong authority
↑
│
Global preference │ Global rule
"user likes shorter" │ "never delete without approval"
│
Local evidence ─────────────────┼───────────────── Project rule
"happened once in Project B" │ "this repo runs test Y"
│
↓
Weak authority
Local scope ←────────────────────────→ Global scope
This chart is not trying to explain the whole memory system. It only makes one point: scope and authority are different things.
A local event can be useful evidence without becoming a global preference.
A global rule can be authoritative even when it is not tied to the current folder.
Once those two get mixed, memory starts behaving weirdly. The agent either overfits to the current project, or it turns one local annoyance into a permanent user trait.
Both are bad.
Four Judgments
The matrix gives us two questions: space and authority.
But effective memory needs four judgments.
First: time.
When did this memory come from? Something from five minutes ago is different from something saved three weeks ago. Recent context helps with “go ahead” and “do that.” A durable preference should survive sessions. A stale preference may need to decay.
Lifecycle is not just summarizing history. It should do promotion, demotion, and decay.
Second: space.
Where does this memory belong? User-level, project-level, folder-level, thread-level, evidence archive.
Space should decide retrieval prior. It should not decide intent.
Third: frame.
What is the user doing in this turn? Continue, shift, branch, resume, or close?
I used to think “per-turn recall” was the answer. Now I think that phrase is not precise enough. The real job is per-turn frame maintenance.
A turn is not just a query. It is a small decision about what conversation we are in.
Fourth: authority.
How much can this memory steer behavior? Hard rule, soft preference, workflow hint, evidence pointer, stale candidate.
This is the part that prevents memory from becoming superstition. Just because something happened once does not mean the agent should obey it forever.
Where Current Systems Feel Different
This is also how I now understand the difference between Claude Code, Codex, and OpenClaw.
In my current use, Claude Code feels lighter. The memory surface is more readable and incremental: files, instructions, project context. You can see a lot of the layering. That is useful. But priority still feels partly driven by prompt physics: what got loaded, what is nearby, what wins attention.
Codex feels heavier. It seems to care more about extraction, summaries, evidence, consolidation, and forgetting. I do not think “heavier” is automatically bad. Memory hygiene probably should be heavy. The issue is whether the system exposes enough of frame and authority, so the user can understand why one memory acted and another did not.
OpenClaw has a different problem. It can add memory-shaped artifacts: diary, dream, wiki, vector search, decay, SQLite, keywords. Those are not wrong. In fact, I would keep most of them.
But none of those are the core design.
The core design is governance.
I Do Not Want To Reinvent Memory
I do not want to throw away RAG, SQLite, keyword search, files, or decay.
If those tools solve the storage and retrieval problem, use them. No need to make memory mystical.
What I want to add is the layer above them.
Each memory item needs metadata like:
1
2
3
4
5
6
scope
authority
source
confidence
frame_triggers
evidence
Recall should not mean “retrieve and inject.”
Recall should mean:
1
2
3
4
5
classify the current frame
choose active scopes
retrieve candidates
rank by authority
apply only what is allowed to act
Offline cleanup should also be careful.
I like the idea of a digest or “dreaming” process, but it should not simply move every memory to its supposedly true place. It should keep local evidence where it happened, then derive higher-level candidates from repeated patterns.
If I complain about one Project B annoyance once, that is local evidence.
If I hit the same annoyance across many projects, that may become a user preference.
If it becomes a high-authority rule, maybe the system should ask me before promoting it.
Promotion needs evidence. Authority needs governance.
The Point
Agent memory is not a warehouse.
It is not “store the past, search it later.”
It is a control surface.
It decides which parts of the past are allowed to act on the present.
Memory should not remember more. It should judge better.
一句话总结： Agent Memory 不是把过去都存起来等以后搜。它是一个 governance layer，决定哪些过去有资格影响现在。
我之前写过一篇 partial signal。其中说到“用户通常不会把话说完整。”
你说：
1
"Let's do a deep dive"
但你真正的意思是：
1
deep dive + 存到 Learning/ + 用 learning guide + 不要当 yes-man
Agent 听懂了 deep dive。但它不知道剩下那半句。
这不是 NLU 问题。它不是没听懂你说什么。它是不知道这句话应该接上哪一套 context。
很多 memory 系统的第一反应是：那就多记一点。更大的 context window。更多 RAG。更多 search result。听起来合理，其实不对。
问题不是 memory 不够多。问题是 memory 不知道自己有没有资格影响这一 turn。
Relevance 不是 authority
这是我现在觉得最重要的一点。
一个 memory 被搜出来，只说明它 relevant。它不一定 authoritative。
比如我在 Project B 里工作。Project B 下面的 memory 当然更可能 relevant。但这不代表我在 Project B 里说的每句话都必须解释成 Project B 的事。
我可能只是临时问一个 blog idea。也可能在聊一个 general design。也可能插一句题外话，然后马上回来。
所以 project folder 只能给 retrieval 加权，不能约束用户意图。
这就是很多 memory system 容易错的地方。它把“在哪里发生”当成“应该怎么做”。空间位置变成了行为约束。
不对。
空间只回答：这条 memory 在哪里更可能有用。
Authority 才回答：这条 memory 有多大权重。
先画一个最简单的图。
1
2
3
4
5
6
7
8
9
10
11
12
13
Strong authority
↑
│
Global preference │ Global rule
"用户喜欢短一点" │ "删除前必须确认"
│
Local evidence ─────────────────┼───────────────── Project rule
"Project B 发生过一次" │ "这个 repo 要跑 test Y"
│
↓
Weak authority
Local scope ←────────────────────────→ Global scope
这张图只讲一件事：scope 和 authority 不是一回事。
Scope 回答：这条 memory 在哪里适用。
Authority 回答：这条 memory 能多大程度影响行为。
比如 Project B 发生过一次 是 local evidence。它有用，但不能直接变成全局偏好。
删除前必须确认 是 global rule。它不只是 relevant，它应该强约束行为。
这两个维度一混，memory system 就会开始乱。
这张图先打掉一个常见错误：把 local/global 当成 weak/strong。
但 effective memory 不止这两个轴。还差两个问题。
Effective Memory 的四个轴
图里已经有两个问题：
- Space：这条 memory 属于哪里？
- Authority：这条 memory 有多大权重？
完整一点看，还要再加两个：
- Time：这条 memory 处在 lifecycle 的哪一段？
- Frame：当前 turn 到底在干什么？
所以最后不是一个二维图，而是四个判断。
1. Time
这条 memory 是什么时候来的？
几分钟前的 active context，跟三周前的用户偏好，不该用同一套机制处理。
几分钟前的东西，解决的是 continue / do that / go ahead 这种问题。它需要 recent context bridge。
一周前的东西，可能是 workflow。
几个月前的东西，可能是 durable preference，也可能已经过期。
Lifecycle 不是总结历史。它要做 promotion、demotion、decay。
2. Space
这条 memory 属于哪里？
User-level。Project-level。Folder-level。Thread-level。Evidence archive。
空间决定 retrieval prior，但不是 intent constraint。
这是我想强调的地方。你在某个 folder 里，不等于你现在一定在做这个 folder 的事。CWD 是 signal，不是 verdict。
3. Frame
这一 turn 用户到底在干什么？
是在 continue，还是 shift？是在 branch，还是 resume？是在 close 一个 topic，还是开一个新 topic？
我曾经觉得 per-turn recall 是解法。现在觉得这个说法还不够准。
真正重要的是 per-turn frame maintenance。
Session-level context 给 continuity。Per-turn frame 决定这一刻该激活哪一组 memory。
聊天会漂移。人就是这样说话的。刚才还在读文章，下一句可能去改 AGENTS.md，再下一句又回到 memory design。
如果系统只在 session 开头判断一次 frame，后面很容易错。
4. Authority
这条 memory 能多大程度影响行为？
Hard rule。Soft preference。Workflow hint。Evidence pointer。Stale candidate。
这些不一样。
“用户喜欢短一点”可以是 soft preference。
“删除文件前必须确认”应该是 hard rule。
“上次在 Project B 遇到过这个坑”只是 evidence pointer，除非它被验证成稳定模式。
如果系统不区分 authority，它就会把一次局部摩擦升级成全局人格。
这很危险。
Claude Code、Codex、OpenClaw
Claude Code 给我的感觉是 readable layering 做得不错。文件、scope、project instruction，这些都直观。它轻，响应也快。但很多 priority 还是靠 prompt physics。谁进了 context，谁写得更强，谁离当前 turn 更近，谁就更容易赢。这段后面需要回到 docs 和源码确认。
Codex 是另一个方向。Extraction、summary、evidence、consolidation、forgetting 都很重。但这个重不一定坏。Memory hygiene 本来就该重一点。问题是它也没有把 frame 和 authority 讲得足够显式。
OpenClaw 的问题不是用了 RAG、SQLite、keyword search。这些都是工具。能用就用。“术”不重要，“道”重要。这些工具有没有服务一个清楚的 memory governance？
如果没有，diary、dream、wiki、vector、decay 都只是 memory-shaped artifact。看起来像记忆，不等于真的改善未来行为。
No reinvent wheel
SQLite 可以留。Keyword 可以留。Vector 可以留。Decay 可以留。文件也可以留。我要加的是 governance 那层。
每条 memory 至少应该知道：
1
2
3
4
5
6
scope
authority
source
confidence
frame_triggers
evidence
Recall 不是搜出来就塞进去。要先判断这一 turn 的 frame，再决定 active scopes，然后按 authority 用 memory。
Offline cleanup 也不是把东西搬到“真正的位置”。很多 memory 没有唯一位置。更好的方式是保留 local evidence，再派生出 higher-level candidate。
Promotion 要有证据。高 authority 的 promotion 甚至应该问用户。
结尾
Agent Memory 不是仓库。
它不是把过去都存起来，等以后搜。它是一个 control surface。
它决定哪些过去，有资格影响现在。
Memory 不该记更多。它该判断得更好。