AI Products Get Stuck on the Small Things. Users Get Frustrated.
TL;DR: An AI product can handle a complicated feature and still get an ordinary piece of work wrong. We ran into this with a simple approval in Context Studio. Fixing it took both a clearer prompt and code that checked what the model left behind.
What Context Studio Is For
We’ve been building Context Studio to read project conversations in Teams and turn them into shared context, including decisions and action items. The idea is that people keep working and talking as usual. The system figures out who has taken on what, and how far the work has progressed.
An Approval Recorded Incorrectly
Then we got stuck on an approval.
Here is roughly how the conversation went:
Nina: The new version is ready. Could you approve it?
Leo: Hold on. I’ve left some comments. Please take a look.
Nina: I’ve made the changes you suggested. Can you take another look?
Leo: Approved.
A little later.
Nina: We fixed another small issue. The version has changed, so we need your approval again.
Leo: Done.
You could have much the same exchange while reviewing a document. Someone leaves comments, the author revises it, and the reviewer approves the revision. Another change may need another approval.
I expected Context Studio to get this straight: Leo had commented on the first version. He had approved the next two.
But in one run, the generated action item amounted to this:
Task: Approve the original and revised versions
Owner: Leo
Status: Done
The owner and completion status were both right. What it got wrong was what Leo had approved.
If I have to go back through the Teams conversation every time to check whether the system has recorded my work correctly, I might as well keep the record myself.
That’s a real problem for a product meant to save me that work.
We didn’t need a transcript of the entire exchange. A short action item was fine, provided it didn’t turn comments on the first version into approval of that version.
Working through the fix also challenged two assumptions about building with AI.
Adding More to the Prompt Isn’t Always the Fix
We did change the prompt. The model needed to distinguish completing a review from approving a particular version, using the surrounding conversation to work out what had happened.
But whether a merged action item still includes the messages attached to its original items is something code can check. We didn’t leave that to another reminder to the model: “Don’t forget anything.”
Code Belongs in the Harness
It is tempting to avoid code because it seems too rigid for messy human conversations. Here, we wanted some rigidity.
When the model merges action items, the code checks whether the original request and completion messages have been retained, and restores missing evidence. When it splits an item into several tasks, an approval needs a clear assignment. If it is unclear which task the approval belongs to, the split is rejected. The same approval cannot simply be copied onto every task.
The model can still decide how to organize the conversation. These checks protect the evidence already linked to the work. They don’t prove that the resulting description is accurate; that still needs checking.
Why Users Get Frustrated
A product can read the messages, generate the records, and fill every field, yet leave its user repeatedly explaining: “I only left comments.” “She was revising the same thing.” “I approved the later version.”
Why should users have to spend their time arguing with the product?
AI helped us build the feature faster. Using it in our own work exposed the small things we had assumed would need no explanation—and that the product still had to get right.
简单说： AI 能做出复杂功能，却可能卡在用户觉得不用解释的小事上。Context Studio 的一次 approval 就暴露了这样的错误。我们既改了 prompt，也让程序检查模型整理后的记录有没有漏掉证据。
Context Studio 想解决什么
AI 能把很复杂的功能做出来，却可能卡在一件用户觉得根本不用解释的小事上。我们做 Context Studio 时，就碰到了这种情况。
Context Studio 会读取 Teams 里的群聊对话，把讨论整理成团队可以继续使用的 context，包括决策和待办。我们希望大家照常聊天、照常工作，系统能从这些对话里整理出谁负责什么、事情进展到哪里。
听起来，提取负责人和完成状态，应该不算特别难。
一次 approval 是怎么记错的
结果我们卡在了一次普通的 approval 上。
这次的对话大致是这样：
Nina： 新版本准备好了，帮忙 approve 一下？
Leo： 等一下，我看到了几个问题，先留了 comments，你看下。
Nina： 看到了，我按你的意见改好了，再看一下？
Leo： Approved。
过了一会儿。
Nina： 刚才又修了个小问题，版本更新了，需要你再 approve 一次。
Leo： Done。
这跟审文档很像：你先提意见，作者修改，你确认新版本。如果之后又有改动，还可能需要再确认一次。
我以为系统会记得很清楚：初版只有修改意见，后面两个版本才分别得到了 Leo 的批准。
但有一轮生成的待办，简化后是这样：
待办： 批准初版和修改后的版本
负责人： Leo
状态： 已完成
负责人是 Leo，状态是已完成，都没错；错的是系统把“他批准了什么”记错了。
如果我每次都得翻回 Teams 聊天，检查它有没有记错，那我还不如自己记。
对于一个希望帮用户省掉这一步的产品，这个落差就很实在了。
接下来，我们得先把“记对了”说清楚。
记录怎么写可以简化，但至少要记对：Leo 没有批准初版，他批准的是后面两个版本。
这里容易掉进两个想当然。
别把每个问题都变成 prompt 问题
模型把“完成 review”和“批准某个版本”混在了一起，我们就在 prompt 里讲清这个区别，让它结合前后对话判断。但合并待办时有没有漏掉消息，是可以直接核对的事情，我们没有继续靠一句“千万别漏”来解决。
程序也是 harness 的一部分
我们用程序约束和检查模型的输出：合并待办时，核对原来的请求和批准消息有没有丢，并把遗漏的证据补回去；拆分待办时，如果一次批准的归属不清楚，就拒绝这个拆分结果。这里固定的是检查标准，模型仍然可以根据对话决定怎么整理待办。程序能查出证据有没有丢，描述是否写对还要另外检查。
为什么用户觉得沮丧
一个产品可以读消息、生成结果、把字段填满。真正用起来，却可能要求用户不断补充：“我刚才只是提意见。”“她是在改同一个东西。”“我批准的是后面那个版本。”
用户为什么还要花精力，不断跟你的产品做斗争？
AI 帮我们更快做出了功能。真正放回自己的工作里用起来，我们才看见那些原来以为不用解释、实际却必须做对的小事。