Introduction
Letting an AI agent handle investigation and proposals — up to here, you can move forward with enthusiasm. But "should the AI execute those proposals automatically?" is another matter. If the AI started rewriting the site or settings on its own, you'd lose sleep over it.
That's why I adopted a human-in-the-loop design — literally, a human inside the loop. An AI proposal is executed only after a human reviews and approves it. This article explains that approval flow and the Slack-based Q&A mechanism.
Why Human Review Is Mandatory
Why We Don't Let AI Execute
AI proposals are useful but not perfect. Misreading data, missing context, overlooking seasonal factors — the AI can confidently propose things a human would immediately recognize as wrong.
An off-target proposal gets applied as-is, and the site or settings change unintentionally. You notice the problem only after the impact lands, and isolating the cause is hard.
Odd proposals are stopped at the approval stage. Only what a human accepted gets executed, so the reason and responsibility for a change are always clear.
AI's execution power is attractive, but in business, "not making mistakes" outranks "speed." The small effort of inserting human review is a necessary cost for continuing to automate with confidence.
Separating "Proposal" from "Execution"
In this system, the agent's authority is limited to registering proposals. Execution-type operations like changing the site or rewriting settings are never permitted to the agent.
Don't hand execution rights to the AI
All the agent can do is "register a proposal to the business hub's API." The authority needed for execution stays on the human side. This separation of authority is the foundation that keeps human-in-the-loop from being a formality.
Separate proposal and execution structurally, and even if the agent runs amok, the impact stops at "an odd proposal gets registered."
Rejection Is a Real Decision Too
Talk of an approval flow tends to focus on approvals, but rejection is just as important.
Rejected proposals are recorded with a reason. Reasons like "this is a seasonal factor, no action needed" or "already handled by a separate initiative" become valuable material for reviewing the agent's accuracy. Accumulating human judgments reveals which proposals land and which miss.
The Approval Flow in Practice
The Proposal Lifecycle
A registered proposal is processed through a series of states.
The agent registers a proposal to the business hub (status: unreviewed)
A new proposal is announced on Slack
An admin checks the content and evidence in the admin screen
Approve to move to execution, or reject and record with a reason
A human executes the approved proposal and records the result
Because you can list which state each proposal is in on the admin screen, "oversights" and "double handling" are less likely.
Decision Material in the Admin Screen
For an admin to decide approve or reject, the decision material has to be in place.
The supporting data especially matters. Because there's numeric backing, you can approve with confidence: "yes, this really should be addressed."
Asynchronous Dialogue via Slack
The Path From Notification to Approval
Honestly, opening the admin screen daily doesn't last. So I set up a Slack-based path.
New proposals are announced on Slack, and staff notice them while doing their morning Slack check. The notification includes the proposal's gist, and you jump to the admin screen only when you want more detail. Simply changing "go look deliberately" into "it flows to you" dramatically improved how well reviews stuck.
The Agent's Q&A Flow
What I love about this system is the mechanism where the agent can ask humans questions.
During investigation, it asks on Slack about things it can't judge (e.g., is this dip due to the sale ending?)
Reply on Slack whenever there's a free moment
Apply the gained context to future investigations
The agent can now fill in background only a human knows — internal circumstances, recent campaigns, store conditions — and the proposals hit the mark more often. And because it's asynchronous, humans answer at their own pace. That ease is the trick to keeping it going.
Designing to Prevent Notification Fatigue
Even useful notifications get ignored if there are too many — the "crying wolf" state.
Compete on notification quality
Cap the number of proposals arriving per day, and for low-priority observations, record them without notifying. Protecting the trust that "what flows into Slack is worth reading" is the key to keeping the system alive.
Notification design is operational wisdom more than technology, but for keeping the human-in-the-loop turning, I feel it's as important as the approval flow itself.
Summary
We covered the human-in-the-loop design of proposal → approval → execution.
- Separate proposal and execution — don't give AI execution rights; make human approval mandatory
- Present decision material — attach evidence, and record both approvals and rejections
- Asynchronous dialogue on Slack — run reviews smoothly with notifications and Q&A
How the proposals riding this flow are created is covered in "Designing Research Prompts Across Multiple Data Sources," and the agent's overall mechanism in the hub article "A Daily AI Research Agent."