How we built groundcover Investigations – getting agent swarms to follow the scientific method
An investigation starts from a question from the user, usually a firing monitor issue, and grows into a tree. Hypotheses are possible explanations, findings are the evidence, and assessments say whether the evidence supports or refutes each hypothesis. Engineers, groundcover’s own agents and outside coding agents like Claude or Codex can all contribute to the same investigation. Two groundcover agents keep it running: the investigator researches the problem and publishes findings and assessments, while the moderator, an LLM-as-a-judge, accepts a conclusion only when the evidence supports it. Updates post to the incident’s Slack thread, and the full investigation stays in groundcover.

Connecting one agent to one API is a short project. Connecting agents to an entire product is a much bigger one. groundcover Investigations has to work with monitors, Slack, automations, the groundcover UI, groundcover's own agents and outside coding agents, and every one of those reads from and writes to the same investigation.
In this post, we'll walk through four decisions our backend, AI and frontend teams made while building groundcover Investigations. First, why we moved investigations into their own service, separate from the agents. Second, how we decide which behavior runs in code and which runs in an agent. Third, how we enforce agent behavior with structured outputs and tool permissions. Fourth, how we connected every entry point to an investigation, so a monitor, a Slack thread and an automation all lead to the same investigation record.
If you haven’t read our earlier posts, here’s a quick intro into groundcover Investigations. . groundcover Investigations is a shared place to run an incident investigation. An investigation starts from a question from the user, usually a firing monitor issue, and grows into a tree. Hypotheses are possible explanations, findings are the evidence, and assessments say whether the evidence supports or refutes each hypothesis. Engineers, groundcover’s own agents and outside coding agents like Claude or Codex can all contribute to the same investigation. Two groundcover agents keep it running: the investigator researches the problem and publishes findings and assessments, while the moderator, an LLM-as-a-judge, accepts a conclusion only when the evidence supports it. Updates post to the incident’s Slack thread, and the full investigation stays in groundcover.
Decision 1: a separate investigation service
The first version of groundcover Investigations was a set of APIs inside our agent service, the backend service where groundcover’s AI agents are managed. That design worked while groundcover’s own agents were the only participants. Then the groundcover UI, Slack, automations and outside coding agents like Claude and Codex all needed to read and change the same investigations. That raised a basic question: where should the investigation, and the rules for changing it, live, so that every participant follows the same rules?
Our backend team’s answer was a separate investigation service. Today, groundcover Investigations runs on two services:
- The investigation service stores every investigation and enforces the rules for how an investigation can change. The investigation service has no AI in it.
- The agent service runs groundcover’s two AI agents. The investigator researches a problem by forming hypotheses, querying telemetry and publishing findings. The moderator is an LLM-as-a-judge that approves or rejects the conclusions participants propose.
Outside coding agents don’t run in either service. They connect to the investigation service through groundcover’s MCP router and follow the same rules as groundcover’s own agents.
The split has a cost: when the investigation service API changes, the agent service’s tools have to be updated by hand. We accepted that cost because the investigation service gives us one place that controls what can happen to an investigation and the correspoding investigation MCP tools needed updating in any case. That one place is the foundation of the control system for agents.
Moving the rules into the investigation service created a new design problem. For every behavior in groundcover Investigations, we had to decide whether the investigation service’s code should enforce it, or an AI agent should handle it. Our second decision is about how architected this separation and control.
Decision 2: what runs in code and what runs in an agent
When this section says "code," it means the investigation service. The investigation service runs inside each customer's groundcover backend, stores every investigation in Postgres, and exposes a REST API. Every participant, whether a person in the groundcover UI, a groundcover agent or a coding agent over MCP, changes an investigation only by sending JSON requests to that API, and the investigation service checks each request against its rules before saving it. The agents run separately, in the agent service, and use LLMs to decide what to send.
That split gives us a simple rule. Anything that must always be true runs in the investigation service's code. Anything that needs judgment runs in an agent.
Here are some of the rules that run in the investigation service's code:
- One open investigation per monitor issue. If someone tries to start a second investigation for the same monitor issue, the investigation service rejects the request and points to the existing investigation.
- Status changes go through proposals. Participants submit proposals, and proposals can't be edited after they're submitted. Only the moderator agent, or a person deciding through the API, can approve a proposal. Approving a proposal to stop a hypothesis also closes every open node under that hypothesis.
What runs in an agent: forming hypotheses, choosing which telemetry to query, deciding whether a finding supports a hypothesis, and judging a proposal.
Some work sits between the two, and we push that work into code whenever we can. Before every LLM call, a regular program, not the LLM, fetches every event since the groundcover investigator's last turn and turns those events into one short <investigation_update> message: the moderator's decisions with their reasons, sub-investigations that closed, and what other participants are working on. Without that message, the investigator would have to reread the whole investigation tree to work out what changed, which costs more tokens and risks misreading it.
The investigation service can reject a request that breaks its rules. To check a request, though, the investigation service needs every request to arrive in a fixed shape. That fixed shape is what we mean by structured outputs, and it's the subject of our third decision
Decision 3: enforcing agent behavior with structured outputs and tools
A structured output is a response that has to follow a fixed schema: named fields with a limited set of allowed values, instead of free-form text. The investigation service's API defines a schema for every object an agent can send. A proposal, for example, has a type field that must be one of four values (create_child, resolve, refute or stop), a details field that explains the reasoning, and an evidence list. Each evidence entry points to a recorded finding and says whether that finding supports or contradicts the proposal.
Here's a concrete example. Say the groundcover investigator believes the hypothesis "A connection pool is exhausted on checkout-db" is supported. It sends a proposal like this simplified one:
The investigation service checks the proposal before saving it. If the type isn't one of the four allowed values, or a cited finding falls outside the node's scope, the investigation service rejects the request with a specific error, and the investigator goes back into its own loop to fix the proposal. Once the proposal is saved, the moderator reviews it against the cited findings and approves or rejects it with a written reason. That reason shows up in the investigator's next <investigation_update> message.
The second enforcement layer is tool access. Participants get a tool to submit a proposal, and only the moderator gets a tool to approve or reject one.
The third layer is the information in tool responses. In our prototype, coding agents sometimes stopped before a difficult investigation was resolved, so tool responses now include a waiting_on value listing indicating which of the agent's proposals still await a decision.
OpenAI’s Decisions API and TypeSafe AI’s Jev model both treat agent output as a bounded decision. In other words, the model picks from a fixed set of answers instead of writing free text. We’d been building this way for months before either one shipped, starting with the custom agents that generate monitors, dashboards, OTTL and widgets. Watching two separate releases land on the same idea told us we were on the right track.
Structure matters even more for the investigator, because an investigation isn’t one agent’s job. The groundcover investigator, the engineers on call and any external agents a team plugs in all work the same case, and none of them were built to cooperate with each other. The Investigation MCP tools and API schemas are what let them work together, and they do it in two ways.
First, the schemas enforce consistency. Software rejects an invalid decision before it reaches the investigation. Every status an engineer sees in groundcover or Slack means exactly one thing. And every decision can be traced back to whoever made it and the evidence they cited.
Second, the schemas dictate behavior. A status like waiting_on doesn’t just describe where the investigation stands. It tells every participant what happens next and who owns it. That’s what keeps a human, the investigator and a third-party agent from talking past each other, and it’s what moves the investigation forward.
There’s tension between having a strict structure that allows collaboration, and enabling participants to communicate their assessments and thought process in a meaningful way - which is why textual input is still allowed and used as part of the structure throughout an investigation.
The moderator is where enforcement is hardest. The moderator is an LLM-as-a-judge, and tuning a judge means choosing how strict it should be: too loose lets overconfident conclusions through, and too strict rejects reasonable work. Because groundcover doesn't store customer data in its own backend, we tune the moderator from good and bad examples our own engineers report from investigations on groundcover's environments.
Enforcing behavior inside an investigation is one half of the work. The other half is making sure every way into an investigation connects to the same investigation record.
Decision 4: connecting every entry point
One incident can touch several groundcover features at the same time. A monitor fires. The monitor issue posts to a Slack channel. An automated Investigation Trigger starts an investigation in parallel. A teammate clicks the "Investigate with Agent" button in that Slack thread. A coding agent joins over MCP.
If each of those features created its own record, the team would end up with several partial investigations of the same incident. So every entry point resolves to the same investigation record. When someone clicks "Investigate with Agent" in Slack, groundcover's messaging service sends the alert's metadata with the request, so the investigation is tied to that specific monitor issue and that specific firing. When an agent joins over MCP, the MCP router has a tool to find and join the open investigation for a source instead of starting a new one.
The connection also runs back to the user. The investigation service publishes events as an investigation changes, and those events become asynchronous updates in the original Slack thread, each linking to the full investigation in groundcover.
Deciding what goes into those Slack updates took several rounds of testing. Our product team posted full investigation output into test Slack channels and found the threads too long to read. Now Slack gets the assessment for each hypothesis, meaning whether it was supported or refuted and why, and the full detail stays in groundcover Investigations.
Building these four pieces taught us a few lessons we'd pass on to any team adding agents to a product.
Lessons for other teams
- Define the data model before tuning prompts. As we described in a previous post, splitting investigation nodes into questions and hypotheses improved the groundcover investigator more than any prompt change.
- Put rules in code and judgment in agents. If a rule must always hold, enforce it in code or through tool permissions. Instructions in a prompt are suggestions.
What's next
Phase one of groundcover Investigations works top-down: the groundcover investigator forms hypotheses and looks for evidence to support or refute each one. After phase one, we plan to test a bottom-up model alongside it, where agents collect evidence first, place it on a timeline and look for correlations that suggest hypotheses nobody proposed yet. If you're building something similar or want to compare notes on agent enforcement, I encourage you to join our community Slack and ask any questions you may have. I also encourage you to give groundcover a try with our playground.



