There is a tempting way to build an AI agent system: give every agent a tool, a shared workspace, and enough autonomy to keep the work moving. It demos beautifully. Then the agents begin to disagree, overwrite one another, repeat bad assumptions, or make changes nobody can explain after the fact.
The problem is not that the agents are insufficiently clever. The problem is architectural: we have often treated generated output as if it were an authorized action.
My new paper, Governed Process Algebra: Compositional Verification of Admissibility Boundaries for Multi-Agent Generative Systems, starts from a simpler rule:
An agent can propose a change. It should not be able to commit that change by itself.
This may sound like a small distinction. In practice, it changes what a multi-agent system can safely become.
The missing layer between an agent and the world
Language models are good at producing plausible next steps. That is useful, but it is not the same thing as being entitled to update a customer record, publish a finding, move money, call an external service, or alter the shared understanding of a case.
The paper describes a system with a clear separation of responsibilities:
Think of a newsroom. Reporters can suggest a story; they cannot silently publish it. A publication process checks sourcing, edits for consistency, and leaves a record of what ran and why. The point is not to suppress the reporters. It is to make their speed usable.
For agents, the same pattern applies. An agent submits a structured proposal to a governance kernel. The kernel checks it against the current shared state. Only if the proposal passes can the kernel make the change. If it fails, the system returns a reason the agent—or a human—can act on.
The important detail is that the shared state is not an open scratchpad. It is a governed substrate.
Five questions before anything changes
The paper uses a canonical sequence of five gates. They are deliberately practical questions, not a claim that every domain needs the same policy.
- 01Is the proposal well-formed?
Does it have the right shape, fields, and types? Can the system even understand what is being requested?
- 02Is this agent allowed to do this?
An extraction agent should not be able to approve a payment simply because it can produce a persuasive sentence.
- 03Is it grounded?
Claims need traceable evidence. A proposal should point back to the documents, data, or prior facts that support it.
- 04Does it fit with what is already known?
The change must not quietly introduce a contradiction into the shared record.
- 05Is it complete enough for this domain?
Before the system acts, does the proposal cover the required checks and preserve the relevant business rules?
This is not just “put a guardrail around an LLM.” A guardrail usually sits around one interaction. The question here is what happens when several agents are working at once, all trying to contribute to the same evolving system.
Why this becomes hard with multiple agents
With one agent, a review step can feel obvious. With a fleet of agents, the failure modes compound.
One agent may add a claim while another derives an analysis from an earlier version of the record. A third agent may resolve a conflict while a fourth drafts a final recommendation. If every agent can write directly, correctness becomes a matter of hope, timing, and log archaeology.
The paper borrows the discipline of process algebra—a field concerned with concurrent systems—to reason about this more precisely. The mathematical machinery stays in the paper. The practical takeaway is straightforward: we should be able to add another well-behaved agent without reopening every safety question from scratch.
That is what “compositional” means here. The guarantees should survive growth.
A banking audit makes the idea concrete
Consider a multi-agent audit system for a cooperative bank.
One agent extracts facts from source documents. Another calculates ratios and tests thresholds. A third resolves conflicts between claims. A fourth turns verified material into audit findings.
None of those agents needs direct permission to write an official finding. Instead, each proposes its contribution. The kernel checks that:
- the extractor has cited the correct document and page;
- the analyst’s calculation is reproducible from accepted claims;
- a conflicting number is surfaced as a conflict rather than silently overwriting the record; and
- a final finding covers the required regulatory areas.
The result is not an agent system that never makes mistakes. It is a system where an ungrounded or unauthorized proposal does not become institutional truth merely because it was generated fluently.
What the paper establishes
The formal work is there to make the architecture more than a design preference. In plain language, it establishes conditions under which a governed system can offer five useful properties:
- Every change to the shared substrate passes through the governance boundary.
- The governance workflow can keep moving instead of agents waiting forever on one another.
- Adding a conforming agent preserves the system’s governance rules.
- The internal implementation of the governance kernel can be improved without changing the observable safety behavior.
- Authority can be scoped and cannot be forged accidentally by another agent.
There is a useful distinction embedded in that list. An agent model can change. Its output quality may improve or deteriorate. The governance boundary is what keeps model substitution from becoming a safety redesign.
Certificates turn rejection into learning
One part of the paper I find especially promising is the certificate produced for each admission or rejection. A rejection is not merely “no.” It can say: the source is missing, the capability is wrong, the proposed change contradicts an accepted fact, or the work is incomplete.
That feedback has immediate operational value: it makes the system inspectable. It also creates a dense training signal. Agents can learn to make better proposals from the specific reasons their proposals did not pass, without relaxing the boundary that keeps the system safe.
The ambition is not to build a bureaucracy around intelligence. It is to build a system in which intelligence can operate at speed while authority remains explicit, bounded, and auditable.
A better default for agentic systems
We are moving from software that executes fixed instructions to systems that continuously generate possible actions. That changes the job of architecture.
The right question is not only: “What can this agent do?”
It is also: “What must be true before this proposal is allowed to change the world?”
Governed Process Algebra is an attempt to make that question part of the system itself—not a policy document, not an after-the-fact dashboard, and not an aspiration buried in a prompt.
Let agents be creative at the edges. Put authority at the center.