AIdeazz Blog About Portfolio

The Shared File as Working Memory: AI Agent Inter-Agent Communication

· by

My AI agents cannot talk to each other. This isn't a design choice; it's a fundamental operational constraint. I run eight processes supervised by PM2, including cto-aipa (202 MB, 125 restarts in less than a day) and algom-stream (83 MB, 55193 restarts over 17 days). These agents, and others like dragontrade-main (189 MB, 3 restarts over 17 days), need to coordinate. But there's no shared conversation, no Claude MCP in Cursor, no way to send one agent a direct message from another. The only things all of them read are HubSpot and a specific Markdown file. This file, NOW.md, isn't documentation. It's the working memory for whichever agent isn't currently running, and the protocol within it is how two agents that cannot talk avoid stepping on each other.

The Cost of Disconnected Agents

The immediate consequence of this communication gap is a reliance on external systems or shared files for state management. My NOW.md file explicitly states: "Cursor Cloud, Cursor Desktop and Claude Code all work this repo and none of them can see each other's chats." This means any complex workflow requiring multiple agents to contribute to a single task must use an external artifact as its central nervous system.

For example, the VibeJobHunterAIPA_AIMCF repository saw 6 commits in the last 48 hours, addressing issues like "a fresh thread gets no clear button" and "a null company took down the whole run." These are symptoms of agents operating with incomplete context. If one agent processes a job lead and another is responsible for follow-up, the state of that lead (e.g., "replied," "rejected") must be written to a shared, accessible location. HubSpot serves this purpose for deals, with 102 deals currently in "They replied" stage. But for internal, real-time coordination, a file becomes the lowest common denominator.

Shared Files as a Primitive IPC

Using a shared file like NOW.md for AI agent inter-agent communication is a primitive form of inter-process communication (IPC). It works, but it's brittle. The file acts as a message queue, a shared memory segment, and a semaphore all at once. An agent reads the file, performs an action, updates the file, and then another agent reads the updated file.

This approach introduces several failure modes:

1. Race Conditions: If two agents attempt to write to the file simultaneously, one write will overwrite the other, leading to lost state. I mitigate this by defining a strict protocol in NOW.md itself, but it relies on agent discipline, not system enforcement.
2. Stale State: An agent might read the file, process for a long time, and then write back stale information, overwriting more recent updates from another agent.
3. Parsing Overhead: Each agent must parse the file to understand the current state and then serialize its updates back into the file. This adds computational overhead, especially for complex states.
4. Lack of Atomicity: Operations on the file are not atomic. A crash mid-write can corrupt the file, leaving it in an unreadable or inconsistent state.

The cto-aipa process, with 125 restarts in less than a day, highlights the fragility. While the exact cause of these restarts isn't always tied to file corruption, any agent relying on a shared file for its working memory is vulnerable to such disruptions.

The Wiki as a Post-Mortem Communication Channel

When the shared file mechanism fails, the wiki becomes the primary communication channel for understanding what went wrong. I've logged two significant incidents in the last two days:

These incidents demonstrate that even when agents appear "online" (all 8 PM2 processes are online), their internal state and communication can be deeply flawed. The wiki serves as a critical, albeit asynchronous and human-mediated, channel for documenting and resolving these inter-agent communication breakdowns.

Building Resilience with Explicit Protocols

Given the constraints, building resilience means establishing explicit protocols for AI agent inter-agent communication, even if it's through a shared file. My NOW.md file defines how agents should interact with it. This includes:

For example, the concierge-selftest.log shows "duplicate suppressed (redundant drafters collapse)" and "ok a Telegram card was produced." This implies a mechanism where agents check for existing work before creating new items, preventing redundant actions. This is a form of implicit communication: by checking the shared state (e.g., a database or a file representing existing cards), agents avoid duplicating efforts.

The job-board-watch.log shows a "FAIL: newest posting is 22 days old (need <=21) - board looks abandoned" and "VERDICT: REJECTED." This is an agent communicating its findings and decision to a downstream system (and likely triggering a Telegram alert). While not direct agent-to-agent chat, it's a critical piece of information flow.

The Path Forward: Structured Communication

The current setup, with its reliance on shared files and external systems, is a testament to shipping with what's available. However, as the system grows, the limitations of this primitive AI agent inter-agent communication become more apparent. The 55193 restarts of algom-stream over 17 days, while not directly tied to communication issues, highlight the need for more robust and observable agent interactions.

Moving forward, I need to explore more structured communication patterns. This doesn't necessarily mean a full-blown message queue or event bus for every interaction, but rather:

Observability: Enhancing logging to capture not just what an agent did, but what information it received and sent, and from where. The cita-sort.log showing "0 card(s) repositioned across 3 board(s)" is a good start, but deeper insights into the why* are needed.

The reality of shipping production AI agents with zero VC funding means leveraging existing tools and patterns. The shared file is a pragmatic solution, but its operational challenges demand constant vigilance and a clear understanding of its limitations.

Frequently Asked Questions

Q: Why not use a message queue or a database for inter-agent communication?
A: My current operational setup prioritizes simplicity and minimal overhead. While message queues or databases offer more robust IPC, they introduce additional infrastructure and maintenance complexity. The shared file is a low-cost, immediate solution for coordination between agents that cannot directly communicate.

Q: How do you prevent race conditions with a shared file?
A: I rely on explicit protocols defined within the NOW.md file itself, which agents are programmed to follow. This includes conventions for reading, processing, and writing, effectively creating a manual semaphore. I do not have a measured rate of race condition occurrences.

Q: What happens if the shared file gets corrupted?
A: File corruption is a risk. Agents are designed to handle malformed input gracefully where possible, or to log errors and restart. The cto-aipa process, with 125 restarts, indicates that agents can recover from various issues, though the specific cause of each restart is not always file corruption.

Q: How do you monitor the state of inter-agent communication?
A: Monitoring relies heavily on individual agent logs (e.g., cita-sort.log, concierge-selftest.log) and the wiki for incident post-mortems. There isn't a centralized dashboard specifically for inter-agent communication state; it's inferred from the outcomes and error logs of the individual agents.

— Elena Revicheva · AIdeazz · Portfolio