My cto-aipa agent restarted 142 times in less than a day. This happened despite a recent commit, 40b1c22, explicitly stating "stale snapshot solved, 4 of 8 pass" on 2026-09-07. The agent, responsible for critical quality control runs on new listings, was effectively stuck in a loop, consuming resources without completing its core task reliably. This isn't a new problem; another agent, algom-stream, shows 55193 restarts over 23 days, indicating a systemic issue with agent stability under certain conditions.
The Illusion of "Solved"
The cto-aipa agent's role is to perform quality control on new listings. A related commit, 456731b, from the same day, mentioned "pii_qc_llm is grading a stale archive, with proof." This suggests the stale snapshot issue is tied to the data the agent processes, not just its internal state. If the agent is consistently pulling old data, its output will be incorrect, and its internal logic might trigger restarts when it detects inconsistencies or fails to process the outdated information.
The Shared Memory Problem
The NOW.md file in the cto-aipa repository serves as a critical piece of shared working memory for my AI agents. It explicitly states: "Cursor Cloud, Cursor Desktop and Claude Code all work this repo and none of them can see each other's chats. No shared conversation, no Claude MCP in Cursor, no way to send the other agent a message. The only things all of them read are HubSpot and this So this file is not documentation. It is the working memory of whichever agent is not currently running, and the protocol below is how two agents that cannot talk avoid."
This setup, while functional for avoiding direct communication, introduces a vulnerability. If one agent writes to NOW.md based on a stale snapshot of external data, and another agent reads that NOW.md file, it propagates the staleness. The cto-aipa agent, with its 142 restarts, is likely encountering this. It might be reading a NOW.md that reflects an outdated view of the world, leading to failed operations and subsequent restarts. The aideazz repository also had 4 commits in the last 48 hours, including 3aedb30 which refreshed the "ai-ops-wiki" journal, indicating active development and potential changes to how agents interact with shared state.
The Data Source Discrepancy
The problem isn't just internal to the agent or its shared memory. The atlas-ga4-sync.log shows "GA4 sync: 0 atlas_ rows for 2026-09-05," "0 atlas_ rows for 2026-09-06," and "0 atlas_ rows for 2026-09-07." This means for three consecutive days, a critical data source for analytics was not syncing any new data. If cto-aipa or any other agent relies on this data for its operations, it would be operating on a stale view of user activity or business metrics.
Similarly, the atlas-lead-machine.log shows "staged 8 · looked at 32 · no-email 15 · already-in-CRM 13 · outside-band 1 · audit-failed 0 · crawler-blocked rescued 0." While it shows some activity, the atlas-outcomes.log reports "staged": 0, "sent": 0, "pushed to Atlas: {"ok":true,"lanes":7}." This discrepancy suggests that even if leads are being processed, they are not being staged or sent, which could lead to agents like cto-aipa having an incomplete or outdated understanding of the sales pipeline. The concierge-selftest.log shows "ok a Telegram card was produced — draft 430a887f54e35e54" and "duplicate suppressed," indicating some parts of the system are working, but the overall data flow might be compromised.
The Broader System Instability
The cto-aipa agent is not alone in its instability. algom-stream has 55193 restarts over 23 days, and serpapi-jobs has 21 restarts over 20 days. Even dragontrade-main has 3 restarts over 23 days. While algom-poll and n8n show 0 restarts over 42 and 26 days respectively, the pattern of frequent restarts across multiple agents points to a broader issue with how state is managed, how data is synchronized, or how errors are handled.
The wiki-ship.log shows "error: failed to push some refs to 'https://github.com/ElenaRevicheva/aideazz.git'" three times, indicating issues with version control synchronization. This could directly impact agents that pull code or configuration from the repository, potentially leading to them running outdated versions or failing to start correctly. The github-token-watch.log shows a "DELIVERY FAILED: telegram said {"ok":false,"error_code":401,"description":"Unauthorized: invalid token specified"}" on 2026-09-07, followed by an "OK: 302 days left" later that day, and "OK: 301 days left" on 2026-09-08. This suggests a temporary credential issue that was resolved, but such incidents can contribute to a stale state if agents fail during the unauthorized period and don't recover gracefully.
Mitigating Stale Snapshots
To address the AI Agent Stale Snapshot problem, I need to implement more robust state management and data validation.
1. Explicit Data Versioning: Instead of relying on implicit timestamps or "last updated" fields, I need to version critical data snapshots. When cto-aipa processes a listing, it should record the version of the data it's working with. If a restart occurs, it can check if the data version has changed and, if so, re-fetch the latest.
2. Atomic Updates for Shared State: The NOW.md file, while useful, is a simple text file. For critical shared state, I need to consider a more robust mechanism that supports atomic reads and writes, ensuring that agents always get a consistent view. This could involve a lightweight key-value store or a message queue for state changes.
3. Proactive Data Source Monitoring: The GA4 sync: 0 atlas_ rows issue should trigger an immediate alert. Agents relying on this data should have circuit breakers that prevent them from operating on stale data and instead report an upstream data issue.
4. Idempotent Operations: All agent operations should be idempotent. If an agent restarts mid-process, re-running the operation should not cause side effects or data corruption. This is crucial when restarts are frequent.
5. Health Checks with Data Freshness: Beyond just checking if a process is "online," health checks need to incorporate data freshness. An agent is not truly healthy if it's online but operating on data that is days old.
The 142 restarts of cto-aipa are a symptom of a deeper problem: a lack of consistent, versioned state across a distributed system of AI agents. The "stale snapshot solved" commit was a step, but the problem is more pervasive and requires a systemic approach to data integrity and agent resilience.
Frequently Asked Questions
Q: How do you define a "stale snapshot" in your AI agents?
A: A stale snapshot occurs when an AI agent operates on data that is not the most current available from its source, leading to incorrect processing or decision-making. For cto-aipa, this was evident when pii_qc_llm was grading a stale archive, as noted in commit 456731b.
Q: What is the primary cause of these frequent restarts?
A: While specific root causes vary, the pattern of 142 restarts for cto-aipa and 55193 for algom-stream suggests issues with agents encountering unexpected data states, failing to recover gracefully from external service interruptions (like the github-token-watch 401 error), or internal logic errors when processing outdated information.
Q: How do you monitor for stale data sources like the GA4 sync issue?
A: Currently, I rely on log tailing and manual checks, as seen with atlas-ga4-sync.log showing "0 atlas_ rows." I do not have an automated alert system specifically for data freshness across all sources, which is a gap I need to address.
Q: Why use NOW.md as shared memory instead of a database or message queue?
A: NOW.md was adopted as a low-overhead, human-readable mechanism for agents like Cursor and Claude Code, which cannot directly communicate, to share working memory. It allows for quick iteration and visibility into agent state without introducing additional infrastructure complexity.
Q: What is the impact of 142 restarts on your Oracle Cloud costs?
A: I do not have that measured directly. While each restart itself is a small event, 142 restarts in a short period indicate wasted compute cycles and potential for increased I/O if the agent is constantly re-fetching data, which would contribute to overall operational costs on Oracle Cloud.