My WhatsApp AI bot, cto-aipa, logged 225 restarts in less than a day. This wasn't a feature; it was a production failure. The core issue wasn't complex AI logic, but fundamental infrastructure gaps: dropped webhooks, ephemeral in-memory state, and external API rate limits that tutorials rarely cover.
Webhook Delivery Gaps: The Silent Killer
The first problem was simple: WhatsApp messages weren't consistently reaching my bot. When a user sends a message, WhatsApp (via Twilio, in my setup) sends a webhook to my server. If that webhook fails to deliver or isn't processed quickly, the message is lost. My cto-aipa process, responsible for handling these incoming messages, was restarting frequently. Each restart meant any in-flight webhook processing was aborted.
I do not have specific webhook delivery failure logs measured, but the cto-aipa process's 225 restarts in under 0 days of uptime (compared to algom-poll at 0 restarts over 88 days, or dragontrade-dashboard at 1 restart over 69 days) strongly indicated instability at the ingress point. A stable process should not restart that often. This instability directly impacted webhook reliability. If the process handling the webhook is restarting, it cannot acknowledge receipt, leading to potential retries or, worse, dropped messages from the user's perspective.
The solution wasn't about optimizing the AI, but hardening the message ingestion. This means implementing a robust queueing mechanism before the bot logic, ensuring that even if cto-aipa restarts, the incoming messages are persisted and can be processed once it's back online. This decouples message receipt from message processing, a critical distinction for any production-grade messaging bot.
In-Memory State Resets on PM2 Restarts
The 225 restarts of cto-aipa also exposed a critical flaw in how I managed conversational state. My initial design relied on in-memory storage for user sessions and context. Each time cto-aipa restarted, this in-memory state was wiped clean.
Imagine a user in the middle of a multi-turn conversation with the bot. They ask a question, the bot processes it, and then cto-aipa restarts. When the bot comes back online, it has no memory of the previous turns. The user's next message is treated as a new conversation, leading to confusion and a broken user experience. This is particularly problematic for AI agents that need to maintain context over several interactions, like my cto-aipa which handles complex deal-related queries.
The fix involved externalizing state. Instead of keeping session data in the cto-aipa process's memory, I needed to persist it to a database or a dedicated state store. This way, even if the process restarts, the state can be reloaded, and the conversation can resume from where it left off. This is a fundamental pattern for resilient applications, but it's often overlooked in initial bot development tutorials that focus on the "happy path" of a single, uninterrupted session.
External API Rate Limits: HubSpot's 429
Beyond internal stability, external dependencies introduced their own set of challenges. My cto-aipa bot interacts with HubSpot to retrieve and update deal information. The hs-watch-manual-emails.log showed a clear problem:
GET /crm/v3/objects/deals/[id]?properties=dealname,dealstage → 429: {"status":"error","message":"You have reached your ten_secondly_rolling limit.","errorType":"RATE_LIMIT","correlationId":"01a16f5e-0b2f-7e80-a3a8-9c3aa49485f4","policyName":"TEN_SECONDLY_ROLLING","groupName":"publicapi:private_app-api-calls-ten-secondly:39045903:51409153"}
This 429 error, indicating "Too Many Requests," meant my bot was hitting HubSpot's "ten_secondly_rolling" rate limit. This isn't a daily or hourly limit; it's a rolling window, making it harder to manage without careful throttling. When this limit was hit, cto-aipa couldn't fetch necessary deal data, leading to incomplete or failed responses to user queries.
The deal-digest.log shows 27 live job deals, and HubSpot totals show 15 qualifiedtobuy, 73 decisionmakerboughtin, 120 contractsent, and 1477 appointmentscheduled deals. My bot needs to access this data, but without respecting the rate limits, it gets blocked.
The solution here was to implement robust rate limiting and retry mechanisms within cto-aipa for all external API calls. This involves:
1. Client-side throttling: Proactively delaying requests to stay within the known limits.
2. Exponential backoff with jitter: When a 429 is received, waiting for an increasing amount of time before retrying, with a random component to avoid thundering herd problems.
3. Circuit breakers: Temporarily stopping requests to a service that is consistently returning errors, to give it time to recover and prevent cascading failures.
These measures ensure that cto-aipa can gracefully handle external API constraints, rather than crashing or failing silently.
The Operator's Role: Shared Memory and Protocol
The operational reality of running these agents is also critical. My NOW.md file explicitly states: "NOW — the shared session between Cursor and Claude Code. Cursor Cloud, Cursor Desktop and Claude Code all work this repo and none of them can see each other's chats. No shared conversation, no Claude MCP in Cursor, no way to send the other agent a message. The only things all of them read are HubSpot and this file. So this file is not documentation. It is the working memory of whichever agent is not currently running, and the protocol below is how two agents that cannot talk avoid."
This highlights a key challenge: when multiple AI agents or development environments are working on the same codebase (cto-aipa had 12 commits in the last 48 hours), coordinating their actions and understanding their current state is paramount. The NOW.md serves as a manual, shared memory, a protocol for agents that cannot directly communicate. This is a human-in-the-loop solution to a technical coordination problem. Without this, conflicting changes or redundant efforts would be inevitable, further contributing to instability.
For example, recent commits in cto-aipa like "CTO AIPA + Atuona free-first voice; claim released" and "Free-first voice: CTO AIPA speaks with Microsoft Edge when OpenAI fails; Atuona transcribes with Groq first" show active development. Without a coordination mechanism like NOW.md, these changes could easily introduce new, unhandled edge cases or conflicts, leading to more restarts.
Frequently Asked Questions
Q: What specific data do you externalize for conversational state?
A: I externalize user session IDs, current conversation context (e.g., last question asked, entities extracted), and any temporary data needed for multi-turn interactions. This data is stored in a persistent database, allowing the bot to retrieve it upon restart and continue the conversation.
Q: How do you implement client-side throttling for external APIs like HubSpot?
A: I use a token bucket or leaky bucket algorithm. Each API call consumes a "token," and tokens are refilled at a rate matching the API's limit (e.g., 10 tokens per 10 seconds for HubSpot's "ten_secondly_rolling" limit). If no tokens are available, the request is queued or delayed.
Q: What's the practical impact of 225 restarts on user experience?
A: For cto-aipa, 225 restarts in less than a day meant that any ongoing conversation was likely interrupted. Users would experience the bot losing context, failing to respond, or providing irrelevant answers, leading to frustration and abandonment.
Q: How do you manage coordination between multiple AI agents or developers working on the same codebase without direct communication?
A: I use a shared, human-readable "working memory" file like NOW.md that outlines current tasks, known issues, and protocols. This serves as a manual communication channel, ensuring that each agent or developer understands the current state and avoids conflicting work.