One process accumulated 21426 restarts on our production instance while eight sibling processes stayed up for 7 days without a single drop. Running pm2 jlist showed algom-stream sitting in waiting restart with 0 days of uptime, consuming nothing, while its companion worker algom-poll held steady at 79 MB of memory. The failure was not a system crash, an out-of-memory kill, or a hardware freeze; it was an unhandled connection drop inside an event-driven loop that degraded quietly under process supervision.
Here is the exact state reported by PM2 across the fleet:
algom-stream: statuswaiting restart, 21426 restarts, uptime 0dalgom-poll: statusonline, 0 restarts, uptime 7d, 79 MBdragontrade-main: statusonline, 0 restarts, uptime 7d, 183 MBdragontrade-dashboard: statusonline, 0 restarts, uptime 7d, 77 MBn8n: statusonline, 0 restarts, uptime 7d, 401 MBwhitespace: statusonline, 0 restarts, uptime 7d, 116 MBserpapi-jobs: statusonline, 0 restarts, uptime 7d, 107 MBpm2-logrotate: statusonline, 0 restarts, uptime 7d, 88 MBcto-aipa: statusonline, 2 restarts, uptime 5d, 263 MB
Eight of nine supervised tasks remained online. If you rely on broad uptime metrics, you miss the failure completely because the machine is green and 8 of 9 tasks look healthy.
The Gap Between Streaming and Polling
The contrast between algom-stream and algom-poll reveals the fundamental danger of streaming architectures. Both services handle data feeds, but their execution models handle network volatility differently.
algom-poll executes on a discrete schedule using node-cron. It wakes up, opens a connection, pulls records, closes the socket, and returns to sleep. If a connection drops midway, the cron job catches the error, records the failure, and terminates normally. PM2 sees a clean process exit code or an idle loop waiting for the next cron trigger, so the supervisor records 0 restarts over 7 days.
algom-stream, by contrast, maintains a persistent long-lived connection. When an upstream WebSocket drops or the remote socket is abruptly severed, an unhandled exception inside the listener crashes the Node.js runtime. PM2 catches the sudden process exit and immediately restarts the script.
Because the upstream provider was either rejecting the connection or timing out, the process crashed on initialization, entered restart backoff, and looped repeatedly until it racked up 21426 restarts. PM2 finally placed the process into a temporary pause: the waiting restart state. In this state, PM2 deliberate delays the next spawn to prevent total CPU lockup.
A stream crashes 21426 times if it lacks an exponential backoff wrapper inside application code. Handing connection retries entirely to a process supervisor like PM2 causes the supervisor to spin up new OS processes until rate limits or backoff delays choke the service.
Shared Memory and Disconnected Agent Coordination
While algom-stream degraded under connection failures, our development workflows continued running across multiple automated coding agents. Managing multiple agents requires strict protocol boundaries when they cannot communicate directly.
In /home/ubuntu/cto-aipa/docs/oracle/NOW.md, our current operator queue explicitly outlines how our agents coordinate without direct messaging:
NOW — the shared session between Cursor and Claude Code
Cursor Cloud, Cursor Desktop and Claude Code all work this repo and none of them can see each other's chats. No shared conversation, no Claude MCP in Cursor, no way to send the other agent a message. The only things all of them read are HubSpot and this file. So this file is not documentation. It is the working memory of whichever agent is not currently running.
In the last 48 hours, cto-aipa recorded 12 commits to maintain this alignment. Recent commits document granular changes across our user-facing and automation touchpoints:
ce635d7e: NOW: film 02 v15 (Android app (beta) heading)c82aebb5: Film 02 v14: WhatsApp door = @espaluz + bot number; why video pixels are not clickable1776d497: Film 02 go-live + click-by-click tracking plan (gaps G1-G8, steps 0-8, builds on the 20 Nov catch)e96b8ce2: NOW: film 02 final = v13 (Spanish and English line)0d326399: NOW: film 02 final = v12 (21 Spanish-speaking countries line)31c15a6e: NOW: EspaLuz support = WhatsApp only; HubSpot chatflow waits on Elena's sign-in
Over the same 48 hours, the aideazz repository logged 4 commits, updating operational documentation and static blog routes:
ae0d273: ai-ops-wiki: refresh journal + AEO surfaces (2026-11-20)ff6f632: chore(blog-static): regenerate routing cost articlef68d2c2: ai-ops-wiki: refresh journal + AEO surfaces (2026-11-19)40e3a3e: chore(blog-static): regenerate small-business pipeline article
When disconnected agents operate on the same repository, commits and Markdown state files replace chat threads. Without a shared memory file, agents overwrite each other's work just as rapidly as an unsupervised streaming script crashes against a closed port.
Operational Logs and API Bottlenecks
A single broken streaming service does not bring down our environment, but unhandled rate limits and parsing issues ripple across background workers if left unchecked.
Our recent tail logs show the reality of multi-service maintenance:
In hs-watch-manual-emails.log, we captured an exact HubSpot API threshold hit:
GET /crm/v3/objects/deals/[id]?properties=dealname,dealstage → 429: {"status":"error","message":"You have reached your ten_secondly_rolling limit.","errorType":"RATE_LIMIT","correlationId":"01a1ff7d-cce6-71eb-9a18-f9cbc324fa24","policyName":"TEN_SECONDLY_ROLLING","groupName":"publicapi:private_app-api-calls-ten-secondly:39045903:51409153"}
The endpoint returned an HTTP 429 error because outbound request volume violated the
TEN_SECONDLY_ROLLING policy. Rather than crashing or triggering 21426 supervisor restarts, the script logged the error and backed off, allowing the pipeline to recover without process death.
Other system logs verify stability across the remaining services:
deal-digest.log: recorded 27 live job deals structured into/home/ubuntu/.openclaw/workspace/deals/TOOLS.md(11037 chars). The deals breakdown showedqualifiedtobuy=17,presentationscheduled=0,decisionmakerboughtin=0,contractsent=6, andappointmentscheduled=4. HubSpot historical totals trackedqualifiedtobuy=17,presentationscheduled=0,decisionmakerboughtin=80,contractsent=130, andappointmentscheduled=1533, with 6 deals in a 14-day window.espaluz-truth.log: written: 66 rows totruth_peopleacross successive runs.concierge-selftest.log: logged a pass across 5 checks in 4555ms to first card, confirming Telegram card delivery and suppressing redundant drafters.followup-radar.log: tracked mail volumes across providers, showingimap.gmail.comwith 653 inbox and 3 sent in the last 45 days,imap.zoho.comwith 162 inbox and 11 sent, and another sweep logging 661 inbox and 3 sent.github-token-watch.log: reported 227 days left until expiration on 2026-11-21, counting down from 229 days on 2026-11-19 and 228 days on 2026-11-20.reply-radar.log: scanned 180 applications with 0 errors, 0 CRM matches, and 0 skipped.apply-queue.log: reported 13 items sunk by kit verdict and 1 new item sent to Telegram.
HubSpot deals search API confirms that deals at "They replied" (dealstage=contractsent) stand at 136, while deals closed won stand at 0. Managing pipeline state requires the same cold discipline as tracking process uptime: you record the exact number returned by the API, not an optimistic projection.
Building Resilient Architecture Against Loop Failures
When a single process accumulates 21426 restarts, it highlights an architectural truth: supervision is not recovery. Relying on PM2 to keep an unstable socket alive causes resource waste and pollutes diagnostic logs.
To stop a service like algom-stream from entering waiting restart, the application layer must own the retry logic.
First, implement an internal backoff before letting Node.js exit. If a streaming socket drops, the client should sleep using an explicit backoff interval rather than throwing an uncaught exception. When the application traps errors internally, PM2 keeps the process in an online state, avoiding runaway restart counters.
Second, decouple message consumption from streaming connections. If the connection fails, data processing must halt gracefully while alerting the operations dashboard, rather than attempting hundreds of connection attempts per minute.
Third, audit the web footprint to verify service integrity across search engines and AI crawlers. Our free AI visibility audit engine (running 34 checks across 4 groups) evaluates technical stability alongside discoverability:
- AI Crawler Access (25% weight): verifies HTTP 200 responses,
robots.txt, XML sitemaps, and crawl access for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot. - Structured Data / GEO (25% weight): audits schema markup, Open Graph tags, canonical URLs, and meta descriptions within the 50–170 character range.
- Answer-Readiness / AEO (30% weight): tracks H1 tags, H2/H3 section headers, content depth past 300 words, and freshness signals.
- Technical Foundation (20% weight): enforces HTTPS, response times under 1.5s, HTML under 1.5 MB, and image alt text coverage of at least 80%.
Our production site scored 100/100, earning an A+ grade by passing 34 of 34 checks. Technical stability requires the same rigor whether you are configuring robots access, managing HubSpot rate limits, or preventing an event stream from spinning into thousands of restarts.
Frequently Asked Questions
Q: Why does PM2 show status "waiting restart" instead of "errored" or "stopped"?
A: PM2 places a process into waiting restart when it exceeds the restart threshold within a short time window. The supervisor applies an automated delay before attempting another spawn to prevent complete CPU starvation.
Q: How does algom-poll avoid the crash loops affecting algom-stream?
A: algom-poll uses discrete polling on a schedule via node-cron, closing its network socket after each run and reporting 0 restarts over 7 days. algom-stream maintains a persistent socket, which crashes the entire runtime when unhandled network drops occur.
Q: What caused the 429 errors recorded in hs-watch-manual-emails.log?
A: The script exceeded HubSpot's TEN_SECONDLY_ROLLING rate limit policy under group publicapi:private_app-api-calls-ten-secondly:39045903:51409153. The application caught the 429 status code and backed off rather than crashing the worker.
Q: How do disconnected AI coding agents collaborate on the cto-aipa repository without direct chat?
A: Agents coordinate via file-based memory inside NOW.md and commit history. Cursor and Claude Code read the shared Markdown state and CRM data, logging 12 commits over 48 hours to execute changes without overwriting active contexts.
— Elena Revicheva · AIdeazz AI Lab · Portfolio · Professional Outlook (PDF)
EspaLuz: live your new country in Spanish. Confidence, belonging and independence for international families in Latin America (web, WhatsApp, Telegram; 14 days free): aideazz.xyz/espaluz