AIdeazz Blog About Portfolio

PM2 algom-stream Waiting Restart: Surviving 21426 Crash Loops

· by

One process accumulated 21426 restarts on our production instance while eight sibling processes stayed up for 7 days without a single drop. Running pm2 jlist showed algom-stream sitting in waiting restart with 0 days of uptime, consuming nothing, while its companion worker algom-poll held steady at 79 MB of memory. The failure was not a system crash, an out-of-memory kill, or a hardware freeze; it was an unhandled connection drop inside an event-driven loop that degraded quietly under process supervision.

Here is the exact state reported by PM2 across the fleet:

Eight of nine supervised tasks remained online. If you rely on broad uptime metrics, you miss the failure completely because the machine is green and 8 of 9 tasks look healthy.

The Gap Between Streaming and Polling

The contrast between algom-stream and algom-poll reveals the fundamental danger of streaming architectures. Both services handle data feeds, but their execution models handle network volatility differently.

algom-poll executes on a discrete schedule using node-cron. It wakes up, opens a connection, pulls records, closes the socket, and returns to sleep. If a connection drops midway, the cron job catches the error, records the failure, and terminates normally. PM2 sees a clean process exit code or an idle loop waiting for the next cron trigger, so the supervisor records 0 restarts over 7 days.

algom-stream, by contrast, maintains a persistent long-lived connection. When an upstream WebSocket drops or the remote socket is abruptly severed, an unhandled exception inside the listener crashes the Node.js runtime. PM2 catches the sudden process exit and immediately restarts the script.

Because the upstream provider was either rejecting the connection or timing out, the process crashed on initialization, entered restart backoff, and looped repeatedly until it racked up 21426 restarts. PM2 finally placed the process into a temporary pause: the waiting restart state. In this state, PM2 deliberate delays the next spawn to prevent total CPU lockup.

A stream crashes 21426 times if it lacks an exponential backoff wrapper inside application code. Handing connection retries entirely to a process supervisor like PM2 causes the supervisor to spin up new OS processes until rate limits or backoff delays choke the service.

Shared Memory and Disconnected Agent Coordination

While algom-stream degraded under connection failures, our development workflows continued running across multiple automated coding agents. Managing multiple agents requires strict protocol boundaries when they cannot communicate directly.

In /home/ubuntu/cto-aipa/docs/oracle/NOW.md, our current operator queue explicitly outlines how our agents coordinate without direct messaging:

NOW — the shared session between Cursor and Claude Code
Cursor Cloud, Cursor Desktop and Claude Code all work this repo and none of them can see each other's chats. No shared conversation, no Claude MCP in Cursor, no way to send the other agent a message. The only things all of them read are HubSpot and this file. So this file is not documentation. It is the working memory of whichever agent is not currently running.

In the last 48 hours, cto-aipa recorded 12 commits to maintain this alignment. Recent commits document granular changes across our user-facing and automation touchpoints:

Over the same 48 hours, the aideazz repository logged 4 commits, updating operational documentation and static blog routes:

When disconnected agents operate on the same repository, commits and Markdown state files replace chat threads. Without a shared memory file, agents overwrite each other's work just as rapidly as an unsupervised streaming script crashes against a closed port.

Operational Logs and API Bottlenecks

A single broken streaming service does not bring down our environment, but unhandled rate limits and parsing issues ripple across background workers if left unchecked.

Our recent tail logs show the reality of multi-service maintenance:

In hs-watch-manual-emails.log, we captured an exact HubSpot API threshold hit:

GET /crm/v3/objects/deals/[id]?properties=dealname,dealstage → 429: {"status":"error","message":"You have reached your ten_secondly_rolling limit.","errorType":"RATE_LIMIT","correlationId":"01a1ff7d-cce6-71eb-9a18-f9cbc324fa24","policyName":"TEN_SECONDLY_ROLLING","groupName":"publicapi:private_app-api-calls-ten-secondly:39045903:51409153"}

The endpoint returned an HTTP 429 error because outbound request volume violated the TEN_SECONDLY_ROLLING policy. Rather than crashing or triggering 21426 supervisor restarts, the script logged the error and backed off, allowing the pipeline to recover without process death.

Other system logs verify stability across the remaining services:

HubSpot deals search API confirms that deals at "They replied" (dealstage=contractsent) stand at 136, while deals closed won stand at 0. Managing pipeline state requires the same cold discipline as tracking process uptime: you record the exact number returned by the API, not an optimistic projection.

Building Resilient Architecture Against Loop Failures

When a single process accumulates 21426 restarts, it highlights an architectural truth: supervision is not recovery. Relying on PM2 to keep an unstable socket alive causes resource waste and pollutes diagnostic logs.

To stop a service like algom-stream from entering waiting restart, the application layer must own the retry logic.

First, implement an internal backoff before letting Node.js exit. If a streaming socket drops, the client should sleep using an explicit backoff interval rather than throwing an uncaught exception. When the application traps errors internally, PM2 keeps the process in an online state, avoiding runaway restart counters.

Second, decouple message consumption from streaming connections. If the connection fails, data processing must halt gracefully while alerting the operations dashboard, rather than attempting hundreds of connection attempts per minute.

Third, audit the web footprint to verify service integrity across search engines and AI crawlers. Our free AI visibility audit engine (running 34 checks across 4 groups) evaluates technical stability alongside discoverability:

Our production site scored 100/100, earning an A+ grade by passing 34 of 34 checks. Technical stability requires the same rigor whether you are configuring robots access, managing HubSpot rate limits, or preventing an event stream from spinning into thousands of restarts.

Frequently Asked Questions

Q: Why does PM2 show status "waiting restart" instead of "errored" or "stopped"?
A: PM2 places a process into waiting restart when it exceeds the restart threshold within a short time window. The supervisor applies an automated delay before attempting another spawn to prevent complete CPU starvation.

Q: How does algom-poll avoid the crash loops affecting algom-stream?
A: algom-poll uses discrete polling on a schedule via node-cron, closing its network socket after each run and reporting 0 restarts over 7 days. algom-stream maintains a persistent socket, which crashes the entire runtime when unhandled network drops occur.

Q: What caused the 429 errors recorded in hs-watch-manual-emails.log?
A: The script exceeded HubSpot's TEN_SECONDLY_ROLLING rate limit policy under group publicapi:private_app-api-calls-ten-secondly:39045903:51409153. The application caught the 429 status code and backed off rather than crashing the worker.

Q: How do disconnected AI coding agents collaborate on the cto-aipa repository without direct chat?
A: Agents coordinate via file-based memory inside NOW.md and commit history. Cursor and Claude Code read the shared Markdown state and CRM data, logging 12 commits over 48 hours to execute changes without overwriting active contexts.

— Elena Revicheva · AIdeazz AI Lab · Portfolio · Professional Outlook (PDF)

EspaLuz: live your new country in Spanish. Confidence, belonging and independence for international families in Latin America (web, WhatsApp, Telegram; 14 days free): aideazz.xyz/espaluz