17031 restarts on a single worker while eight sibling processes sat at zero or near-zero restarts. That was the state of my process table when pm2 jlist reported process algom-stream in waiting restart. While dragontrade-main, dragontrade-dashboard, and algom-poll kept running with 0 restarts across 5d of uptime, an unhandled crash loop burned through thousands of boots without yielding.
When you run automation agents on a single host without venture funding, an unmonitored supervisor configuration will happily turn a network hiccup into a CPU-churning death spiral. Process supervision without backoff logic does not provide resilience; it simply automates the denial of service against yourself.
The Anatomy of an Asymmetric Process Failure
On my Oracle Cloud instance, PM2 supervises 9 processes. Running pm2 jlist showed 8 online of 9. A quick glance at top-level metrics might suggest the system was healthy, but looking at individual process metrics told the real story:
dragontrade-main: online, 0 restarts, up 5d, 196 MBdragontrade-dashboard: online, 0 restarts, up 5d, 80 MBalgom-poll: online, 0 restarts, up 5d, 80 MBpm2-logrotate: online, 0 restarts, up 5d, 85 MBwhitespace: online, 0 restarts, up 5d, 113 MBn8n: online, 0 restarts, up 5d, 409 MBserpapi-jobs: online, 0 restarts, up 5d, 118 MBcto-aipa: online, 3 restarts, up 0d, 203 MBalgom-stream: waiting restart, 17031 restarts, up 0d
Every worker shared the same host OS, the same runtime environment, and the same upstream network interfaces. Yet algom-poll had survived for 5d with 0 restarts, while its counterpart algom-stream collapsed into a cycle that re-executed 17031 times.
The divergence came down to the operational model of streaming versus polling. Polling jobs invoke short-lived, discrete requests on a fixed timer (algom-poll stayed quiet at 80 MB). If an upstream payload breaks or drops, the tick completes, logs an issue, and exits cleanly until the next scheduled interval. In contrast, algom-stream maintained a continuous connection. When an unhandled streaming exception hit the runtime, the node process threw an uncaught error and exited immediately. Because PM2 was instructed to ensure the service remained up, it restarted the process instantly.
Why Fast Restarts Compound Upstream Rate Limits
Instant restarts without an exponential backoff turn transient edge-case errors into self-inflicted outages. When a streaming socket terminates abnormally, the process immediately boots, reads credentials, and attempts to re-establish the stream. If the remote server dropped the socket due to rate-limiting or backpressure, reconnecting inside 100 milliseconds guarantees another failure.
We saw this exact behavior elsewhere in our infrastructure logs. In atlas-outcomes.log, our sync mechanism was abruptly halted by rate-limiting:
hs-outcomes-to-atlas fatal: Error: HubSpot 429: {"status":"error","message":"You have reached your secondly limit.","errorType":"RATE_LIMIT","correlationId":"01a19ce1-be60-723e-95e3-76f3998adeb6","policyName":"SECONDLY","groupName":"publicapi:crm:s
When an upstream endpoint returns HTTP 429, rapid reconnect loops aggravate the penalty. A service restarting in a tight loop burns through CPU cycles and disk writes while hammering upstream gateways. If the supervisor does not enforce exponential backoff delays, the process restarts indefinitely. The 17031 restarts logged on algom-stream reflect precisely what happens when an uncaught stream error meets an immediate restart policy without a delay factor.
Memory Profiles and Worker Isolation
One reason sibling processes survived without cascading crashes was memory isolation. Each process ran under its own PM2 worker execution context. Even as algom-stream cycled repeatedly, n8n remained unaffected at 409 MB, cto-aipa held 203 MB, and pm2-logrotate stayed at 85 MB.
When workers crash and restart instantly, the node runtime does not typically accumulate an in-memory leak across restarts because the memory space is torn down on exit. However, the system pays the penalty in process instantiation, module resolution, and disk writes. If logging to disk occurs during each bootstrap phase, you quickly overwhelm your log aggregators.
In our environment, pm2-logrotate was active and held at 85 MB over its 5d run. Without an active log rotation worker running alongside the supervisor, a worker restarting 17031 times would easily exhaust inode limits or disk space with standard error output. But isolating memory and rotating logs only protects the host; it does not solve the underlying defect causing the stream worker to fail.
Remediation: Backoff Logic and Resilient Streams
Supervising streaming jobs requires different controls than running HTTP API servers or static web dashboards. An API server like dragontrade-dashboard listens for incoming HTTP requests; it fails gracefully if a single route handler throws. A streaming process actively consumes an open connection; an unhandled socket drop will bring down the entire runtime if not explicitly trapped.
To prevent a PM2 restart loop crash backoff failure, three configurations are mandatory:
1. PM2 Exponential Backoff Restart Delay:
Configure exp_backoff_restart_delay in the ecosystem file. Rather than attempting an immediate reboot, PM2 increments the restart delay exponentially (e.g., doubling the wait time up to a specified maximum). This stops the process from cycling thousands of times within hours when an external API goes down.
2. Top-Level Stream Event Handlers:
Streaming connections must handle error, end, and close events explicitly. If the upstream provider closes the socket, the worker should enter an idle or waiting state instead of letting the error bubble up to an unhandled exception that kills the Node.js process.
3. Status Decoupling:
Separate connection health from process health. A process should only exit if its internal state is irrecoverably corrupt. If a network socket drops, the process should stay alive, log the failure, back off using an internal jittered retry timer, and re-establish the connection without forcing the process supervisor to intervene.
Frequently Asked Questions
Q: Why did algom-stream accumulate 17031 restarts while algom-poll had 0 restarts?
A: algom-poll executes scheduled checks and exits cleanly, whereas algom-stream maintains a persistent connection. An unhandled stream drop caused the stream script to crash completely, and PM2 immediately restarted it without an exponential delay.
Q: Did the high restart count on algom-stream affect the memory usage of other services?
A: No. Sibling processes maintained stable memory footprints over 5d of uptime: dragontrade-main held at 196 MB, dragontrade-dashboard at 80 MB, and n8n at 409 MB. Each Node.js process runs in its own memory space, so crashes remained isolated.
Q: What log rotation configuration prevented disk exhaustion during 17031 restarts?
A: The pm2-logrotate module was online and running (holding at 85 MB with 0 restarts over 5d), which ensured that standard error and standard out streams were clipped and rotated rather than filling the disk.
Q: How do you stop PM2 from restarting an agent in a tight loop during an upstream outage?
A: Set exp_backoff_restart_delay in your PM2 ecosystem configuration. This forces PM2 to back off exponentially between reboots instead of instantly spawning new runtime processes when external services return errors like HTTP 429.
— Elena Revicheva · AIdeazz AI Lab · Portfolio · Professional Outlook (PDF)