My cto-aipa process has restarted 81 times today. It's up 0 days. This is not good. My algom-stream process has restarted 55193 times over 7 days. That's worse. Both are LangGraph agents. The algom-poll process, also a LangGraph agent, has 0 restarts over 26 days. This discrepancy highlights the core challenge of shipping stateful agents: checkpointing. I've rewritten the LangGraph state management three times to get to the algom-poll stability. The first two failed silently, costing weeks of lost work.
The Silent Killer: Schema Mismatch in Checkpoints
My initial approach to LangGraph stateful agents production checkpointing was straightforward: define a TypedDict for the state and pass it to the StateGraph. I used oracledb for persistence, storing the entire state object as a JSON string in a BLOB column. This worked for simple flows.
Then I added a new field to the state schema. A small, seemingly innocuous change. The agent continued to run, logging no errors. For weeks, the cto-aipa agent processed jobs, but the new state field was never persisted. Every time the process restarted (which, as the 81 restarts today indicates, happens), the agent would lose the progress related to that new field. I discovered this only when a downstream process, expecting the new state, failed.
The problem was a silent schema mismatch. LangGraph's JsonCheckpointSaver (which I was using under the hood with oracledb as the backend) would serialize the current state, but when loading, it would only deserialize fields that existed in the original schema at the time of the checkpoint's creation. New fields were simply ignored on load, and old fields not present in the current runtime schema were dropped on save. No error, no warning. Just data loss.
This meant the cto-aipa agent was effectively discarding critical parts of its work. The git log for cto-aipa shows 12 commits in the last 48 hours, indicating active development and schema changes. Each change risked this silent failure.
Checkpoint Corruption and the "One-Writer" Rule
My second attempt to fix the state problem involved a more robust oracledb schema and explicit versioning of the state. I added a version field to the state and a migration function. This was an improvement, but it introduced a new class of failures: checkpoint corruption.
The algom-stream agent, which has 55193 restarts over 7 days, was particularly susceptible. This agent processes high-volume, real-time data. Multiple instances of the agent, or even different parts of the same agent, could attempt to update the checkpoint simultaneously. While oracledb provides transaction isolation, the LangGraph CheckpointSaver abstraction doesn't inherently prevent race conditions at the application level if not configured carefully.
I observed oracledb.DatabaseError exceptions related to constraint violations or data integrity issues during checkpoint saves. The root cause was often that one agent instance would read a checkpoint, another would write an updated version, and then the first instance would attempt to write its (now stale) update, leading to a corrupted state or a failed transaction. The algom-stream process would restart, attempt to load a corrupted checkpoint, fail, and restart again. This explains the extremely high restart count.
The fix was to enforce a "one writer" rule per agent instance. Each agent process, like algom-poll (0 restarts over 26 days), must have exclusive access to its checkpoint. This means:
1. Unique thread_id per agent instance: LangGraph uses thread_id to identify checkpoints. Each running agent process must generate a unique thread_id on startup.
2. Atomic updates: The CheckpointSaver must perform atomic read-modify-write operations. For oracledb, this meant ensuring the UPDATE statement included a WHERE clause checking the version or a timestamp to detect concurrent modifications. If another process updated the state, the current process would need to re-read the latest state and re-apply its changes.
This "one writer" rule, combined with careful oracledb transaction management, significantly reduced checkpoint corruption.
The Stable Pattern: Explicit State Transitions and Immutability
The algom-poll agent, running for 26 days with 0 restarts, represents the stable pattern I finally landed on for LangGraph stateful agents production checkpointing. The key insight was to treat the agent's state as an immutable ledger of events, rather than a mutable object.
Instead of directly modifying the state object, each step in the LangGraph pipeline emits a diff or an event. The CheckpointSaver then applies these events to the current state to derive the new state. This pattern is often called event sourcing.
Here's how it works:
1. State is a dict of lists: My state is no longer a flat TypedDict. Instead, it's a dict where each key represents a type of event or data stream, and its value is a list of immutable records. For example: {"messages": [...], "tasks_completed": [...]}.
2. Nodes append, never modify: Each node in the LangGraph graph is designed to append new items to these lists. If a task is completed, it appends a {"task_id": "xyz", "status": "completed"} record to the tasks_completed list. It never goes back and modifies an existing record.
3. Checkpoint is the full event history: The oracledb checkpoint stores the entire history of these events. When an agent loads its state, it reconstructs the current state by replaying all events.
4. Idempotent operations: Because nodes only append, running a node multiple times with the same input will simply append duplicate events (which can be filtered out later if needed) but will not corrupt the state. This makes recovery from restarts much simpler.
This approach solves the schema mismatch problem because new fields are simply new types of events or new properties within an event. The "replay" mechanism naturally incorporates them. It solves the corruption problem because concurrent writes only append to lists; they don't modify existing data, reducing contention.
The algom-poll agent, which polls external APIs and updates its internal state, benefits greatly from this. If it restarts, it loads the full event history, reconstructs its current view of the world, and continues from where it left off, appending new poll results. This is why it has 0 restarts over 26 days.
Cost Implications and Oracle Cloud
Running these agents on Oracle Cloud Infrastructure (OCI) means every restart, every failed transaction, and every wasted computation due to lost state has a direct cost. My n8n process, which orchestrates some of these agents, has 0 restarts over 10 days, indicating its stability. The serpapi-jobs process, with 21 restarts over 4 days, is another area where checkpointing stability is critical.
The oracledb database, running on OCI, is the backbone for checkpointing. The cost of oracledb is tied to CPU and storage. Inefficient checkpointing, leading to frequent re-processing of tasks or large, redundant state writes, directly impacts my OCI bill. The algom-poll agent's stability directly translates to predictable, lower operational costs.
The aideazz repository shows 12 commits in the last 48 hours, including ai-ops-wiki: record the fix that shipped, not the stopgap. This reflects the iterative process of debugging and refining these production systems. The concierge-selftest.log shows ✅ PASS — 4 checks, 4586ms to first card, indicating that the concierge agent, which uses a similar state management pattern, is functioning correctly.
Frequently Asked Questions
Q: How do you handle large state sizes with the event-sourcing approach in oracledb?
A: The oracledb BLOB column can handle large states. For extremely large states, I implement snapshotting: periodically, the full reconstructed state is saved as a new "base" checkpoint, and older events are pruned. This reduces the replay time and storage for event history.
Q: What if an agent needs to modify an existing record, not just append?
A: If a true modification is required, the "event" itself represents the modification. For example, instead of {"task_id": "xyz", "status": "completed"}, you might append {"event_type": "task_updated", "task_id": "xyz", "new_status": "completed"}. The state reconstruction logic then applies these updates in order.
Q: How do you manage schema evolution for the event types themselves?
A: Each event record includes a version field. The state reconstruction logic is designed to handle different event versions, applying transformations as needed during replay. This is similar to database migrations but applied to the event stream.
Q: Does this event-sourcing approach increase latency for state reads/writes?
A: Initial state loading can be slower due to replaying events. However, subsequent appends are fast. For critical paths, I cache the current state in memory and only persist the new events to oracledb, updating the in-memory state after a successful write.