My first LangGraph agent, a simple document summarizer, silently dropped 80% of its jobs for three weeks. The logs showed Agent finished successfully, but the output queue was empty. The problem wasn't the LLM, the prompt, or the vector store. It was a state schema mismatch, a silent killer in LangGraph's checkpointing mechanism, costing me 1,200 CPU hours on Oracle Cloud before I found it.
This wasn't my only LangGraph production headache. Before I shipped my first multi-agent system for AIdeazz, I went through three complete rewrites of the core LangGraph state management and checkpointing logic. Each rewrite addressed a different failure mode: silent data loss, corrupted checkpoints, and finally, the pattern that brought stability to multi-step, stateful agent pipelines.
The Silent State Schema Mismatch
My initial LangGraph agent processed incoming documents, summarized them, and then routed the summary to a specific output channel (Telegram, WhatsApp, email). The state was a simple TypedDict:
class AgentState(TypedDict):
document_id: str
raw_text: str
summary: Optional[str]
output_channel: str
status: Literal["processing", "summarized", "failed"]
The agent worked locally. When deployed to Oracle Cloud, processing 100 documents per hour, it appeared to work. The status field would update to summarized in the database, and the agent logs confirmed completion. But the summary field in the database was always NULL.
I spent days debugging the summarization step itself, convinced the LLM was hallucinating or the prompt was malformed. I added more logging, printed intermediate steps, and even ran the summarization logic outside LangGraph. It always produced a summary.
The issue was in the summary: Optional[str] field. My initial state definition had summary: str. Later, I updated it to Optional[str] to handle cases where summarization might fail or not be needed immediately. LangGraph's SqliteSaver (and by extension, other BaseCheckpointSaver implementations) deserializes the stored state into the current state schema. If a field was present in the stored state but removed or changed type in the new schema, it would be silently dropped during deserialization. Conversely, if a new field was added, it would be None.
My database still held the old schema's state. When the agent loaded a checkpoint, the summary field, which was str in the stored state, was deserialized into the new Optional[str] schema. LangGraph's internal _load_state method, when encountering a type mismatch or a missing field in the new schema, would simply discard the value from the old checkpoint. No error, no warning. The summary was there in the database, but it never made it into the agent's runtime state.
The Fix: Explicit schema versioning and migration. I now embed a schema_version: int in every AgentState and implement a migrate_state(state: AgentState, target_version: int) -> AgentState function. Before loading a checkpoint, I check its version and apply necessary migrations. This adds boilerplate but prevents silent data loss.
class AgentStateV1(TypedDict):
document_id: str
raw_text: str
summary: str # Old schema
schema_version: Literal[1]
class AgentStateV2(TypedDict):
document_id: str
raw_text: str
summary: Optional[str] # New schema
output_channel: str
status: Literal["processing", "summarized", "failed"]
schema_version: Literal[2]
def migrate_state(state: dict, target_version: int) -> dict:
current_version = state.get("schema_version", 1) # Assume V1 if not present
if current_version == target_version:
return state
if current_version == 1 and target_version == 2:
# Example migration: add new fields with defaults
state["output_channel"] = "default"
state["status"] = "processing"
state["schema_version"] = 2
return state
raise ValueError(f"Unsupported migration from V{current_version} to V{target_version}")
# Before loading:
# loaded_state = checkpoint_saver.get_tuple(thread_id).checkpoint["v"]
# current_state = migrate_state(loaded_state, TARGET_SCHEMA_VERSION)
# graph.invoke(current_state, config={"configurable": {"thread_id": thread_id}})
Corrupted Checkpoints and Race Conditions
My second rewrite came after a week of sqlite3.DatabaseError: database disk image is malformed errors. This happened when running multiple instances of the same LangGraph agent, each with its own SqliteSaver, against a shared SQLite file on a network file system.
The problem was a classic race condition. SQLite is robust for single-writer, multiple-reader scenarios. LangGraph's SqliteSaver performs multiple operations: get_tuple, deserialize, modify state, serialize, put_tuple. If two agents tried to update the same thread ID's checkpoint concurrently, one would overwrite the other's changes, or worse, write a partially updated or corrupted blob. The SqliteSaver doesn't implement file-level locking or transaction management for concurrent writes from separate processes.
My agents were deployed as Docker containers on Oracle Container Engine for Kubernetes (OKE). Each pod had its own SqliteSaver instance, and I was mounting a shared NFS volume for the SQLite database. This setup is fundamentally flawed for SqliteSaver with concurrent writers.
The Fix: Centralized, atomic checkpoint storage. I switched from SqliteSaver to a custom BaseCheckpointSaver implementation backed by Oracle Autonomous Database (ADB) and Redis.
1. Redis for ephemeral state and locking: Before an agent starts processing a thread, it acquires a lock for that thread_id in Redis with an expiry. If it can't acquire the lock, it retries or queues the job.
2. ADB for persistent checkpoints: The actual checkpoint data (the serialized LangGraph state) is stored in a JSON column in an ADB table. Updates are performed within a database transaction, ensuring atomicity. The put_tuple method now performs an UPSERT operation, updating the JSON column.
This pattern ensures that only one agent can modify a given thread's state at any time, and the updates are atomic and durable. The cost of ADB is higher than SQLite, but the stability is non-negotiable for production.
# Simplified custom saver logic
class ADBCheckpointSaver(BaseCheckpointSaver):
def __init__(self, db_connection_pool, redis_client):
self.db_pool = db_connection_pool
self.redis = redis_client
def get_tuple(self, thread_id: str) -> Optional[CheckpointTuple]:
# Acquire Redis lock
lock_key = f"langgraph_lock:{thread_id}"
if not self.redis.set(lock_key, "locked", ex=60, nx=True): # 60s expiry, only if not exists
raise LockAcquisitionError(f"Could not acquire lock for thread {thread_id}")
try:
with self.db_pool.acquire() as conn:
with conn.cursor() as cursor:
cursor.execute("SELECT checkpoint_data FROM checkpoints WHERE thread_id = :1", [thread_id])
row = cursor.fetchone()
if row:
checkpoint_data = json.loads(row[0])
# Deserialize into CheckpointTuple
return CheckpointTuple(
config={"configurable": {"thread_id": thread_id}},
checkpoint=checkpoint_data,
parent_config=None, # Or retrieve if stored
)
return None
finally:
self.redis.delete(lock_key) # Release lock
def put_tuple(self, checkpoint_tuple: CheckpointTuple) -> None:
thread_id = checkpoint_tuple.config["configurable"]["thread_id"]
lock_key = f"langgraph_lock:{thread_id}"
if not self.redis.get(lock_key):
raise LockAcquisitionError(f"Lock for thread {thread_id} not held during put_tuple")
with self.db_pool.acquire() as conn:
with conn.cursor() as cursor:
checkpoint_json = json.dumps(checkpoint_tuple.checkpoint)
cursor.execute(
"""
MERGE INTO checkpoints c
USING (SELECT :1 AS thread_id, :2 AS checkpoint_data FROM DUAL) d
ON (c.thread_id = d.thread_id)
WHEN MATCHED THEN UPDATE SET c.checkpoint_data = d.checkpoint_data
WHEN NOT MATCHED THEN INSERT (thread_id, checkpoint_data) VALUES (d.thread_id, d.checkpoint_data)
""",
[thread_id, checkpoint_json]
)
conn.commit()
The "Always Restart" Pattern for Multi-Step Pipelines
Even with schema versioning and atomic checkpointing, my multi-agent systems, especially those involving external API calls or human-in-the-loop steps, were brittle. An agent might make an API call, receive a 200 OK, but then fail to parse the response due to a network glitch or an unexpected payload. The state would be saved, but the agent was stuck. Retrying the same step often led to duplicate actions (e.g., sending the same email twice).
My agents often involve:
1. Ingestion & Pre-processing (Groq for quick classification)
2. Complex Reasoning (Claude 3.5 Sonnet for multi-step planning)
3. External API Calls (CRM, payment gateways)
4. Human Review (via Telegram/WhatsApp)
5. Final Output Generation
A failure at step 3 meant the agent was stuck. LangGraph's default behavior is to resume from the last saved state. If that state was "just before the failed API call," it would retry the API call. If the API call was idempotent, fine. If not, it was a problem.
The Fix: The "Always Restart" pattern. Instead of letting LangGraph resume from the exact point of failure, I designed my agent nodes to be idempotent and to always re-evaluate their current state from the beginning of the current logical step.
Each logical step (e.g., "Summarize Document", "Make API Call", "Await Human Approval") is a LangGraph node. Inside each node, before performing any action, the agent first checks if the action has already been completed based on the current state.
For example, in an "Execute API Call" node:
def execute_api_call_node(state: AgentState) -> AgentState:
if state.api_call_status == "completed":
print("API call already completed, skipping.")
return state
try:
# Perform API call
response = make_external_api_call(state.api_payload)
state["api_response"] = response.json()
state["api_call_status"] = "completed"
return state
except Exception as e:
state["api_call_status"] = "failed"
state["error_message"] = str(e)
return state
This pattern means that if an agent fails mid-node, and then is restarted, it will re-enter the node, see api_call_status is not "completed", attempt the API call, and then update the status. If it fails again, the status remains "failed". If it succeeds, the status becomes "completed". The next time the graph runs, it will see "completed" and skip the API call.
This makes each node effectively idempotent from the perspective of the graph. The graph can be restarted from any point, and it will gracefully pick up where it left off without duplicating work or getting stuck in a retry loop on a non-idempotent action. This also simplifies error handling: instead of complex retry logic within LangGraph, I rely on an external orchestrator (a simple Python script running on a cron job) to periodically re-invoke agents that are in a "failed" or "pending" state.
This "Always Restart" pattern, combined with robust checkpointing and explicit schema management, finally brought the stability needed for production multi-agent systems on Oracle Cloud. My current agents handle thousands of messages daily, routing between Groq for fast initial processing, Claude 3.5 Sonnet for complex reasoning, and custom tools for external interactions, all while maintaining state across potentially long-running processes.
Frequently Asked Questions
Q: How do you handle schema changes for in-flight agents with the versioning approach?
A: When a new schema version is deployed, agents processing older checkpoints will first load the old state, then migrate_state will transform it to the new schema. This transformed state is then saved back to the checkpoint store, effectively upgrading the checkpoint. Agents starting new threads will use the latest schema.
Q: What's the overhead of using Redis for locking and ADB for checkpoints compared to a simpler solution?
A: Redis adds ~2-5ms latency for lock acquisition/release. ADB adds ~10-50ms for checkpoint UPSERT operations, depending on network latency and payload size. This is acceptable for most multi-agent systems where LLM calls dominate latency (hundreds of ms to seconds). The stability gain far outweighs this overhead for production.
Q: How do you manage the LockAcquisitionError in your ADBCheckpointSaver?
A: When LockAcquisitionError is raised, the agent's current invocation is aborted. The external orchestrator (e.g., a message queue consumer or a cron job) responsible for invoking agents will catch this error and typically re-queue the message or mark the thread for a later retry. This ensures that only one agent attempts to process a specific thread at a time.
Q: Does the "Always Restart" pattern mean you re-run LLM calls if a node fails after the LLM call but before saving state?
A: Yes, if an LLM call completes but the subsequent state update or external action fails before the LangGraph node returns and its state is checkpointed, the LLM call might be re-run on restart. To prevent this for expensive LLM calls, I often add a llm_response_cached: bool flag to the state and save the raw LLM response. The node then checks this flag and uses the cached response if available.
Q: Why Oracle Autonomous Database (ADB) specifically?
A: ADB offers fully managed, auto-scaling, and highly available PostgreSQL-compatible or Oracle Database instances. For AIdeazz, it integrates seamlessly with other Oracle Cloud Infrastructure (OCI) services I use (OKE, OCI Functions, OCI AI Services) and provides strong performance guarantees without requiring dedicated DBA resources, which is critical for a lean operation.