My fact-checker agent, cto-aipa, experienced a silent failover regression on 2026-10-25. The issue stemmed from Claude Sonnet's rejection of a forced tool_choice parameter, leading to true statements being incorrectly flagged. This wasn't a crash, but a subtle degradation in output quality that required a pivot to plain-English findings to restore reliability. The cto-aipa process, which typically runs with minimal restarts, logged 228 restarts today, indicating the rapid iteration needed to resolve this.
The tool_choice Constraint and Its Unexpected Breach
My agents use tool_choice to ensure specific functions are called, especially for critical tasks like fact-checking. The cto-aipa agent's fact-checker component relies on this to enforce structured output, guiding the model to use its tools for verification. On 2026-10-25, I pushed a commit (f5f4e4aa) to switch the fact-checker to claude-opus-5-5 while promotional credits lasted, aiming for the strongest model until November 8. This was a temporary measure, with the long-term plan (commit 14899f9b) to have model tiers follow monthly budget allocations: Opus, then Claude, then free.
However, the failover logic was designed to gracefully degrade to claude-sonnet-5-5 if Opus was unavailable or failed. This is where the problem emerged. claude-sonnet-5-5 silently rejected the tool_choice parameter when it was forced. Instead of erroring out, it simply ignored the instruction, leading to unstructured, unverified responses. The system continued to operate, but the fact-checking mechanism was effectively bypassed. This resulted in a critical failure: true statements were no longer being correctly identified and were instead flagged as potentially false or unverified.
Diagnosing the Silent Regression
The regression was silent because the API call itself didn't fail. It returned a response, but that response lacked the structured output expected from the tool. This meant no explicit error was thrown that would trigger an immediate alert. I identified the issue by observing the output of the openclaw-fallback.log, which showed repeated entries like "🔎 Fact-check — PrepClaw's reply at 19:04 UTC:" without the expected verification details. The cto-aipa process's 228 restarts in the last 0 days, compared to dragontrade-dashboard's 1 restart in 70 days or algom-poll's 0 restarts in 89 days, was a strong indicator of active development and debugging.
The commit dff369a4 on 2026-10-25 explicitly states: "fact-checker: fix silent failover since 15:30 (sonnet-5-5 rejects forced tool_choice); plain-English findings; true statements no longer flagged". This confirms the root cause and the time of detection. The key was realizing that the model wasn't failing, but rather silently circumventing the intended tool use.
The Pivot to Plain-English Findings
To restore reliability and ensure the fact-checker model could still provide value even without strict tool_choice enforcement, I implemented a pivot to "plain-English findings." This involved modifying the prompt and post-processing logic to extract verification details directly from the model's natural language output, rather than relying solely on structured tool calls.
This change, captured in commit dff369a4, meant that instead of expecting a JSON object from a specific tool, the system would now parse the model's free-form text for indicators of verification or contradiction. While less precise than a structured tool call, it allowed the fact-checker to continue operating effectively with claude-sonnet-5-5 as a failover, preventing the incorrect flagging of true statements. The subsequent commit 75e0a80e, "fact-checker failover regression fixed," confirmed the resolution.
Model Tiering and Cost Management
The incident highlighted the complexities of managing model tiers, especially when balancing performance with cost. My system's model-tier logic (commit 14899f9b) is designed to dynamically switch between Opus, Claude, and free tiers based on budget. The temporary shift to claude-opus-5-5 (f5f4e4aa) was a performance optimization during a promotional credit period.
This experience reinforces the need for robust failover mechanisms that account for subtle API behavior differences across models, even within the same provider's family. A model's rejection of a parameter, even if silent, can have significant downstream effects on agent reliability and output quality. The cto-aipa process, with its 199 MB memory usage, is a critical component, and ensuring its resilience across different model backends is paramount.
Lessons Learned for Fact-Checker Model Failover and Tool Choice Regression
1. Silent Failures are the Most Dangerous: An API call that succeeds but ignores critical parameters is harder to detect than one that throws an error. Monitoring output quality and expected structure is as important as monitoring API success rates.
2. Model-Specific tool_choice Behavior: Not all models, even from the same provider, handle tool_choice identically. Explicitly testing failover paths with each potential model is crucial.
3. Plain-English as a Robust Fallback: When structured output fails, a well-engineered prompt that encourages plain-English findings can provide a valuable, albeit less precise, fallback. This saved the fact-checker from being completely inoperable.
4. Rapid Iteration is Key: The 12 commits in cto-aipa within the last 48 hours, including specific fixes for the tool_choice regression, demonstrate the speed required to address production issues in AI agent systems.
This incident underscores that even with careful planning, unexpected model behaviors can break failover logic. The solution wasn't just to switch models, but to adapt the output interpretation, ensuring the fact-checker remained effective.
Frequently Asked Questions
Q: How did you detect the silent tool_choice rejection without an explicit error?
A: I detected it by observing the openclaw-fallback.log for the fact-checker's output. The log showed the agent was running and producing responses, but the expected structured verification details were missing, indicating the tool_choice was being ignored rather than executed.
Q: What was the impact on the fact-checker's performance during the regression?
A: During the regression, the fact-checker incorrectly flagged true statements. While I do not have a precise percentage of misclassifications measured, the core function of verifying truthfulness was compromised until the plain-English findings pivot was implemented.
Q: Why not just force the model to error if tool_choice is rejected?
A: Some models, like claude-sonnet-5-5 in this case, do not throw an error for an ignored tool_choice. They simply proceed without using the tool. My system's failover logic assumed an error would occur, leading to the silent regression.
Q: How does the "plain-English findings" approach compare to structured tool calls in terms of accuracy?
A: Plain-English findings are generally less precise than structured tool calls because they rely on natural language parsing, which can be ambiguous. However, they provide a functional fallback when structured calls fail, ensuring the system can still deliver a useful, albeit less granular, output.