AI Ops RunbookAIdeazz AI Lab ENES Run the pipeline
Blue-hour panorama of low hills and a still bay, the sky fading from indigo to amber — generated on the AIdeazz ATUONA pipeline
~181K lines of original code 21 mo one operator · one server 190+ essays · daily 14:30 29 AI Ops Wiki incidents · 23 failure modes 11 podcast episodes 9 films · automated edit 5 LLM providers · one chain 215 sitemap URLs · 7 machine-readable surfaces
Live in production · October 2026

AIdeazz AI Lab
Operations & Marketing Engine

This is not a demo. Eleven AI co-founder agents plus a marketing engine run 24/7: your website finds you, content publishes daily, leads land in CRM, and Atlas measures whether campaigns actually moved the needle.

Below: plain-language status for business owners, then technical depth for engineers who need proof. Every outage, worked through one beat at a time — and the failure modes named.

Get recommended by ChatGPT Close on WhatsApp AI search visibility WhatsApp agents AI automation
Author
Elena Revicheva
Updated
October 30, 2026
Status
Live in production
~181K
lines of original code · read from production
190+
essays · grounded daily blog · 14:30 every day
29 · 23
AI Ops Wiki incidents · named failure modes
5
LLM providers · one fallback chain on every agent

For engineers: authoritative product details — canonical agent list, GitHub repos, Oracle VM deploy paths, systemd/PM2 names, health checks, local Windows clones, and resilience postmortems — are maintained in ORACLE_ALL_PRODUCTS_RESILIENCE.md (engineering appendix). This public page stays founder-readable: no IPs, ports, or secrets.

Note · What a business owner should take awayElena Revicheva · AIdeazz AI Lab
Start here

What AIdeazz AI Lab is — right now.

Your production AI co-founder fleet plus a marketing engine that connects discovery → content → leads → measurement. Not a prototype.

Start hereThe pieces of the lab12 pieces
Public facePortfolio, blog, inquiry form

Portfolio, blog, inquiry form

Where to see it: aideazz.xyz/portfolio

AI Visibility AuditFree live API + widget — score any site 0–100 on GEO · AEO · tech SEO (ChatGPT / Perplexity / Claude / Gemini readiness)

Free live API + widget — score any site 0–100 on GEO · AEO · tech SEO (ChatGPT / Perplexity / Claude / Gemini readiness)

Where to see it: aideazz.xyz/api

Tech hubCRM, blog, outreach, performance tracking

CRM, blog, outreach, performance tracking

Where to see it: CTO AIPA Telegram

Marketing radarWeekly ad-market intelligence on the lab’s seven sellable services — open windows arrive as ready-to-run sales briefs, and CRM outcomes feed back so the radar learns what converts

Weekly ad-market intelligence on the lab’s seven sellable services — open windows arrive as ready-to-run sales briefs, and CRM outcomes feed back so the radar learns what converts

Where to see it: Atlas live radar

CRM spineAll agents feed one HubSpot pipeline

All agents feed one HubSpot pipeline

Where to see it: HubSpot Sales Pipeline

Measure layerWebsite sessions, form leads, deals by campaign

Website sessions, form leads, deals by campaign

Where to see it: Atlas concept cards (performance block)

Learning recordAI Ops Wiki — 29 production incidents and 23 named failure modes, each incident with a named failure mode and a verified number. Generated, never hand-edited

AI Ops Wiki — 29 production incidents and 23 named failure modes, each incident with a named failure mode and a verified number. Generated, never hand-edited

Where to see it: ai-ops-wiki.html

Provenance dossierNine systems and why they exist — engineering provenance for every product, figures read from production

Nine systems and why they exist — engineering provenance for every product, figures read from production

Where to see it: nine-systems

Publishing engine190+ essays (grounded daily blog), 11-episode podcast, Dev.to canonical crossposts

190+ essays (grounded daily blog), 11-episode podcast, Dev.to canonical crossposts

Where to see it: blog · podcast

AI Film StudioAtuona — source clips in, finished film out: automated edit, sidechain-ducked mix, −16 LUFS master. 9 films live

Atuona — source clips in, finished film out: automated edit, sidechain-ducked mix, −16 LUFS master. 9 films live

Where to see it: atuona.xyz/aifilmstudio

Investor / narrative deckThe live pitch — September 2026 actualization, honest traction

The live pitch — September 2026 actualization, honest traction

Where to see it: pitch.html

Sibling productsEspaLuz tutoring, job hunter, social bots, creative AI, daily briefing

EspaLuz tutoring, job hunter, social bots, creative AI, daily briefing

Where to see it: Portfolio

AI Visibility layer · September 2026
Every product in the lab is built to be read and quoted by AI answer engines, not only ranked by Google. The AI Visibility Audit API is both the product I sell and the instrument I run on my own surfaces: 34 automated checks in four weighted categories — AI crawler access 25%, structured data (GEO) 25%, answer-readiness (AEO) 30%, technical foundation 20% — plus a separate verdict for each of six AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot). Free to try, public demo key.
AI visibility · re-audited 26 Oct 2026 with the API itselfScores of our own surfaces6 surfaces
aideazz.xyzportfolio + blog100 · A+

aideazz.xyz — portfolio + blog

AI crawlers allowed: 6 / 6

podcast.aideazz.xyz11 episodes, each linked to its post100 · A+

podcast.aideazz.xyz — 11 episodes, each linked to its post

AI crawlers allowed: 6 / 6

This runbook100 · A+

This runbook

AI crawlers allowed: 6 / 6

/apithe audit tool itself98 · A+

/api — the audit tool itself

AI crawlers allowed: 6 / 6

AI Ops Wiki29 incidents · 23 failure modes93 · A+

AI Ops Wiki — 29 incidents · 23 failure modes

AI crawlers allowed: 6 / 6

atuona.xyzAI film studio & gallery93 · A+

atuona.xyz — AI film studio & gallery

AI crawlers allowed: 6 / 6 — was 69 · C and 0 / 6 on 19 Sep: the audit's own finding, since fixed

How it runsThe marketing engine2 views
GEO / AEO / Tech SEO across the productsFrom a build session to an AI answer engine citing it

Build sessions → AI Ops Wiki incidents (published nightly, alerts on failure) → grounded daily blog (190+ posts, EN/ES) + Dev.to canonical → podcast episodes that link back to their posts → llms.txt, a dated geo-manifest and a 215-URL sitemap that name the API, blog, wiki and podcast for AI crawlers → distribution on LinkedIn, Instagram and YouTube (Buffer / Make) and X (Algom Alpha, X API) → a weekly citation probe asks Google AI Overviews, Gemini, OpenAI search and Perplexity whether any of it is actually cited.

The closed marketing loopLive since July 2026 · 11 steps, CRM outcomes flow back
SEO & discoverabilityDaily blogInquiry form with trackingHubSpotLead triageOutreachAtlas detects a market window for a service we sellSales brief lands in TelegramOutreach goes outCRM outcome (sent / replied / won) flows back into AtlasNext week’s briefs are smarter
Status · 27 Oct 2026What is loaded, partial and next3 lines
LoadedDiscovery, grounded daily blog (one article every day, 20–27 Oct), one CRM endpoint for every agent, triage, Atlas Monday radar + lead machine, Lead Concierge (daily self-test: PASS), read-only mail watch on two inboxes, WhatsApp click tracking, CRM outcomes back into Atlas

Discovery, grounded daily blog (one article every day, 20–27 Oct), one CRM endpoint for every agent, triage, Atlas Monday radar + lead machine, Lead Concierge (daily self-test: PASS), read-only mail watch on two inboxes, WhatsApp click tracking, CRM outcomes back into Atlas

PartialConversion — measured in HubSpot on 27 Oct: 119 of 333 judged-fit job deals reached “they replied”; 68 Atlas leads contacted; website inquiries 9 sent → 1 reply. Turning replies into paid work is the current game.

Conversion — measured in HubSpot on 27 Oct: 119 of 333 judged-fit job deals reached “they replied”; 68 Atlas leads contacted; website inquiries 9 sent → 1 reply. Turning replies into paid work is the current game.

NextX API credits for Algom; showcase package for clients; automated weekly blog from each Atlas snapshot

X API credits for Algom; showcase package for clients; automated weekly blog from each Atlas snapshot

Marketing roadmapSix phases and where each stands6 phases
01Foundation — SEO, analytics, trust signalsLoaded
02Content — daily Dev.to blogLoaded
03Attribution + Atlas measureCRM outcomes loop live (Jul 21)
04Outbound (WhatsApp-first prospect play + Atlas briefs)Live · measured per service
05Lead triageOperational
06Client showcase packagePriority
No terminal requiredHow to verify it yourself5 checks
01Open Atlas — the snapshot refreshes every Monday ~9:15 AM Panama and should show the most recent Monday’s date.
02Open the angle API — live market angle for a service lane, with real CRM outcomes, as JSON. This is the same intelligence the sales briefs use.
03Visit aideazz.xyz/blog — new articles publish on a daily cadence, including posts written from Atlas’s own weekly snapshot.
04Submit the inquiry form on aideazz.xyz/portfolio — you receive a confirmation email when it works (or use the form right on this page, below the go-to-market pipeline).
05Engineers: full runbook in Marketing Engine roadmap on GitHub.
Aerial view of long lavender rows converging toward a golden horizon at dusk, one calm system running in order
One operator · one server

How many AI capabilities does one operator run?

11
Tracked capabilities
10 live · 1 roadmap (AILA). All auto-restart; I am the architect and the on-call.
Lines of original code

How much original code runs in production?

181Klines

Nine systems and why each exists, with figures counted from the running code.

Read from production · provenance dossierProvenance
Essays published

How much has the lab published?

190+essays

Every figure traced to a measurement before it ships; fails closed otherwise.

Grounded daily blog · /blogPublishing
Named failure modes

How many production failures are documented?

23failure modes

29 production incidents, each with a named failure mode and a verified number.

Postmortems · AI Ops WikiLearning record
Manual steps

How many manual steps are left between a commit and a post?

0manual steps

A commit worth talking about reaches LinkedIn, the blog, X and Instagram without anyone retyping it.

Milestone → social pipelineAutomation
Section 01

Products & agents. Each row is a shipping capability.

What customers or partners touch, with how it runs underneath (Linux services, process supervisor, or serverless). Naming matches the internal resilience matrix without exposing infrastructure coordinates.

The fleetEvery agent, one line each12 entries · 10 live
01EWEspaLuz WhatsAppSpanish tutoring on WhatsApp — conversation, drills, corrections.systemdLive
Channel: Spanish tutoring on WhatsApp — conversation, drills, corrections.Real PayPal subscription payments (two of $7.77 in the 30 days to 27 Oct 2026); every trial lands in HubSpot through CTO AIPA. Service espaluz-whatsapp, memory in PostgreSQL + pgvector.
02ETEspaLuz TelegramThe same tutoring on Telegram, with a kids mode.systemdLive
Channel: Same tutoring product on Telegram, with a kids mode where the heroes Pere and Titi answer in their own voices on video.Trials land in HubSpot through CTO AIPA. Two-layer memory: retrieval + pgvector RAG (espaluz_rag.py). Service espaluz-familybot.
03EIEspaLuz InfluencerOne brand post a day at 23:00 Panama.systemdLive
Brand: One post a day at 23:00 Panama; two days in three it promotes the AI Visibility audit, and it picks up real shipping milestones handed over by the CMO agent.Groq text → Make.com → social (last verified post 26 Oct, Make.com 200); each post is logged in HubSpot. Unit espaluz-influencer.
04AAAlgom Alpha (@reviceva)Always-on X presence — paused until X API credits return.PM2Paused · X credits
Growth: Always-on X presence (education + narrative); folds major releases into the timeline without sounding like raw developer logs.Stream sampling, engagement runner, and account-activity hooks coordinated with the CTO bot for alerts / follow-back. PM2 workers include dragontrade-main and satellite processes. Since 17 Oct 2026 the X API credits are depleted: the processes run, the output is zero — shown because this page reports output, not uptime.
05VJVibeJob HunterHourly job search across 13+ boards, judged and staged in HubSpot.systemdLive
Product: Hourly job search across 13+ boards → rules + an LLM judge with five-provider fallback → a HubSpot deal → a Telegram alert with one-tap ⏳ I applied / ❌ Not a fit. Reads Gmail + Zoho read-only for employer replies. Never applies on its own.333 judged-fit job deals; 119 reached “💬 They replied” (HubSpot, 27 Oct 2026). Her decisions feed back into the judge hourly. Workers vibejobhunter + Bright Data ingest serpapi-jobs.
06CMAI Marketing Co-Founder (CMO AIPA)One LinkedIn post a day, led by real engineering milestones.systemdLive
Revenue narrative: One LinkedIn post a day (Make.com → Buffer), led by a real engineering milestone from CTO AIPA when there is one; hands the same milestones to Algom (X) and the Influencer (Instagram).Last verified post 27 Oct 01:00 UTC (“Post completed successfully”). Claude + connectors for social; Hunter.io enrichment → HubSpot. Paired FastAPI bridge vibejobhunter-web exposes an internal health route.
07OCOpenClaw (interview coach)Mock interviews by voice and text, English and Spanish.systemdLive
Coach: Mock interviews by voice and text, English and Spanish, on Telegram and WhatsApp — coached from a deals sheet rebuilt from HubSpot every 30 minutes and from fact-checked study packs for each company.Standalone gateway service openclaw-gateway; probed via private health URL on the app host.
08CTTech Co-Founder (CTO AIPA)Control tower + CRM hub every other agent writes through.PM2Live
Control tower + CRM hub: every other agent writes to HubSpot through one endpoint here, one tag per agent, so provenance is a filter. Reviews code pushes, broadcasts milestones to marketing, and runs the Lead Concierge, the AI Visibility API, the grounded daily blog (one article every day) and the job apply kit.Express orchestrator under PM2 (cto-aipa), Oracle Autonomous DB via wallet-based TLS — credentials never live in this HTML.
08.1SBSprint Briefing (Sprinter)Daily audio briefing from the month’s board, voice notes and commits.LambdaLive
Founder ritual: Daily audio briefing synthesized from tasks, notes, and captures. Since 18 Sep 2026 it reads only the current month’s Trello board (Just for Today + To Dos), voice notes from the last 48 hours and the last day’s commits — and says so plainly when nothing is new, instead of replaying old items.AWS Lambda on a schedule; pulls context through the CTO service over HTTPS with shared-secret auth — no database wallet inside Lambda.
09ATCreative Co-Founder (Atuona CCF)Creative partner with a fully automated AI film studio.PM2Live
Creative partner: Separate bot persona + public gallery, now with a fully automated AI film studio — source clips in, finished film out: voiceover locked to clip duration, crossfades on a running-offset accumulator, sidechain-ducked music, −16 LUFS master. 9 films live.Single PM2 orchestrator binary; site ships via static edge hosting.
10ASAtlas Shifted (Marketing Strategist)Reads the live ad market every Monday and stages leads.PM2Live
Marketing intelligence: Every Monday 9 AM Panama it reads the live ad market, marks each angle ENTER / WATCH / AVOID, alerts when one of the lab’s service markets opens, and its lead machine stages qualified local businesses into HubSpot — 101 so far, 68 contacted.Detect → stage → measure: real HubSpot deal stages per market are read back so next Monday’s call learns what converted (Mon 26 Oct: 26 businesses looked at, 8 staged + 1 with no website). Public ad-transparency data only. Live at aideazz.xyz/atlas.
11ALAILALong-horizon orchestration — architecture, not yet running.—In design
Roadmap: Long-horizon personal orchestration — documented architecture, not yet a standalone production process.Nothing runs yet. Until it ships, the agents hand work over through HubSpot, files and schedules — see below.
The fleetCross-cutting capabilities7 shared layers every agent stands on
GEO + SEO infrastructure · aideazz.xyz · AI-crawler signals · bilingual blog · lead capture → Oracle
CRM pipeline (HubSpot) + automatic prospecting from SERP, hiring boards & product launches — staged deals carry a tested WhatsApp link + one-click email; sends auto-move the deal, create follow-up tasks, and Resend webhooks mark delivered/opened
X growth automation (stream listening + engagement + alerts)
Morning briefing audio via AWS Lambda + secure CTO data bridge
Executive rhythm: Trello digests to Telegram
Live web data layer: Bright Data Web Unlocker · SERP · Scraping Browser + autonomous Claude research agent (/research_company · /research_employer · /research_competitor in Telegram)
Five-provider LLM failover on every agent (Claude · OpenAI · Gemini · Grok · Groq), each with its own eval harness. Order is set per use case, not by one fleet template: quality surfaces lead with Claude, classifiers lead with a plain fast model, bulk generation leads with the cheapest that clears the floor. Each product's chain is proven by breaking providers cumulatively — with four of five dark, the agent still answers. Provider retirements are a one-line config change, verified against the eval before it ships.
Operational invariant
Exactly one deployed checkout per GitHub repository — prevents version drift, duplicate secrets, and “which folder is live?” incidents. Pairs like CTO + creative co-founder deliberately share a codebase but run as distinct personalities/interfaces.
The fleetHow the agents hand work to each otherthrough HubSpot, files and schedules
How the agents hand work to each otherThrough HubSpot, files and schedules — not live chat
  • One CRM spine: VibeJob Hunter, Atlas, WHITESPACE, EspaLuz and Algom all write to HubSpot through CTO AIPA’s single endpoint, each with its own tag — so every deal says which agent made it.
  • Inquiry → reply: a website form or chat becomes a HubSpot deal, AI drafts the answer, Elena taps ✅ in Telegram, Resend sends it from the company domain and stamps delivery on the deal.
  • Job post → interview: VibeJob Hunter stages the deal → her ⏳/❌ tap moves it → “🔥 I act today” triggers the apply kit’s letter, CV and defenses → OpenClaw coaches her for that exact company from the deals sheet.
  • She teaches the judge: her stage moves and reasons are read back every hour into the job judge’s feedback.
  • Inbox → deal: a reply in Gmail or Zoho becomes a note, a task and a Telegram alert on its deal — read-only, so her mail stays unread until she opens it.
  • Atlas learns from sales: Monday’s market windows and leads go into HubSpot; their real stages come back into Atlas the same morning.
  • Code → content: a feat: push to GitHub becomes a milestone that CTO AIPA hands to the CMO for LinkedIn, X and Instagram.
The fleetWhat each agent produced in HubSpot3,220 deals, counted 27 Oct 2026
VibeJob Hunter · judged fit119 they replied 43 sent · 5 act today333
VibeJob Hunter · board ingest20 sent · 3 act today · 296 filtered out1,659
Atlas lead machine68 contacted 31 act today101
Website inquiries (Lead Concierge)9 sent · 1 replied50
Prospect search (CTO AIPA)25 act today / this week · 1 replied510
EspaLuz9 sent · 1 replied · PayPal subscriptions live96
Algom Alpha (X)paused none past triage92
All agentsEvery deal named by the agent that made it3,220
Section 02

Reliability & uptime practice. A failure must reach a human, not a log.

For founders: scheduled probes ask each product whether it still responds. For engineers: one bash driver on the primary Oracle VM runs roughly every five minutes; keep-alive traffic avoids idle reclamation; systemd caps restart storms.

Dew-covered spider web lit gold at dawn — a natural network where every thread is connected and watched

No noisy herd restarts

Watchdog · ~5 min detection

Failed probes recycle only the affected unit. Concrete URLs and scripts remain in the private appendix — not pasted here.

Graceful degradation

Model retirements are config, not rewrites

Providers retire hosted models every few months (Groq drops Llama 3.3 for its dev tier in Aug 2026). Every agent resolves its model id through one fleet-wide switch, so a cutover is a one-line .env change — tested against a golden-set eval before flipping, and instantly reversible. The switch shipped fleet-wide in July 2026, weeks ahead of the deadline. For founders: the AI vendor treadmill is priced into the architecture, not paid for in rebuild weeks.

Silent failure · Sep 2026

Two silent failures surfaced the same week

The wiki publisher had been jammed for 15 days by an interrupted rebase that one catch {} swallowed every night, so the AEO manifest froze while every run looked finished. The morning briefing was running a three-month-old Lambda bundle with a revoked token and a retired model id — the fleet-wide model switch had never reached the one agent deployed off the VM. Fixes: the publisher now syncs before it generates and alerts Telegram on any failure; the Lambda was rebuilt from source and joined the switch. Rule since: verify the deployed artifact, not the config.

ReliabilityHealth check per agent7 checks
CTO AIPA + AtuonaOrchestrator HTTP OK via localhost probe

Health check: Orchestrator HTTP OK via localhost probe

Recovery: pm2 restart cto-aipa

EspaLuz WhatsAppTutoring webhook answers OK from localhost

Health check: Tutoring webhook answers OK from localhost

Recovery: systemctl restart espaluz-whatsapp

VibeJob Hunter + CMOMarketing bridge health endpoint OK internally

Health check: Marketing bridge health endpoint OK internally

Recovery: systemctl restart vibejobhunter-web vibejobhunter

OpenClaw ShortlistHTTP GET gateway loopback → 200

Health check: HTTP GET gateway loopback → 200

Recovery: systemctl restart openclaw-gateway

Sprint BriefingCloudWatch + EventBridge schedule

Health check: CloudWatch + EventBridge schedule

Recovery: Lambda retries / DLQ policy

PM2 stacks (e.g. Algom)cron HTTP + pm2 jlist status online

Health check: cron HTTP + pm2 jlist status online

Recovery: pm2 restart <app>

All systemd agentsProcess liveness via systemctl

Health check: Process liveness via systemctl

Recovery: systemd restart policy

Section 03

Go-to-market automation. Ship once, publish everywhere, answer first.

When engineering ships something worth talking about, the stack fans it out across LinkedIn, blogs, X, and Instagram — without a human retyping the same story five times. And when a lead comes back in, the same stack answers first.

Quality gate
Only commits tagged feat:, launch:, or release: notify the marketing agent. Housekeeping commits (fix:, docs:, chore:, …) stay invisible to customers.
Go-to-marketThe pipeline, step by step7 steps
01
GitHub Webhook

Commit detected → CTO AIPA

Push events hit the secured webhook. Groq/Claude review diffs, classify milestones, enqueue pending updates for downstream marketers.

02
LinkedIn · 20:00 Panama

CMO generates + posts

Claude Sonnet copy → Make.com delivery. Zero manual paste.

03
Daily blog · aideazz.xyz/blog + dev.to

Grounded blog publish — fail-closed on claims

The daily publisher collects production evidence before it writes and refuses to publish any number it cannot trace to a measurement (grounding gate, Aug 2026). Near-duplicates are blocked by token overlap, not slug equality. Ships to aideazz.xyz/blog with dev.to canonical crosspost; sliding-window mutex and always-notify Telegram guard against silent double publishes.

04
X · Every 5th post slot

Algom Alpha tweet

x-tech-updater.js merges milestones in plain language (Haiku / Groq), guarded against duplicate queue states.

05
Instagram · Even days 18:00 Panama

EspaLuz Influencer

Milestone-aware caption + Make.com media pipeline; falls back to standard queue when nothing pending. Since Sep 2026 the same channel also carries the AI Visibility API campaign two days in three (EspaLuz on the third), alternating English and Spanish.

06
Inbound · HubSpot → Claude Fable 5 → Telegram

Lead Concierge — form to sent reply in minutes

Portfolio inquiries and HubSpot chat become CRM contacts in seconds (server-side push, no cron wait). Claude Fable 5 drafts the reply — right language, right proof links — and a dedicated concierge watchdog re-drafts through the five-provider chain (Claude → OpenAI → Gemini → Grok → Groq) if the primary dies, so a single vendor outage cannot kill the inbox; on 19 Aug the chain absorbed a dead primary in 2.4 seconds. The draft lands in Telegram with Send / Edit / Skip buttons; one tap emails the lead from the company domain via Resend (optional Cc + magic-byte-checked PDF attachments) and logs the sent text to the contact’s CRM timeline. Full audit trail, human approval on every send.

07
Outbound · leads found → HubSpot on autopilot

Automatic lead search + CRM that works itself

Prospects are discovered automatically — SERP, hiring boards, product launches, Bright Data enrichment — and staged into HubSpot as company + deal + contact + note, each note carrying a tested WhatsApp link and a one-click outreach email. From there the CRM works itself: an outgoing email is auto-detected and moves the deal to Sent while a soft follow-up task (+4 days) is created; Resend webhooks mark delivered / opened on the deal — true delivery events, not assumptions (delivered · bounced · complained · delayed · opened · clicked); a reply moves the deal to Replied and closes the open follow-ups. Every record names its author agent by prefix, so provenance is a filter, not an investigation.

River bending in a glowing oxbow loop under purple mist at golden hour — the closed loop of the marketing pipeline
Try it

Don’t take the diagram’s word for it — this form IS the pipeline.

Submit it and you become a UTM-attributed HubSpot contact within seconds, Claude drafts a reply with full context of this operation, and it reaches my Telegram for one-tap approval. You’ll hear back fast — that’s the whole point of step 6.

Protected by reCAPTCHA Enterprise · Attributed as utm_source=sop-ai-ops so you can ask me later exactly how your own submission moved through the CRM.
Section 04

Release discipline. We ship like a product company.

Board-friendly translation: predictable processes, isolated secrets, verifiable rollouts — even though agents move faster than most teams.

Release disciplineThe six release rules6 rules
01One live checkout per codebaseCheckout

Eliminates “which folder is prod?” debates; paired bots share code intentionally but never duplicate repos.

02Green build, then swapBuild

Pull latest → compile/tests succeed → only then restart supervised processes. Broken artifacts never replace what customers already rely on.

03Secrets isolationSecrets

Each bot owns its environment file; crypto wallets never touch GitHub; TypeScript strict mode catches sloppy typings before prod.

04Crash-proof process registryRegistry

Every agent process is registered to auto-start on server boot and auto-restart on failure — no manual babysitting after a power cycle or kernel update.

05No silent failuresSignals

Crash handlers log before exit so supervisors show why something died; watchdog cadence targets ~5 minute detection.

06Verify after deployVerify

Health signal green, database connectivity logs clean, one real Telegram interaction — all pass before the incident is closed.

Section 05

Incident response template. Symptoms, causes, fix, proof.

For stakeholders: regressions are handled like financial reconciliations — symptoms, compounded causes, fix, proof — so the same automation trap rarely strikes twice.

PostmortemsTwo incidents, written up in full2 incidents
Duplicate side effectHubSpot duplicate posting loop
May 10, 2026 — same milestone tweet emitted twice ~6 minutes apart
Symptom

Pending HubSpot milestones resurfaced every x-tech-updater.js cycle.

Root causes

Triple mismatch: legacy posted vs filter on posted_x; mark endpoint keyed on timestamp while older rows used received_at; backlog needed posted_x backfill.

Fix applied

GET excludes either flag; mark endpoint tries timestamp → received_at → title; JS client sends title for fallback matching.

Verified by

API snapshot {"ok": true, "pending": [], "total": 0, "held": true} + two full automation cycles without duplication.

Resolution

≈2 hours from detection → patched APIs → verified on live automation cycles. Full narrative retained in the engineering appendix linked below.

Vacuous health checkThe engagement loop that never ran
May 25, 2026 — config said “32 engagements/day”; logs said zero cycles had ever completed
Symptom

Asked the logs to prove a claimed engagement rate. Startup banner found 4,357 times; cycle-completed action line found zero times. The behavior never happened, no matter what the config said.

Root causes

Three layers deep: the first engagement run was scheduled 5 minutes after startup; the process was being restarted every 5 minutes by an external cron; and that cron was a health check whose grep never matched PM2’s box-drawing table output — so it judged a healthy process dead, forever.

Fix applied

Health check rewritten to read structured state: pm2 jlist | jq on the process status field instead of grepping rendered text. Process stayed up; the first engagement cycle in the bot’s history fired the same day, with real replies and follows verified from logs.

Rule earned

Verify from logs, not config. Never claim agent behavior without grepping for the ACTION line (not the setup line). The fix is now a standing operating rule across the fleet.

Dark ceramic bowl repaired with gold kintsugi seams on sand at blue hour — failures documented, not hidden

Kintsugi, not cover-ups

SOP update rule: materially production-facing incidents earn the same structured write-up — so institutional memory compounds instead of resetting. The full running record, with each failure mode named and defined in plain language, lives in the AI Ops Wiki.

aideazz.xyz/ai-ops-wiki.html ↗
Section 06

Stack reference. Boring where uptime matters, sharp where it differentiates.

Boring reliability primitives where uptime matters; sharp AI + CRM + social APIs where differentiation matters.

Stack referenceTen layers of the stack10 layers
Process Mgmt3 tools
PM2systemdAWS Lambda + EventBridge
Databases2 tools
Oracle Autonomous DB (enterprise-grade encrypted connection, multi-table estate)PostgreSQL + pgvector (semantic memory for tutoring agents)
CRM & Outreach3 tools
HubSpot CRM v3 + v4 associationsHunter.ioResend
Social3 tools
X API v2 (Account Activity, filtered stream, engagement worker)Make.comTelegram Bot API
AI / LLMs9 tools
Five-provider waterfall on every product — Claude · OpenAI · Gemini · Grok · Groq, order set per use case; with four of five dark, the agent still answersClaude Sonnet / Haiku (quality paths)OpenAI (gpt-4o-mini + text-embedding-3-small; TTS / Whisper now the backup — voice runs Groq-first with a free Microsoft Edge fallback since 24 Oct 2026)Gemini + Grok (xAI) in the chainGroq open-model inference — model id via one fleet-wide switch; provider deprecations are a one-line config changeBright Data (public ad-transparency capture)Atlas creative: Flux 1.1 Pro stills + a Runway → Luma → Gemini video failover chainLangChainLangGraph
Lead Gen4 tools
HN AlgoliaGitHub RESTProduct Hunt GraphQLBright Data (SERP API + Web Unlocker + Scraping Browser — replaced paid SerpAPI) — ~150–250 net-new companies/month after filtering
Hosting / CDN4 tools
OCI Ubuntu VM (VM.Standard.E5.Flex, 12 GB)AWS Lambda4everland IPFS frontendsCloudflare DNS
Monitoring5 tools
Cron health driverPM2 logsCloudWatchcurl probesOCI keep-alive
Content & SEO9 tools
Bilingual daily blogdev.to crosspostGA4 + Atlas campaign measure (Jul 2026)Google Search ConsoleGEO packAI Visibility Audit API (34 checks)llms.txt + dated geo-manifestweekly AI citation probelead capture → Oracle → HubSpot
Project Mgmt2 tools
Trello API (daily + weekly Telegram briefings)GitHub webhooks across the fleet
Section 07

Questions people actually ask. Answered plainly.

The questions that come up most — and marked up so AI answer engines can lift them directly.

What is the AIdeazz AI Lab?

A production fleet of eleven AI co-founder agents plus a marketing engine, running 24/7 on a single Oracle VM at zero monthly infrastructure cost. It publishes content daily, captures leads into a CRM, and measures whether campaigns moved the needle rather than reporting vanity metrics.

How does AIdeazz keep AI agents from dying silently?

Every product declares a health check, a restart command and a process name in one resilience config. A watchdog polls those checks and restarts a dead process, and failures surface in Telegram rather than waiting to be noticed. Silent death was the original failure mode, so a service that stops answering is treated as an incident, not a gap in the graph.

Does the marketing engine generate leads, or only traffic?

Inbound inquiries post from the website into Oracle and HubSpot instantly, then a Make scenario drafts a reply that is approved from Telegram before it sends. More than 3,200 deals and 1,500 contacts exist in the CRM (counted 26 October 2026). Attribution is instrumented end to end with UTM tags, though campaign volume is still low.

How is content optimized for AI search, not just Google?

A 34-check audit engine scores every page for AI crawler access, structured data, answer-readiness and technical foundation. robots.txt explicitly allows 28 AI crawlers including GPTBot, ClaudeBot and PerplexityBot, and llms.txt plus a geo-manifest give answer engines a content map. A weekly citation probe then asks Google AI Overviews, Gemini, OpenAI search and Perplexity whether AIdeazz was actually cited.

How does AIdeazz stop AI from publishing made-up numbers?

The daily article generator collects production evidence before it writes — process uptimes, real commit messages, CRM counts — and a grounding gate refuses to publish any figure that cannot be traced to a measurement. Publishing fails closed: better nothing than something unsourceable. Every incident where a system fooled the operator gets a named failure mode in the AI Ops Wiki.

What is the AI Visibility Audit API?

A free public API that scores any website 0–100 for AI search visibility: 34 automated checks in four weighted categories — AI crawler access 25%, structured data (GEO) 25%, answer-readiness (AEO) 30%, technical foundation 20% — plus a separate verdict for each of six AI crawlers, including GPTBot, ClaudeBot and PerplexityBot. It returns a grade and a prioritised fix list. The same API audits every AIdeazz surface: on 26 October 2026 aideazz.xyz, the podcast and this runbook scored 100/A+. Try it free at aideazz.xyz/api.

Who is this page for?

Business owners who want a plain-language view of what is live, and engineers who want proof. The engineering appendix with repositories, deploy paths, health checks and postmortems lives on GitHub, so this page stays readable and contains no IP addresses, ports or secrets.