- One lab, many products: tutoring bots, job hunter, social growth, CRM pipeline, daily blog, and a marketing radar (Atlas) that reads the ad market and exports test campaigns — with real Google Analytics feedback since July 2026.
- Honest numbers: 190+ essays published, 29 wiki incidents and 23 named failure modes, ~181K lines of code read from production (dossier); CRM holds 3,200+ deals (live count, 26 Oct 2026); revenue is still effectively zero — the systems are the proof, not the sales curve.
- Risk control: automated health checks and isolated secrets mean one bot failing does not take down the whole fleet.
- Why this page exists: public proof that AIdeazz runs like a product company — not slide decks — so clients, partners, and search engines can verify operational maturity.
What AIdeazz AI Lab is — right now.
Your production AI co-founder fleet plus a marketing engine that connects discovery → content → leads → measurement. Not a prototype.
Start hereThe pieces of the lab12 pieces
Public facePortfolio, blog, inquiry form
Portfolio, blog, inquiry form
Where to see it: aideazz.xyz/portfolio
AI Visibility AuditFree live API + widget — score any site 0–100 on GEO · AEO · tech SEO (ChatGPT / Perplexity / Claude / Gemini readiness)
Free live API + widget — score any site 0–100 on GEO · AEO · tech SEO (ChatGPT / Perplexity / Claude / Gemini readiness)
Where to see it: aideazz.xyz/api
Tech hubCRM, blog, outreach, performance tracking
CRM, blog, outreach, performance tracking
Where to see it: CTO AIPA Telegram
Marketing radarWeekly ad-market intelligence on the lab’s seven sellable services — open windows arrive as ready-to-run sales briefs, and CRM outcomes feed back so the radar learns what converts
Weekly ad-market intelligence on the lab’s seven sellable services — open windows arrive as ready-to-run sales briefs, and CRM outcomes feed back so the radar learns what converts
Where to see it: Atlas live radar
CRM spineAll agents feed one HubSpot pipeline
All agents feed one HubSpot pipeline
Where to see it: HubSpot Sales Pipeline
Measure layerWebsite sessions, form leads, deals by campaign
Website sessions, form leads, deals by campaign
Where to see it: Atlas concept cards (performance block)
Learning recordAI Ops Wiki — 29 production incidents and 23 named failure modes, each incident with a named failure mode and a verified number. Generated, never hand-edited
AI Ops Wiki — 29 production incidents and 23 named failure modes, each incident with a named failure mode and a verified number. Generated, never hand-edited
Where to see it: ai-ops-wiki.html
Provenance dossierNine systems and why they exist — engineering provenance for every product, figures read from production
Nine systems and why they exist — engineering provenance for every product, figures read from production
Where to see it: nine-systems
Publishing engine190+ essays (grounded daily blog), 11-episode podcast, Dev.to canonical crossposts
AI Film StudioAtuona — source clips in, finished film out: automated edit, sidechain-ducked mix, −16 LUFS master. 9 films live
Atuona — source clips in, finished film out: automated edit, sidechain-ducked mix, −16 LUFS master. 9 films live
Where to see it: atuona.xyz/aifilmstudio
Investor / narrative deckThe live pitch — September 2026 actualization, honest traction
The live pitch — September 2026 actualization, honest traction
Where to see it: pitch.html
Sibling productsEspaLuz tutoring, job hunter, social bots, creative AI, daily briefing
EspaLuz tutoring, job hunter, social bots, creative AI, daily briefing
Where to see it: Portfolio
AI visibility · re-audited 26 Oct 2026 with the API itselfScores of our own surfaces6 surfaces
aideazz.xyzportfolio + blog100 · A+
aideazz.xyz — portfolio + blog
AI crawlers allowed: 6 / 6
podcast.aideazz.xyz11 episodes, each linked to its post100 · A+
podcast.aideazz.xyz — 11 episodes, each linked to its post
AI crawlers allowed: 6 / 6
This runbook100 · A+
This runbook
AI crawlers allowed: 6 / 6
/apithe audit tool itself98 · A+
/api — the audit tool itself
AI crawlers allowed: 6 / 6
AI Ops Wiki29 incidents · 23 failure modes93 · A+
AI Ops Wiki — 29 incidents · 23 failure modes
AI crawlers allowed: 6 / 6
atuona.xyzAI film studio & gallery93 · A+
atuona.xyz — AI film studio & gallery
AI crawlers allowed: 6 / 6 — was 69 · C and 0 / 6 on 19 Sep: the audit's own finding, since fixed
How it runsThe marketing engine2 views
GEO / AEO / Tech SEO across the productsFrom a build session to an AI answer engine citing it
Build sessions → AI Ops Wiki incidents (published nightly, alerts on failure) → grounded daily blog (190+ posts, EN/ES) + Dev.to canonical → podcast episodes that link back to their posts → llms.txt, a dated geo-manifest and a 215-URL sitemap that name the API, blog, wiki and podcast for AI crawlers → distribution on LinkedIn, Instagram and YouTube (Buffer / Make) and X (Algom Alpha, X API) → a weekly citation probe asks Google AI Overviews, Gemini, OpenAI search and Perplexity whether any of it is actually cited.
The closed marketing loopLive since July 2026 · 11 steps, CRM outcomes flow back
Status · 27 Oct 2026What is loaded, partial and next3 lines
LoadedDiscovery, grounded daily blog (one article every day, 20–27 Oct), one CRM endpoint for every agent, triage, Atlas Monday radar + lead machine, Lead Concierge (daily self-test: PASS), read-only mail watch on two inboxes, WhatsApp click tracking, CRM outcomes back into Atlas
Discovery, grounded daily blog (one article every day, 20–27 Oct), one CRM endpoint for every agent, triage, Atlas Monday radar + lead machine, Lead Concierge (daily self-test: PASS), read-only mail watch on two inboxes, WhatsApp click tracking, CRM outcomes back into Atlas
PartialConversion — measured in HubSpot on 27 Oct: 119 of 333 judged-fit job deals reached “they replied”; 68 Atlas leads contacted; website inquiries 9 sent → 1 reply. Turning replies into paid work is the current game.
Conversion — measured in HubSpot on 27 Oct: 119 of 333 judged-fit job deals reached “they replied”; 68 Atlas leads contacted; website inquiries 9 sent → 1 reply. Turning replies into paid work is the current game.
NextX API credits for Algom; showcase package for clients; automated weekly blog from each Atlas snapshot
X API credits for Algom; showcase package for clients; automated weekly blog from each Atlas snapshot
Marketing roadmapSix phases and where each stands6 phases
No terminal requiredHow to verify it yourself5 checks
How much original code runs in production?
Nine systems and why each exists, with figures counted from the running code.
How much has the lab published?
Every figure traced to a measurement before it ships; fails closed otherwise.
How many production failures are documented?
29 production incidents, each with a named failure mode and a verified number.
How many manual steps are left between a commit and a post?
A commit worth talking about reaches LinkedIn, the blog, X and Instagram without anyone retyping it.
Products & agents. Each row is a shipping capability.
What customers or partners touch, with how it runs underneath (Linux services, process supervisor, or serverless). Naming matches the internal resilience matrix without exposing infrastructure coordinates.
The fleetEvery agent, one line each12 entries · 10 live
01EWEspaLuz WhatsAppSpanish tutoring on WhatsApp — conversation, drills, corrections.
02ETEspaLuz TelegramThe same tutoring on Telegram, with a kids mode.
03EIEspaLuz InfluencerOne brand post a day at 23:00 Panama.
espaluz-influencer.04AAAlgom Alpha (@reviceva)Always-on X presence — paused until X API credits return.
dragontrade-main and satellite processes. Since 17 Oct 2026 the X API credits are depleted: the processes run, the output is zero — shown because this page reports output, not uptime.05VJVibeJob HunterHourly job search across 13+ boards, judged and staged in HubSpot.
vibejobhunter + Bright Data ingest serpapi-jobs.06CMAI Marketing Co-Founder (CMO AIPA)One LinkedIn post a day, led by real engineering milestones.
vibejobhunter-web exposes an internal health route.07OCOpenClaw (interview coach)Mock interviews by voice and text, English and Spanish.
openclaw-gateway; probed via private health URL on the app host.08CTTech Co-Founder (CTO AIPA)Control tower + CRM hub every other agent writes through.
cto-aipa), Oracle Autonomous DB via wallet-based TLS — credentials never live in this HTML.08.1SBSprint Briefing (Sprinter)Daily audio briefing from the month’s board, voice notes and commits.
09ATCreative Co-Founder (Atuona CCF)Creative partner with a fully automated AI film studio.
10ASAtlas Shifted (Marketing Strategist)Reads the live ad market every Monday and stages leads.
11ALAILALong-horizon orchestration — architecture, not yet running.
The fleetCross-cutting capabilities7 shared layers every agent stands on
/research_company · /research_employer · /research_competitor in Telegram)The fleetHow the agents hand work to each otherthrough HubSpot, files and schedules
- One CRM spine: VibeJob Hunter, Atlas, WHITESPACE, EspaLuz and Algom all write to HubSpot through CTO AIPA’s single endpoint, each with its own tag — so every deal says which agent made it.
- Inquiry → reply: a website form or chat becomes a HubSpot deal, AI drafts the answer, Elena taps ✅ in Telegram, Resend sends it from the company domain and stamps delivery on the deal.
- Job post → interview: VibeJob Hunter stages the deal → her ⏳/❌ tap moves it → “🔥 I act today” triggers the apply kit’s letter, CV and defenses → OpenClaw coaches her for that exact company from the deals sheet.
- She teaches the judge: her stage moves and reasons are read back every hour into the job judge’s feedback.
- Inbox → deal: a reply in Gmail or Zoho becomes a note, a task and a Telegram alert on its deal — read-only, so her mail stays unread until she opens it.
- Atlas learns from sales: Monday’s market windows and leads go into HubSpot; their real stages come back into Atlas the same morning.
- Code → content: a
feat:push to GitHub becomes a milestone that CTO AIPA hands to the CMO for LinkedIn, X and Instagram.
The fleetWhat each agent produced in HubSpot3,220 deals, counted 27 Oct 2026
Reliability & uptime practice. A failure must reach a human, not a log.
For founders: scheduled probes ask each product whether it still responds. For engineers: one bash driver on the primary Oracle VM runs roughly every five minutes; keep-alive traffic avoids idle reclamation; systemd caps restart storms.
No noisy herd restarts
Watchdog · ~5 min detectionFailed probes recycle only the affected unit. Concrete URLs and scripts remain in the private appendix — not pasted here.
Model retirements are config, not rewrites
Providers retire hosted models every few months (Groq drops Llama 3.3 for its dev tier in Aug 2026). Every agent resolves its model id through one fleet-wide switch, so a cutover is a one-line .env change — tested against a golden-set eval before flipping, and instantly reversible. The switch shipped fleet-wide in July 2026, weeks ahead of the deadline. For founders: the AI vendor treadmill is priced into the architecture, not paid for in rebuild weeks.
Two silent failures surfaced the same week
The wiki publisher had been jammed for 15 days by an interrupted rebase that one catch {} swallowed every night, so the AEO manifest froze while every run looked finished. The morning briefing was running a three-month-old Lambda bundle with a revoked token and a retired model id — the fleet-wide model switch had never reached the one agent deployed off the VM. Fixes: the publisher now syncs before it generates and alerts Telegram on any failure; the Lambda was rebuilt from source and joined the switch. Rule since: verify the deployed artifact, not the config.
ReliabilityHealth check per agent7 checks
CTO AIPA + AtuonaOrchestrator HTTP OK via localhost probe
Health check: Orchestrator HTTP OK via localhost probe
Recovery: pm2 restart cto-aipa
EspaLuz WhatsAppTutoring webhook answers OK from localhost
Health check: Tutoring webhook answers OK from localhost
Recovery: systemctl restart espaluz-whatsapp
VibeJob Hunter + CMOMarketing bridge health endpoint OK internally
Health check: Marketing bridge health endpoint OK internally
Recovery: systemctl restart vibejobhunter-web vibejobhunter
OpenClaw ShortlistHTTP GET gateway loopback → 200
Health check: HTTP GET gateway loopback → 200
Recovery: systemctl restart openclaw-gateway
Sprint BriefingCloudWatch + EventBridge schedule
Health check: CloudWatch + EventBridge schedule
Recovery: Lambda retries / DLQ policy
PM2 stacks (e.g. Algom)cron HTTP + pm2 jlist status online
Health check: cron HTTP + pm2 jlist status online
Recovery: pm2 restart <app>
All systemd agentsProcess liveness via systemctl
Health check: Process liveness via systemctl
Recovery: systemd restart policy
Go-to-market automation. Ship once, publish everywhere, answer first.
When engineering ships something worth talking about, the stack fans it out across LinkedIn, blogs, X, and Instagram — without a human retyping the same story five times. And when a lead comes back in, the same stack answers first.
feat:, launch:, or release: notify the marketing agent. Housekeeping commits (fix:, docs:, chore:, …) stay invisible to customers.Go-to-marketThe pipeline, step by step7 steps
GitHub WebhookCommit detected → CTO AIPA
Push events hit the secured webhook. Groq/Claude review diffs, classify milestones, enqueue pending updates for downstream marketers.
LinkedIn · 20:00 PanamaCMO generates + posts
Claude Sonnet copy → Make.com delivery. Zero manual paste.
Daily blog · aideazz.xyz/blog + dev.toGrounded blog publish — fail-closed on claims
The daily publisher collects production evidence before it writes and refuses to publish any number it cannot trace to a measurement (grounding gate, Aug 2026). Near-duplicates are blocked by token overlap, not slug equality. Ships to aideazz.xyz/blog with dev.to canonical crosspost; sliding-window mutex and always-notify Telegram guard against silent double publishes.
X · Every 5th post slotAlgom Alpha tweet
x-tech-updater.js merges milestones in plain language (Haiku / Groq), guarded against duplicate queue states.
Instagram · Even days 18:00 PanamaEspaLuz Influencer
Milestone-aware caption + Make.com media pipeline; falls back to standard queue when nothing pending. Since Sep 2026 the same channel also carries the AI Visibility API campaign two days in three (EspaLuz on the third), alternating English and Spanish.
Inbound · HubSpot → Claude Fable 5 → TelegramLead Concierge — form to sent reply in minutes
Portfolio inquiries and HubSpot chat become CRM contacts in seconds (server-side push, no cron wait). Claude Fable 5 drafts the reply — right language, right proof links — and a dedicated concierge watchdog re-drafts through the five-provider chain (Claude → OpenAI → Gemini → Grok → Groq) if the primary dies, so a single vendor outage cannot kill the inbox; on 19 Aug the chain absorbed a dead primary in 2.4 seconds. The draft lands in Telegram with Send / Edit / Skip buttons; one tap emails the lead from the company domain via Resend (optional Cc + magic-byte-checked PDF attachments) and logs the sent text to the contact’s CRM timeline. Full audit trail, human approval on every send.
Outbound · leads found → HubSpot on autopilotAutomatic lead search + CRM that works itself
Prospects are discovered automatically — SERP, hiring boards, product launches, Bright Data enrichment — and staged into HubSpot as company + deal + contact + note, each note carrying a tested WhatsApp link and a one-click outreach email. From there the CRM works itself: an outgoing email is auto-detected and moves the deal to Sent while a soft follow-up task (+4 days) is created; Resend webhooks mark delivered / opened on the deal — true delivery events, not assumptions (delivered · bounced · complained · delayed · opened · clicked); a reply moves the deal to Replied and closes the open follow-ups. Every record names its author agent by prefix, so provenance is a filter, not an investigation.
Don’t take the diagram’s word for it — this form IS the pipeline.
Submit it and you become a UTM-attributed HubSpot contact within seconds, Claude drafts a reply with full context of this operation, and it reaches my Telegram for one-tap approval. You’ll hear back fast — that’s the whole point of step 6.
Release discipline. We ship like a product company.
Board-friendly translation: predictable processes, isolated secrets, verifiable rollouts — even though agents move faster than most teams.
Release disciplineThe six release rules6 rules
01One live checkout per codebaseCheckout
Eliminates “which folder is prod?” debates; paired bots share code intentionally but never duplicate repos.
02Green build, then swapBuild
Pull latest → compile/tests succeed → only then restart supervised processes. Broken artifacts never replace what customers already rely on.
03Secrets isolationSecrets
Each bot owns its environment file; crypto wallets never touch GitHub; TypeScript strict mode catches sloppy typings before prod.
04Crash-proof process registryRegistry
Every agent process is registered to auto-start on server boot and auto-restart on failure — no manual babysitting after a power cycle or kernel update.
05No silent failuresSignals
Crash handlers log before exit so supervisors show why something died; watchdog cadence targets ~5 minute detection.
06Verify after deployVerify
Health signal green, database connectivity logs clean, one real Telegram interaction — all pass before the incident is closed.
Incident response template. Symptoms, causes, fix, proof.
For stakeholders: regressions are handled like financial reconciliations — symptoms, compounded causes, fix, proof — so the same automation trap rarely strikes twice.
PostmortemsTwo incidents, written up in full2 incidents
Duplicate side effectHubSpot duplicate posting loop
Pending HubSpot milestones resurfaced every x-tech-updater.js cycle.
Triple mismatch: legacy posted vs filter on posted_x; mark endpoint keyed on timestamp while older rows used received_at; backlog needed posted_x backfill.
GET excludes either flag; mark endpoint tries timestamp → received_at → title; JS client sends title for fallback matching.
API snapshot {"ok": true, "pending": [], "total": 0, "held": true} + two full automation cycles without duplication.
≈2 hours from detection → patched APIs → verified on live automation cycles. Full narrative retained in the engineering appendix linked below.
Vacuous health checkThe engagement loop that never ran
Asked the logs to prove a claimed engagement rate. Startup banner found 4,357 times; cycle-completed action line found zero times. The behavior never happened, no matter what the config said.
Three layers deep: the first engagement run was scheduled 5 minutes after startup; the process was being restarted every 5 minutes by an external cron; and that cron was a health check whose grep never matched PM2’s box-drawing table output — so it judged a healthy process dead, forever.
Health check rewritten to read structured state: pm2 jlist | jq on the process status field instead of grepping rendered text. Process stayed up; the first engagement cycle in the bot’s history fired the same day, with real replies and follows verified from logs.
Verify from logs, not config. Never claim agent behavior without grepping for the ACTION line (not the setup line). The fix is now a standing operating rule across the fleet.
Kintsugi, not cover-ups
SOP update rule: materially production-facing incidents earn the same structured write-up — so institutional memory compounds instead of resetting. The full running record, with each failure mode named and defined in plain language, lives in the AI Ops Wiki.
aideazz.xyz/ai-ops-wiki.html ↗Stack reference. Boring where uptime matters, sharp where it differentiates.
Boring reliability primitives where uptime matters; sharp AI + CRM + social APIs where differentiation matters.
Stack referenceTen layers of the stack10 layers
Process Mgmt3 tools
Databases2 tools
CRM & Outreach3 tools
Social3 tools
AI / LLMs9 tools
Lead Gen4 tools
Hosting / CDN4 tools
Monitoring5 tools
Content & SEO9 tools
Project Mgmt2 tools
Questions people actually ask. Answered plainly.
The questions that come up most — and marked up so AI answer engines can lift them directly.
What is the AIdeazz AI Lab?
A production fleet of eleven AI co-founder agents plus a marketing engine, running 24/7 on a single Oracle VM at zero monthly infrastructure cost. It publishes content daily, captures leads into a CRM, and measures whether campaigns moved the needle rather than reporting vanity metrics.
How does AIdeazz keep AI agents from dying silently?
Every product declares a health check, a restart command and a process name in one resilience config. A watchdog polls those checks and restarts a dead process, and failures surface in Telegram rather than waiting to be noticed. Silent death was the original failure mode, so a service that stops answering is treated as an incident, not a gap in the graph.
Does the marketing engine generate leads, or only traffic?
Inbound inquiries post from the website into Oracle and HubSpot instantly, then a Make scenario drafts a reply that is approved from Telegram before it sends. More than 3,200 deals and 1,500 contacts exist in the CRM (counted 26 October 2026). Attribution is instrumented end to end with UTM tags, though campaign volume is still low.
How is content optimized for AI search, not just Google?
A 34-check audit engine scores every page for AI crawler access, structured data, answer-readiness and technical foundation. robots.txt explicitly allows 28 AI crawlers including GPTBot, ClaudeBot and PerplexityBot, and llms.txt plus a geo-manifest give answer engines a content map. A weekly citation probe then asks Google AI Overviews, Gemini, OpenAI search and Perplexity whether AIdeazz was actually cited.
How does AIdeazz stop AI from publishing made-up numbers?
The daily article generator collects production evidence before it writes — process uptimes, real commit messages, CRM counts — and a grounding gate refuses to publish any figure that cannot be traced to a measurement. Publishing fails closed: better nothing than something unsourceable. Every incident where a system fooled the operator gets a named failure mode in the AI Ops Wiki.
What is the AI Visibility Audit API?
A free public API that scores any website 0–100 for AI search visibility: 34 automated checks in four weighted categories — AI crawler access 25%, structured data (GEO) 25%, answer-readiness (AEO) 30%, technical foundation 20% — plus a separate verdict for each of six AI crawlers, including GPTBot, ClaudeBot and PerplexityBot. It returns a grade and a prioritised fix list. The same API audits every AIdeazz surface: on 26 October 2026 aideazz.xyz, the podcast and this runbook scored 100/A+. Try it free at aideazz.xyz/api.
Who is this page for?
Business owners who want a plain-language view of what is live, and engineers who want proof. The engineering appendix with repositories, deploy paths, health checks and postmortems lives on GitHub, so this page stays readable and contains no IP addresses, ports or secrets.