AIdeazz Blog About Portfolio

Automating Source Rejection: Behind Our Job Board Requalification Failure Alert

· by

A source evaluation script failed hard in our logs: FAIL: only 26.7% still live after removing expired and stale postings (need >=40.0%). The automated monitor dropped the integration on the spot, logged VERDICT: REJECTED, and sent a Telegram alert. When an automated sourcing channel fails to keep fresh postings above your health floor, letting agents keep polling it wastes API tokens, poll cycles, and review capacity on expired ghost vacancies.

Treating job ingestion feeds as persistent, reliable pipes is a mistake. Without automated re-qualification gates, scrapers and external feeds slowly rot until an operator spends hours diagnosing why outbound applications are landing in closed recruiting queues.

The Re-Qualification Gate: 26.7% vs 40.0%

In our setup on Oracle Cloud, the scraping pipeline runs under PM2 management. Process serpapi-jobs runs online (with 8 restarts across 4 days of uptime, consuming 117 MB of memory) to gather raw postings from configured boards. Raw ingestion alone is insufficient. If a board retains links to closed requisitions, downstream agents build customized CVs and draft pitches for jobs that no longer exist.

Our watcher script job-board-watch runs health audits against tracked sources. The exact log line produced 32.5 hours ago captured the decision:

FAIL: only 26.7% still live after removing expired and stale postings (need >=40.0%)
VERDICT: REJECTED
RESULT: a wired board FAILED re-qualification (Telegram alert sent)

The cutoff is strict: if a monitored channel cannot sustain a live posting rate of 40.0%, it fails qualification. Dropping to 26.7% triggered an immediate job board requalification failure. The system rejected the source, severed the ingestion pipeline, and dispatched an alert directly to our operations chat on Telegram. The agent refuses to guess whether the board will recover tomorrow; it stops routing candidate profiles to dead requisitions today.

Guarding the Pipeline from Scraping to CRM

Discarding a bad board prevents downstream poison from entering the rest of our architecture. Downstream from ingestion, our PM2 supervisor keeps 9 online processes running out of 9:

When source validation fails at the boundary, you protect the CRM stages from bloat. Our deal digestion job (deal-digest.log, running against HubSpot) monitors live deals. In our latest scan, the CRM digest parsed 27 live job deals across active pipeline stages: qualifiedtobuy=12, presentationscheduled=0, decisionmakerboughtin=4, contractsent=6, and appointmentscheduled=5.

HubSpot lifetime totals across stages record qualifiedtobuy=12, presentationscheduled=0, decisionmakerboughtin=75, contractsent=122, and appointmentscheduled=1495. If job-board-watch had not dropped that failing board at 26.7%, dozens of zombie postings would have bloated those stages, demanding tailoring and document generation for positions that had already expired.

Auditing Apply Artifacts and Routing Failures

Every validated posting moves into staging where tailored resumes are assembled. Our apply-kit-audit.log shows the strict matching requirements verified prior to dispatch:

✓LINK ✓L ✓CV ✓D ✓R ✓C Implementation / Solutions Consultant @ Puente Talent Partners
CV_Elena_Revicheva_Puente_Talent_Partners_Implementation_Solutions_Consultant.pdf
✓LINK ✓L ✓CV ✓D ✓R ✓C Technical Account Manager @ Puente Talent Partners
CV_Elena_Revicheva_Puente_Talent_Partners_Technical_Account_Manager.pdf

Generating these assets requires real compute, calling libraries like @anthropic-ai/sdk, express, dotenv, and document parsers. When files pass audit, apply-queue.log batches the notifications:

Feeding this assembly line from a source with only 26.7% live listings wastes time and resources. When scrapers push ghost postings into the apply queue, real human attention gets wasted reviewing roles that return automated rejection notices within minutes.

Mail Watches and Feedback Loops

When applications are active, our inbound monitoring infrastructure tracks whether companies actually respond. The inbound radar scripts (reply-radar.log and followup-radar.log) monitor mailboxes continuously:

In repo VibeJobHunterAIPA_AIMCF (5 commits in the last 48 hours), we hardened the inbound mail loop to handle delivery quirks. Commit 1233eb0 ensured that LinkedIn and Workable notices alert on Telegram without generating spurious deals named after the platform. Commit 6d6bc64 instituted hot-mail alerts with 3 delivery retries and plain-text fallback, handling no-reply ATS senders like Teamtailor and CI&T. Commit 27b84ef expanded the mail watch to a 72-hour window, routing every failure directly to Telegram. Commit d45de62 established a UID cutover, and commit 7c64e7b ensured IMAP scanning scans Spam without marking mail read or imposing an artificial 50-mail cap.

Every layer of this system—from inbound mail classification to outbound sourcing—depends on verifiable thresholds. When a source drops to 26.7% live vacancies against a 40.0% health floor, shutting it down immediately is the only way to keep the operational pipeline clean.

Frequently Asked Questions

Q: Why enforce a strict 40.0% live posting floor instead of just filtering individual expired listings on ingestion?
A: Filtering individual links catches known dead pages, but an overall drop to 26.7% signals that the remote board is scraping stale aggregations or failing to synchronize its index. Ingesting from a decayed index causes candidate evaluation runs to burn API calls on roles that return silent rejections or expired forms.

Q: How does job-board-watch verify whether a posting is stale or live?
A: I do not have that measured in the available logs beyond the recorded result: FAIL: only 26.7% still live after removing expired and stale postings (need >=40.0%).

Q: What happens downstream after the Telegram alert fires on a re-qualification failure?
A: The board status transitions to VERDICT: REJECTED, disabling the source from future polling passes by serpapi-jobs until manual inspection or a subsequent scheduled re-qualification pass succeeds.

Q: Does disconnecting a source affect existing CRM deals?
A: No. In-flight deals remain untouched in their respective pipeline stages (such as the 12 deals in qualifiedtobuy or the 6 deals in contractsent), ensuring ongoing correspondence is tracked through our IMAP monitors without disruption.

— Elena Revicheva · AIdeazz · Portfolio