AIdeazz Blog About Portfolio

Pipeline Triage: Why 34 Scraped Contacts Dropped Before Staging

· by

In today's run of atlas-lead-machine, the pipeline evaluated 42 prospect records and staged exactly 8. The remaining 34 were dropped by automated filtering gates before any outreach took place. Out of the raw pool, CRM deduplication removed 22 records, missing inbox routing dropped 12 records, and technical audit verification failed 2 records.

Most lead generation setups measure success by how many rows a scraper pulls from directories or maps. That metric is meaningless. If an automated prospector pushes unvalidated records straight into active CRM deal stages, it pollutes the pipeline, wastes API quota, and triggers rate limits on outbound workers.

Filtering is not an optimization step added later. It is the defensive core of the pipeline.

Gate 1: CRM Deduplication and Existing Pipeline Collisions

The largest drop in the batch happened at the deduplication layer. In atlas-lead-machine.log, 22 records out of 42 were discarded immediately with the marker already-in-CRM.

One specific failure recorded in the log was ↺ Shaka Costa Rica — already in the CRM, skipped. When an agent crawls a directory or re-scrapes a market sector, it repeatedly pulls businesses already progressing through sales stages. If those entities are not checked against existing records in HubSpot, the system creates orphaned duplicate deals. In our current deal tracking, HubSpot shows 1529 deals at appointmentscheduled, 14 at qualifiedtobuy, 80 at decisionmakerboughtin, and 129 at contractsent.

[lead-machine] done · staged 8 · looked at 42 · no-email 12 · already-in-CRM 22 · outside-band 0 · audit-failed 2 · crawler-blocked rescued 0

Without a strict deduplication gate executing before staging, those 22 records would have entered active sequences. In addition to email checks, the pipeline tests phone numbers against existing records. In this execution, the sub-routine for sites lacking domain infrastructure evaluated 10 seen candidates, marked 5 eligible, found 0 already in HubSpot by phone, and staged 2 (hitting its strict safety ceiling of max 2).

Pushing duplicates into CRM pipelines also distorts deal visibility. Our deal-digest.log shows 37 live job deals tracked across active stages, while the HubSpot deals search API shows 135 deals sitting at the "They replied" stage (contractsent) and 0 closed won. When duplicate records slip through, follow-up daemons track redundant threads.

Gate 2: Missing Routing and Email Verification Drops

The second gate screens for operational deliverability. The engine logged no-email 12.

Twelve records had discoverable business names and physical presence, but no functional, direct communication channel. Scraping scripts often attempt to bypass this by substituting generic contact forms, unmonitored registrar addresses, or broken mailto links. If the automated agent cannot isolate a clean address, the prospect is dropped immediately.

Our follow-up systems operate under strict balance constraints. For instance, followup-radar.log monitors multiple active mailboxes:

Meanwhile, reply-radar.log tracks incoming responses, recording 151 scanned passes with 1 automated or own thread skipped and 0 errors. Staging leads with non-existent or invalid email addresses forces delivery workers to absorb SMTP bounces. A bounce damages sending domain reputation across all linked inboxes. A lead without a verified contact channel is dead inventory, and treating it as a viable contact inflates false pipeline volume.

Gate 3: Structural Audits and Technical Eligibility Checks

Passing deduplication and email discovery is still not enough for a prospect to be staged. The pipeline enforces structural quality checks on the target's digital footprint. In this run, the log records audit-failed 2.

When an inbound or outbound lead is evaluated, the business entity must meet baseline digital readiness standards. We run an automated technical visibility engine (visibility-audit.ts) that scores targets across 34 checks in 4 distinct groups:

When a scanned company fails critical technical checks—such as returning an unparseable payload, blocking crawler access entirely, or lacking base technical infrastructure—the lead generation audit failure rate increases, and the record is discarded. In this batch, 2 targets failed these audit criteria and were cut from staging.

Supervisor Telemetry and Execution Isolation

These filtering steps run on an isolated Oracle Cloud instance managed by PM2. System reliability depends on keeping background workers from cascading when external audits or scraping tasks stall.

Currently, PM2 supervises 9 processes, with 8 online:

[PM2] Process cto-aipa online (216 MB) · algom-stream waiting restart (7235 restarts)

The pipeline scripts run as scheduled tasks or isolated cron jobs. When lead generation runs without rigid gates, workers choke on bad data payloads, leading to runaway memory spikes and process loops like the one observed in algom-stream. By cutting 34 low-grade records at the script layer, downstream consumers like atlas-outcomes avoid execution stalls. Recent log runs confirm clean downstream handoffs: pushed to Atlas: {"ok":true,"lanes":8}.

Filtering is not lost yield. Out of 42 raw prospects, the 8 that passed all checks—deduplication, direct email routing, and technical verification—were clean enough to stage directly into git registries and CRM draft cards without human repair.

Frequently Asked Questions

Q: Why drop records for missing emails instead of using an automated address guesser?
A: Permutation-based address guessing generates high bounce rates. Pushing unverified patterns across mailboxes monitored in followup-radar.log (which tracks 771 and 200 inbox threads across accounts) risks domain burning. If a direct address cannot be confirmed during the crawl, the record is dropped.

Q: What caused the 2 audit failures in the run?
A: The audit layer evaluates targets using 34 technical checks. If a prospect's domain fails critical criteria—such as blocking HTTP crawlers, missing canonical structures, or failing base mobile and SSL thresholds—the engine marks it audit-failed and rejects it.

Q: What is the risk of deduplicating only by email?
A: Lead lists frequently contain branch locations or separate team members from the same company without an identical email address. Our pipeline combines direct email lookups with phone number matching against existing HubSpot records, as shown when screening the 10 no-website candidates.

Q: Does rejecting 34 out of 42 leads reduce overall sales volume?
A: No. In our pipeline, HubSpot currently contains 1529 deals at appointmentscheduled and 135 at contractsent with 0 closed won. Staging 34 unvalidated prospects increases API usage and worker load without increasing closed deals. High rejection rates protect outbound deliverability.

— Elena Revicheva · AIdeazz AI Lab · Portfolio · Professional Outlook (PDF)