Engineering Case Study · LinkedIn Data Flow
Building a webhook-driven lead intelligence pipeline in Laravel
LinkedIn Data Flow orchestrates scraping, AI validation, CRM freshness checks, contact enrichment, company intelligence and HubSpot synchronization through queues and webhooks.
- Role
- Pipeline architecture, Laravel backend, Vue interface, queue orchestration, API integrations, testing and deployment.
System Responsibilities
- Data Pipeline
- API Integration
- Automation
Technologies
- Laravel
- Vue
- Apify
- OpenAI
- FullEnrich
- Pappers
- HubSpot
- Queues
Verified Evidence
6 verified results documented below.
Overview
High volumes of B2B profiles needed safe, cost-aware enrichment across several external services without duplicating existing CRM records.
- Process high volumes of profiles safely.
- Avoid unnecessary enrichment costs.
- Prevent duplicate CRM records.
- Keep external services decoupled.
- Track each run and its failures.
- Make the workflow maintainable without a separate orchestrator.
The problem
B2B lead enrichment involves many external services, long-running tasks, rate limits and duplicate data. A sequential request-based implementation would be fragile and difficult to operate.
Constraints that shaped the system
- External APIs complete work asynchronously.
- Provider rate limits and partial failures.
- CRM records may already be fresh.
- Profiles require validation before enrichment.
- Company enrichment depends on previous stages.
- The system must recover from stage-level errors.
System architecture
The platform uses Laravel queues and webhook callbacks as the orchestration backbone. Each stage owns one responsibility, profile batches run concurrently and external completion events resume the pipeline without polling.
An event-driven pipeline, not a chain of fragile scripts.
Each stage has a clear responsibility. External services notify the platform through webhooks, background jobs isolate long-running work and deduplication prevents unnecessary enrichment and CRM noise.
Stage 1 of 8
Search Request
Creates a traceable search run and captures the configuration required by the pipeline.
- Input
- New search request
- Output
- Apify Scraping
- Execution
- Synchronous command followed by asynchronous execution.
- Failure consideration
- Invalid or incomplete search configuration must fail before external work starts.
Cross-cutting concerns
- Queues
- Retry strategy
- Idempotency
- ● Logging
- Error tracking
- Rate limiting
- Batch processing
- Webhook verification
Key engineering decisions
Webhooks over polling
AcceptedExternal APIs complete work asynchronously, and polling repeatedly for completion would add unnecessary execution overhead.
External providers notify the platform when work is complete.
Why
- Reduced unnecessary API calls.
- Lower execution overhead.
- Clear event boundaries.
- Better scalability.
Alternatives considered
- Polling providers repeatedly for completion status
Batch processing
AcceptedHigh volumes of profiles needed to be processed safely without uncontrolled memory growth.
Process profiles in batches of 100 with concurrent validation requests.
Why
- Controlled memory usage.
- Better throughput.
- Easier retry boundaries.
CRM freshness rule
AcceptedCRM records may already be fresh, and re-enriching them would waste cost and add noise.
Skip recently enriched contacts.
Why
- Reduce enrichment cost.
- Avoid CRM noise.
- Respect contact freshness.
Implementation highlights
- Apify-powered profile search.
- AI-assisted profile validation.
- HubSpot deduplication and freshness checks.
- FullEnrich contact enrichment integration.
- Pappers company data integration.
- AI-assisted company intelligence synthesis.
- HubSpot synchronization and run finalization.
Quality and operations
Operations
- Idempotent webhook handling.
- Provider-specific retry policies.
- Secure webhook verification.
- Structured run state tracking.
- Error persistence per run.
- Queue isolation between stages.
- Rate-limit awareness per provider.
Results and evidence
- Implementation fact
Seven processing stages, coordinated through a traceable eight-step operational flow
- Implementation fact
Webhook-driven external orchestration
- Verified metric
Profile batch size100
- Implementation fact
Multi-level CRM deduplication
- Implementation fact
Freshness protection
Skips recently enriched contacts
- Implementation fact
Queue isolation between pipeline stages
Lessons from the system
- Async integrations should be modeled as events and states.
- Cost control is an architectural concern.
- A pipeline needs operational visibility as much as transformation logic.