Data Pipeline Orchestration
An AI-assisted data pipeline that classifies, enriches, and routes incoming records using a combination of prompt engineering and rule-based logic — eliminating manual data triage entirely.
Overview
An operations team was receiving hundreds of unstructured data records daily — form submissions, inbound emails, API payloads — each needing to be classified by type, enriched with additional context, and routed to the correct downstream system or team.
A junior analyst was spending 3+ hours per day on this triage work. The challenge was that the classification rules were complex, context-dependent, and constantly evolving — making a purely rule-based system impractical.
Technical Approach
- Designed a Make.com scenario that intercepts all incoming records from multiple sources (email, forms, APIs) and normalizes them into a standard JSON structure.
- Each record is passed to Claude with a carefully engineered classification prompt that includes examples of each category and edge-case handling instructions.
- Claude returns a structured JSON response with classification, confidence score, extracted entities, and a suggested routing destination.
- High-confidence records are routed automatically; low-confidence records are flagged for human review with Claude's reasoning attached.
- Enrichment step pulls additional context from CRM and external APIs before final routing, ensuring downstream systems receive complete records.
Results & Learnings
The pipeline achieved 98% classification accuracy on the first deployment, with the remaining 2% correctly flagged for human review. Manual triage time dropped to near zero. The analyst now spends their time on exception handling and improving the classification prompt — a much higher-value activity.
Key insight: the hybrid approach (AI classification + rule-based routing) was more robust than either alone. Claude handles the ambiguous cases that rules can't anticipate; deterministic rules handle the high-volume, predictable cases cheaply and reliably.