A production delta-index pipeline for search and ads retrieval moved from scheduled batch processing to continuously running micro-batch streaming, cutting worst-case freshness lag from roughly ten minutes to thirty seconds. The system generated an inverted index from ads, campaign, and behavioral data, and previously relied on externally scheduled batch jobs that suffered from compounding delays: Incremental data often waited a full scheduling interval before processing, failures forced re-running entire windows, and traffic bursts could skip or delay runs entirely.
The team first attempted record-level streaming, viewed as the theoretically correct approach, but abandoned it. The indexing logic operated on grouped product and item representations rather than individual records, so per-record updates risked partial-update states and inconsistent search results without meaningful benefit.
The team converged on micro-batch streaming using Spark Structured Streaming with a thirty-second trigger interval, much finer than the five-to-seven-minute partition arrival cadence.
Each cycle compared the latest visible partition against an externally maintained watermark and advanced only when a newer partition appeared, skipping intermediate partitions rather than processing them sequentially. This approach was safe because delta runs used overlapping sliding windows, so skipped partitions were naturally recovered by later runs, with periodic full index rebuilds as a backstop.
An early attempt to reuse success-file and completion-marker logic from the batch system proved unreliable under continuous evaluation, because object storage listings and markers became visible at inconsistent times. Replacing this earlier attempt with deterministic, time-driven watermark comparisons solved the problem.
Long-running execution also introduced gradual memory growth and unpredictable garbage collection pauses. Rather than eliminating these effects, the team scheduled restarts every twenty-four hours and added a watchdog to monitor liveness and enforce restarts, turning failures into routine, recoverable events.
The result was roughly fifty percent lower end-to-end latency, achieved without adopting full record-level streaming.
This content is a short summary of a recent InfoQ article by Parveen Saini, "From Batch to Micro-Batch Streaming: Lessons Learned the Hard Way in a Delta Index Pipeline".
To get notifications when InfoQ publishes content
on these topics, follow "AI, ML & Data Engineering", "Big Data", and "Database" on InfoQ.
The more senior you get, the less pushback you hear. The people who could test your thinking mostly report to you. And the hardest decisions this year, like how to govern AI in production, or when to move an agent to live traffic, you make and defend on your own.
Each cohort in InfoQ's Online Certification Program is led by a working practitioner, someone still doing the job day to day. For five weeks, four hours a week, you work online with them and a small cohort of senior engineers and software architects from other companies. Each week you take a QCon talk and work through how it applies to what you're each building at work.
Upcoming InfoQ Online Certification Programs:
- Architecture with Luca Mezzalira, Principal Solutions Architect.
The sociotechnical side of the job: trade-offs and communication, decentralized decision-making, platform engineering, and the AI architecture decisions you're now being asked to own.
Three intakes are open, starting August 13, September 14, and October 9. Two groups each, 09:30 BST and 10:00 EDT.
- Engineering Leadership with Michelle Brush, Engineering Director for SRE at Google.
Moving from writing the code to setting the technical direction, and being accountable for it. Your decisions get tested against senior peers from other companies. Starting August 21 or October 9, Fridays 10:00 PDT
- AI Security & Privacy Engineering with Katharine Jarmul, author of O'Reilly's Practical Data Privacy.
Securing and governing AI systems in production: sensitive data handling, threat modeling and red teaming, controls and sandboxes, governance and auditing.
Starting August 26, Wednesdays 2:00 PM CEST, with an October 14 intake to follow.
See all cohorts and dates.
/minibooks/agentic-ai-architecture/en/cover/125-emag-Agentic-AI-Architecture-cover-1782835229194.jpg)
