CDC or batch: match the pipeline to the workload
Freshness is one requirement alongside source impact, recoverability and operating cost.
Batch processing and change data capture solve different scheduling and data-movement problems. Some customers need frequent changes for operational decisions; others can work with periodic datasets. A proposal should explain that trade-off and demonstrate the selected method with the actual source and target.
Two update patterns
SC-SP-03 · REV A · 2026.07
Scheduled batch
Read a dataset
defined window
Transform and load
bounded run
Publish results
agreed schedule
Change capture
Read changes
supported source
Transform and write
continuing flow
Monitor lag
recovery process
Illustrative patterns; latency, impact and recovery depend on implementation.
Start with the business timing
Ask when the result is used and what stale data would change. A scheduled report may fit batch processing, while an operational view may need a shorter update interval. Batch frequency, volume and load windows all matter. CDC can reduce the need to reread unchanged rows, but it is not inherently the correct choice for every system.
Check the capture mechanism
Log-based CDC reads supported change records, while other methods may query or use different source facilities. Confirm privileges, source settings, licensing implications and supported versions. Measure source impact during initial load and steady operation. Describing log-based capture as zero overhead or universally nonintrusive would overstate what has been established.
Test consistency and recovery
Define the handover from initial load to ongoing changes. Exercise updates, deletes, ordering, type conversions and schema evolution. Confirm the delivery semantics and recovery behaviour of the exact source-reader and target-writer combination. A claim of exactly-once processing should state its scope and assumptions, including how the target handles retries.
Choose a sustainable operating model
Compare monitoring, backfill, incident handling, source-log retention and costs for both options. Some architectures use batch for bounded backfills and CDC for continuing changes. Agree how the team will detect lag or missing data and how it will validate a replay. Prefer the simplest design that meets the measured requirement.
// Key Takeaways
What to remember
- Choose an update interval from the business task.
- Measure source impact and validate connector behaviour.
- Test backfill, retries, deletes and schema changes before acceptance.
// FAQ
Frequently asked questions
Is CDC always better than batch?
No. The required freshness, change volume, source capabilities and operating effort determine the fit. Some designs use both.
Is log-based capture free of source impact?
No such universal guarantee follows from the capture method. Check configuration and measure initial-load and steady-state impact on the actual source.
Last updated:
Talk to us about selling your product in Japan.
Tell us about your product, the buyers you want to reach and the technical support they will need. We can discuss sales, evaluations and support in Japan.