CDC vs Batch ETL: Why Japanese Data Teams Are Switching

A batch pipeline collects a day's changes, then moves and transforms them in one overnight run. The cost is not the compute, it is the wait. Data sits unqueryable until the window runs, the analytics that follow are always at least a day behind, and every stage between the source and the warehouse adds latency and a place for reconciliation to break. In a legacy stack a change can take three to five days to reach a decision. For anything time-sensitive, from fraud to inventory to personalisation, the answer arrives after it is useful.

CDC reads a database's transaction log, the same log the database already writes for durability, and streams each insert, update, and delete as it commits. Nothing is added to the source and no query load is placed on it, so a production system carries on unaffected. Striim captures the logs of Oracle, SQL Server, PostgreSQL, MySQL, and more than 100 other sources, applies in-flight transformation and enrichment, and delivers to the target in sub-second latency. The data is current the moment it lands, so the analytics on top of it are too.

CDC is not always the answer, and a good architecture uses both. Batch still fits a large historical backfill, a one-time migration of years of records, or a report that genuinely only needs yesterday's numbers, where the simplicity of a scheduled job outweighs latency. CDC wins wherever freshness has value: fraud detection, real-time inventory, live personalisation, operational dashboards, and feeding current data to AI. The test is simple. If a decision loses value as the data ages, the pipeline feeding it should be CDC, not a nightly job.

Moving off batch does not mean a risky cutover. The standard path runs CDC in parallel with the existing batch job: stand up the CDC stream, let it populate the new target, reconcile the two outputs until they match, then retire the batch job once the streaming path is trusted. Because CDC is log-based and non-intrusive, the source database is never taken offline, which matters for a Japanese enterprise that cannot pause a production system. SingleStore Flow provides CDC and bulk ingestion into SingleStore for exactly this pattern, including self-hosted for on-premise workflows.

Japan concentrates the workloads where batch hurts most. Megabanks clear tens of millions of transactions a day, telcos route billions of daily events across their networks, and manufacturers stream terabytes of sensor data, all of it losing value by the hour under a nightly pipeline. Add the data residency rules that push these systems on-premise, and the requirement is specific: sub-second, log-based CDC that runs in the customer's own environment. Both Striim and SingleStore meet that bar with full on-premise parity, which is why the switch is moving from pilot to standard here.

Batch ETL collects a day of changes and moves them in one scheduled run, usually overnight, so analytics are always at least a day behind. Change data capture reads the database transaction log and streams each change as it commits, in sub-second latency. Batch trades freshness for simplicity; CDC keeps the target continuously current.

No. Log-based CDC reads the transaction log the database already writes for durability, so it adds no queries and no write load to the source. That is why it is called non-intrusive and why it is safe against a production system that cannot be slowed down. Striim uses this log-based approach across Oracle, SQL Server, PostgreSQL, and 100+ sources.

Batch still fits when latency does not matter: a large one-time historical backfill, a migration of years of records, or a report that genuinely only needs yesterday's data. In those cases a scheduled job is easier to run than a streaming pipeline. Use CDC whenever a decision loses value as the data ages.

Run CDC in parallel with the existing batch job. Stand up the CDC stream to populate the new target, reconcile its output against the batch output until they match, then retire the batch job once the streaming path is trusted. Because CDC is log-based, the source is never taken offline during the switch.

Yes. Both Striim and SingleStore run fully on-premise with feature parity to their cloud versions, and SingleStore Flow supports self-hosted CDC and bulk ingestion. This lets Japanese enterprises keep regulated data inside their own environment while still moving to sub-second streaming pipelines.