CDC · ANALYTICS Prod → Dev
de850a
RUNNING
started 22s ago
Product details
What CDC is doing, and the two engines it uses to do it
The first mockup is the CDC list — every replication worker this project has spun up, with the freshest one featured at the top. The second is Step 2 of the Start CDC wizard, where the operator picks between Row-window sweep and Binlog-driven capture (with PROs and CONs spelled out). The third is Step 3 — checkpoint policy, retention, FK handling, and where to start reading the source. Below, the cards walk through both engines, the 'set-and-forget' behavior settings, and why a nightly Compare/Reconcile keeps CDC honest.
The list at the top
The first mockup is every CDC worker this project has spun up. The most recently started run is featured at the top with the broadcast icon, the plan name, a copy-able run hash, an amber RUNNING pill with a pulsing dot, a 'started Xs ago' line, and a 'view live →' link that drops into the live event stream. Underneath, the table catalogs every stream — RUN, PLAN, STATUS (RUNNING in amber or STOPPED in gray), STARTED date stamp, and ELAPSED runtime — so a quick glance answers 'what's still going?'
Step 1 — Plan
The Start CDC wizard opens on a single Plan dropdown — pick the saved plan that defines the source, the target, the table pairs, and the column rules. CDC reads from that plan and never goes off-script; if the plan doesn't include a table, CDC won't replicate it. (This is the same dropdown Compare and Reconcile use, for the same reason: Plans are the single source of truth.)
Step 2 — Engine (the choice that matters most)
Row-window sweep is the friendly default — Cupcake periodically scans the source tables in primary-key windows, asks 'what changed since the last bookmark?', and applies the diff. PRO: zero source-side configuration, works on any MySQL-compatible database. CON: lag scales with table size, so very large tables can fall behind. Binlog-driven capture focuses work on changed tables — Cupcake uses the source binlog to identify changed tables and applies row windows to the target. PRO: narrows the tables scanned when changes are localized. CON: the source has to have ROW-format binlog enabled and Cupcake needs a replication-grade user. Check source readiness before starting. A required but unavailable binlog can prevent a stream from starting or resuming.
Step 2 (cont.) — Bootstrap snapshot first
Underneath the engine choice is a toggle: should Cupcake run a baseline Snapshot before starting CDC? Default is ON, because most new streams point at a target that hasn't been populated yet. Flip it OFF only when the target already matches the source state (for example, you just finished a sync and you're attaching CDC to it without a gap).
Step 3 — Behavior (the 'set and forget' tuning)
Checkpoint policy decides how often the worker bookmarks its progress: 'Every N rows', 'On interval', or 'Both' (whichever happens first — the friendliest for variable load). Event retention controls how long Cupcake keeps the CDC event metadata around for debugging. FK checks during apply is the most opinionated knob — the recommended setting for long-running streams is OFF, because strict FK enforcement makes the worker refuse late-arriving child rows and stall the loop. With FK off, the stream stays moving; orphaned-FK drift gets caught by the nightly Compare/Reconcile pair, not by a stuck worker.
Step 3 (cont.) — CDC start position
Three choices: Latest checkpoint (resume from the worker's saved source position — the default for an existing stream being restarted), Current position (start from the source's right-now position, ignoring any history), or Specific position (paste a binlog coordinate or engine-specific marker — useful when reconstructing a specific point-in-time replay). The little summary bar at the bottom of the wizard reads back the choices in plain English — 'Engine: binlog · Checkpoint: both every 10,000 rows or every 30s · FK on' — so the operator catches a misclick before Continue.
Why CDC is always paired with a nightly Compare/Reconcile
Even the most reliable CDC loop will let an orphan row or a stale row through eventually — FK off, late-arriving parent, race conditions on a busy schedule, brief network blip. A nightly Compare run on the same Plan catches drift in row counts; if the number is loud, Reconcile closes the gap. CDC alone gets the target close; CDC + Compare + Reconcile is what keeps it correct.
Feature highlights
What CDC gives back
Two engines, one control surface
Row-window sweep is a PK-window scan that works on any MySQL-compatible source. Binlog-driven capture uses the binlog to focus work on changed tables. Use a PK sweep when the source does not expose its binlog. Check source readiness and the selected mode before starting.
Set-and-forget by design
Checkpoint policy (Both: every N rows or every N seconds, whichever first), retention (7 days), and FK checks disabled is the operator's friendliest combo for long-running streams. The worker bookmarks its progress, restarts cleanly after a worker rotation, and doesn't refuse rows that arrive out of strict FK order — that gets caught later by Compare.
Pair every stream with a nightly Compare/Reconcile
Even the most reliable CDC loop will occasionally let an orphan row or a stale row through (FK off, late-arriving parent, race conditions on busy schedules). A nightly Compare run on the same Plan catches drift in row counts, and a Reconcile closes the gap if it's loud. CDC + Compare + Reconcile is the trio that turns 'we copy data' into 'the target is correct.'