Hardening Payment Correctness and Automating Lifecycle Cleanup in a B2B Payments Service

Problem

QuickBooks Bill Pay's B2B payments service accepts payment batches funded through multiple methods, some invalid in combination — Credit Card funding paired with Real-Time Payments (RTP) instructions, for example — and tracks each Payment Instruction through an in-flight lifecycle until it settles or is aborted.

Two correctness gaps existed. Unsupported payment-batch combinations could reach persistence and downstream processing before anything caught them. And Payment Instructions that stalled in an in-flight state, rather than progressing to completion or an explicit abort, had no automated way to be cleaned up.

Context

This is an existing, live B2B payments service (Java 21, Spring Boot), not a greenfield build; I joined the assignment after an internal transfer from Intuit's Data Exchange Platform / IDX team. The service already had established validation ordering, abort-handling conventions, and a production DynamoDB-backed Payment Instruction store, so changes had to fit those conventions rather than introduce a parallel approach.

Constraints

• Existing mixed-funding validation precedence could not be disturbed — a new check had to slot into the existing ordering, not replace it.
• A scheduled job touching payment state had to be safe to run across multiple application instances without double-processing the same Payment Instruction.
• Scan-then-act cleanup can act on stale data if a Payment Instruction changes state between discovery and abort; that race had to be closed, not accepted.
• A full-table DynamoDB scan for stale candidates doesn't scale and can't cheaply return candidates in age order, so discovery needed an access pattern that stays ordered and scales with the number of stale candidates rather than the size of the table.

My Role

I implemented the batch-level validation and payment-state-aware abort-handling fixes end to end, including the shared validation-utility extraction and the feature-flagged rollout. I also designed and implemented the runtime-refreshable expiration TTL, the scheduled stale-Payment-Instruction cleanup job including its DynamoDB GSI, and the failed-credit-card read-path filter. For the GSI I owned the data-access design end to end: sparse-key and shard-key design, projection strategy, and query-level verification against a representative-scale dataset. I did not design the original payment-batch architecture, the Credit Card or RTP systems, the downstream persistence/dispatch architecture, or the base Payment Instruction table. This is payment-correctness hardening, lifecycle automation, and DynamoDB access-pattern design inside an existing system's boundaries, not architecture ownership of the platform.

Architecture / Approach

Payment correctness. The batch validator now rejects Credit Card + RTP combinations at the validation boundary, before a batch reaches persistence or downstream dispatch, returning a structured HTTP 400 with a CC_RTP_UNSUPPORTED sub-code instead of letting invalid state into the system. It covers both the recurring-template creation path and the post-approval scheduling path, and the shared check lives in a single PaymentInstructionUtil rather than being duplicated across them. Rollout is per company behind a feature flag, with Splunk alerting on the rejection path.

Separately, I fixed a batch-abort bug: credit-card Payment Instructions in PENDING_APPROVAL were being aborted by logic meant for other payment states. The fix made abort handling payment-state-aware, leaving abort behavior for other methods and states untouched.

Lifecycle cleanup architecture. Payment Instructions that stay in-flight past their expected window need to auto-abort rather than sit indefinitely. I first made that expiration window a runtime-refreshable, Spring-bound TTL — fractional-minute precision, validated against non-positive input — so it can be tuned operationally without a redeploy.

On top of that, I designed a ShedLock-coordinated scheduled job so exactly one application instance runs each sweep. Candidate discovery began as a paginated full-table DynamoDB scan; I redesigned it around a sparse, sharded DynamoDB GSI. The index is sparse because only PENDING records carry an entry, so a record leaving PENDING drops out of the index on its own. Candidates are spread across shard partition keys with the timestamp as sort key, so each shard queries records older than the TTL cutoff and pages through them oldest-first. Work per execution is bounded, and the starting shard rotates between runs so the same shards don't always absorb the first slice; shard assignment itself stays stable across changes to the shard count. This is application-level access-pattern design, not database infrastructure sharding.

Safe mutation and rollout. The GSI is a discovery mechanism, not authoritative state — a Payment Instruction can resolve itself between discovery and abort. The read path is built around that distinction: eligibility is first screened from data projected onto the index (PENDING status, cutoff, pending-approval state, and detection of incomplete projected data), and then, immediately before mutating, the job re-reads current persisted state and re-verifies eligibility, treating anything already resolved as a no-op rather than an error. Exactly one fresh base-table read survives, inside the existing asynchronous PaymentInstructionService.abort() path, right before dispatch. Screening from projected data took per-candidate base-table reads from two to one for an aborted candidate and to zero for a skipped one, without giving up the read that guards against acting on a stale index entry.

Operational safety around the sweep: a global kill switch, per-company feature-flag gating, and per-candidate failure isolation so one bad Payment Instruction can't halt the run, plus structured logging and metrics (abort volume, candidate age, query size).

A related read-path fix adds null-safe filtering so failed credit-card Payment Instructions no longer appear in PaymentInstructionServiceImpl.getAll() results, leaving ACH and legacy methods and existing ordering untouched.

Key Decisions

• Reject before persistence, not after. The CC+RTP check runs at the validation boundary, so invalid batches never enter persistence or dispatch instead of being cleaned up downstream. The shared check is extracted into PaymentInstructionUtil so the two batch-creation paths can't drift apart.
• Indexed, sharded discovery over a full-table scan. A sparse GSI indexes only PENDING candidates; sharding across partition keys avoids a hot status partition; a per-shard timestamp sort key gives age-ordered paging. Bounded work per run and starting-shard rotation fall out of this decision — they keep each sweep's cost predictable and its load spread as the backlog grows.
• Treat the GSI as discovery data; re-read authoritative state before mutating. This is a correctness decision, not a performance one. The index carries enough projected data to screen out most candidates cheaply, but the job still takes exactly one fresh persisted-state read immediately before abort, so it never mutates on an index entry that has gone stale; already-resolved candidates are a no-op.
• Global kill switch plus staged company-level enablement, not all-or-nothing. A scheduled job that mutates payment state across every company needs both an instant stop and gradual exposure, with per-candidate failure isolation so one bad record can't halt the sweep for everyone.

Trade-offs

Keeping one fresh base-table read inside abort() costs a DynamoDB read per aborted candidate on top of the GSI query — more than trusting the index outright — but a stale-index abort on a live Payment Instruction is a worse bug to carry than that read is to pay for, and removing the redundant earlier fetch paid most of the cost back. A sharded GSI trades some query complexity, fan-out across shards instead of one partition, for avoiding a hot-partition bottleneck as in-flight volume grows. Company-level feature-flagging adds the complexity of two live behaviors to reason about during rollout, in exchange for gradual exposure and an instant stop.

Impact

Implemented behavior. The batch validator rejects unsupported Credit Card + RTP combinations before persistence or downstream dispatch; the pending-approval abort path is payment-state-aware; the Payment Instruction expiration TTL is runtime-refreshable; the scheduled sweep discovers and aborts stale in-flight Payment Instructions through the sharded GSI, with an authoritative-state re-read before each abort; and failed credit-card Payment Instructions are filtered out of Payments UI/API reads.

Validation. The validation and abort-handling changes are covered by a focused test suite plus the full module regression suite. The TTL configuration has a paired configuration change and unit coverage. The scheduled job and GSI are covered by a targeted automated test suite spanning scheduler guards, distributed-job behavior, GSI query and multi-shard aggregation semantics, and concurrent-abort and failure-isolation scenarios, and the GSI was provisioned and query-verified end to end at a representative scale. Representative-scale verification is not production-scale evidence. The read-path filter is validated against the controller contract.

Safeguards. The sweep is gated behind a global kill switch and company-level feature-flag gating, with per-candidate failure isolation so one bad Payment Instruction cannot halt a run.

Ownership boundary. This is payment-correctness hardening, lifecycle automation, and DynamoDB access-pattern design inside an existing platform, not ownership of the payment-batch architecture, the Credit Card or RTP systems, downstream persistence and dispatch, or the base Payment Instruction table. No transaction-volume, latency, revenue, or incident-reduction figures were measured or are claimed.

Lessons

Discovery data is not authoritative state. It's easy to write a scan-then-act job that's correct in the common case and wrong exactly when timing gets unlucky; the fix — one fresh persisted-state read immediately before the mutation — is cheap once you're looking for it.

Performance and correctness weren't in tension once each read had a defined purpose: the index can carry enough projected data to make most reads unnecessary, as long as exactly one fresh read stays right before the state change that can't tolerate staleness. The GSI work was also where I built hands-on DynamoDB depth — sparse keys, shard distribution, projection strategy, query-time trade-offs — reasoning about the database directly rather than through an abstraction.

The original ticket asked for a scheduled full-table scan. Getting to the sharded-GSI, revalidate-before-abort design meant questioning that starting point and reasoning about how the index would actually be queried and where staleness could enter the wider payment-state system — not just closing the ticket as written.