Hardening Payment Correctness and Automating Lifecycle Cleanup in a B2B Payments Service

Problem

QuickBooks Bill Pay's B2B payments service accepts payment batches funded through multiple methods, some invalid in combination — Credit Card funding paired with Real-Time Payments (RTP) instructions, for example — and tracks each Payment Instruction through an in-flight lifecycle until it settles or is aborted.

Two correctness gaps existed. Unsupported payment-batch combinations could reach persistence and downstream processing before anything caught them. And Payment Instructions that stalled in an in-flight state, rather than progressing to completion or an explicit abort, had no automated way to be cleaned up.

A third gap sat on the recovery path. When a credit-card payment in a batch failed, the batch-payment polling API did not tell the client which failures were actually eligible for another attempt. Retryability was left for the client to infer from raw status codes, so the "Reschedule" action could be offered for failures that would never succeed on a retry, and the failure classification and the retryability signal could contradict each other.

A fourth gap only appeared in production. After a payment lifecycle-state change, mixed-outcome batches, where some payments succeed and others fail, could produce incorrect downstream notification behavior.

Context

This is an existing, live B2B payments service (Java 21, Spring Boot), not a greenfield build; I joined the assignment after an internal transfer from Intuit's Data Exchange Platform / IDX team. The service already had established validation ordering, abort-handling conventions, and a production DynamoDB-backed Payment Instruction store, so changes had to fit those conventions rather than introduce a parallel approach.

Constraints

• Existing mixed-funding validation precedence could not be disturbed — a new check had to slot into the existing ordering, not replace it.
• A scheduled job touching payment state had to be safe to run across multiple application instances without double-processing the same Payment Instruction.
• Scan-then-act cleanup can act on stale data if a Payment Instruction changes state between discovery and abort; that race had to be closed, not accepted.
• A full-table DynamoDB scan for stale candidates doesn't scale and can't cheaply return candidates in age order, so discovery needed an access pattern that stays ordered and scales with the number of stale candidates rather than the size of the table.
• Existing notification, aggregation, and backward-compatible behavior had to stay intact; the remediation could correct how payment state was interpreted, but not change what downstream consumers already depended on.

My Role

I implemented the batch-level validation and payment-state-aware abort-handling fixes end to end, including the shared validation-utility extraction and the feature-flagged rollout. I owned the backend retry-policy slice of the Credit Card Online Execution flow: the failure taxonomy, the failure-to-retryability mapping, its precedence and legacy-fallback rules, and the typed failure metadata the batch-payment polling API exposes for it. I also designed and implemented the runtime-refreshable expiration TTL, the scheduled stale-Payment-Instruction cleanup job including its DynamoDB GSI, and the failed-credit-card read-path filter, and worked past execution into the downstream services, retargeting and hardening how credit-card payments get booked into QuickBooks. I also owned the investigation and remediation of a cross-service payment-state defect. For the GSI I owned the data-access design end to end: sparse-key and shard-key design, projection strategy, and query-level verification against a representative-scale dataset. I did not design the original payment-batch architecture, the Credit Card or RTP systems, the downstream persistence/dispatch architecture, the base Payment Instruction table, or the reschedule UX. This is payment-correctness hardening, backend domain-rule ownership for payment recovery, lifecycle automation, and DynamoDB access-pattern design inside an existing system's boundaries, not architecture ownership of the platform.

Architecture / Approach

Payment correctness. The batch validator now rejects Credit Card + RTP combinations at the validation boundary, before a batch reaches persistence or downstream dispatch, returning a structured HTTP 400 with a CC_RTP_UNSUPPORTED sub-code instead of letting invalid state into the system. It covers both the recurring-template creation path and the post-approval scheduling path, and the shared check lives in a single PaymentInstructionUtil rather than being duplicated across them. Rollout is per company behind a feature flag, with Splunk alerting on the rejection path.

Separately, I fixed a batch-abort bug: credit-card Payment Instructions in PENDING_APPROVAL were being aborted by logic meant for other payment states. The fix made abort handling payment-state-aware, leaving abort behavior for other methods and states untouched.

Credit-card failure taxonomy and retry eligibility. When a credit-card payment fails, the client needs one clear answer: can this be rescheduled or not. I moved that decision into the backend core domain layer instead of leaving each client to interpret raw status codes. A canonical failure taxonomy classifies each failure (issuer declined, skipped after a prior decline, risk decline, not processed, and legacy status cases), and a single mapping turns that classification into a retryable or non-retryable outcome. Persisted failure information (FailureType) is authoritative when present; when it is missing, the mapping falls back to the legacy BillPaymentStatus so older records still resolve to a defined answer. The batch-payment polling API now carries this as typed metadata on each failed payment, failureDetails.type and failureDetails.isRetryable, so the UI can offer a reschedule action only for failures that are genuinely eligible. I also closed a consistency gap where the failure type and the retryability flag could disagree, so the two fields now always describe the same outcome, and covered the mapping, its precedence and fallback rules, and the decline and recovery edge cases with a regression suite.

Lifecycle cleanup architecture. Payment Instructions that stay in-flight past their expected window need to auto-abort rather than sit indefinitely. I first made that expiration window a runtime-refreshable, Spring-bound TTL — fractional-minute precision, validated against non-positive input — so it can be tuned operationally without a redeploy.

On top of that, I designed a ShedLock-coordinated scheduled job so exactly one application instance runs each sweep. Candidate discovery began as a paginated full-table DynamoDB scan; I redesigned it around a sparse, sharded DynamoDB GSI. The index is sparse because only PENDING records carry an entry, so a record leaving PENDING drops out of the index on its own. Candidates are spread across shard partition keys with the timestamp as sort key, so each shard queries records older than the TTL cutoff and pages through them oldest-first. Work per execution is bounded, and the starting shard rotates between runs so the same shards don't always absorb the first slice; shard assignment itself stays stable across changes to the shard count. This is application-level access-pattern design, not database infrastructure sharding.

Safe mutation and rollout. The GSI is a discovery mechanism, not authoritative state — a Payment Instruction can resolve itself between discovery and abort. The read path is built around that distinction: eligibility is first screened from data projected onto the index (PENDING status, cutoff, pending-approval state, and detection of incomplete projected data), and then, immediately before mutating, the job re-reads current persisted state and re-verifies eligibility, treating anything already resolved as a no-op rather than an error. Exactly one fresh base-table read survives, inside the existing asynchronous PaymentInstructionService.abort() path, right before dispatch. Screening from projected data took per-candidate base-table reads from two to one for an aborted candidate and to zero for a skipped one, without giving up the read that guards against acting on a stale index entry.

Operational safety around the sweep: a global kill switch, per-company feature-flag gating, and per-candidate failure isolation so one bad Payment Instruction can't halt the run, plus structured logging and metrics (abort volume, candidate age, query size).

A related read-path fix adds null-safe filtering so failed credit-card Payment Instructions no longer appear in PaymentInstructionServiceImpl.getAll() results, leaving ACH and legacy methods and existing ordering untouched.

Downstream services. Credit-card booking into QuickBooks had been triggered off an intermediate lifecycle event; when the upstream execution flow stopped emitting it, I retargeted the downstream trigger to the first event in the next lifecycle stage instead, without touching ACH's separate event path. Since events can arrive more than once, the booking path checks for an already-created QuickBooks transaction before creating one; and since a retry or crash can land mid-write, recovery leaves the bookkeeping state somewhere a later pass can reconcile rather than somewhere ambiguous. A downstream bookkeeping failure is also never treated as a payment failure — the credit-card funds already moved on the execution side, and there is no single transaction spanning both services to roll back.

Payment-state consistency across asynchronous paths. A production issue exposed a class of defect that no single path's tests would show. After a payment lifecycle-state change, the same payment outcome was not being interpreted consistently across asynchronous, cross-service processing paths. In a mixed-outcome batch, where not every Payment Instruction resolves the same way, that inconsistency showed up as incorrect downstream notification behavior.

It was hard to find because the failure only became visible in the interaction across service and asynchronous boundaries, even though the individual paths could look correct in isolation. The existing tests passed because they did not carry the production-critical context: a mixed outcome moving through the full asynchronous path.

I owned the investigation and the remediation end to end, tracing where the same payment outcome was interpreted inconsistently across payment execution and lifecycle state, asynchronous event-driven processing, and the downstream notification and aggregation behavior that consumes those states. The fix made the interpretation of the outcome consistent across those paths while keeping existing notification, aggregation, and backward-compatible behavior intact.

To close the gap that let it through, I added targeted regression coverage for the failure mode, including mixed-outcome batches, and validated the remediation with production-like end-to-end validation and runtime observability, so that confidence rested on actual flow behavior rather than isolated unit behavior alone. The fix is deployed and running in production.

This is separate from the downstream booking change above, but it sits in the same lifecycle and state-consistency domain: a lifecycle change can be locally correct and still change how downstream consumers interpret payment state.

Key Decisions

• Reject before persistence, not after. The CC+RTP check runs at the validation boundary, so invalid batches never enter persistence or dispatch instead of being cleaned up downstream. The shared check is extracted into PaymentInstructionUtil so the two batch-creation paths can't drift apart.
• Decide retryability in the backend, expose it as a contract. Retry eligibility is a business rule about which failed payments may be rescheduled, so it belongs in the core domain layer as one canonical failure-to-retryability mapping, not re-derived per client from status codes. The batch-payment polling API publishes the result as typed failureDetails metadata, with persisted failure type taking precedence over a legacy payment-status fallback so the answer is defined for old and new records alike.
• Indexed, sharded discovery over a full-table scan. A sparse GSI indexes only PENDING candidates; sharding across partition keys avoids a hot status partition; a per-shard timestamp sort key gives age-ordered paging. Bounded work per run and starting-shard rotation fall out of this decision — they keep each sweep's cost predictable and its load spread as the backlog grows.
• Treat the GSI as discovery data; re-read authoritative state before mutating. This is a correctness decision, not a performance one. The index carries enough projected data to screen out most candidates cheaply, but the job still takes exactly one fresh persisted-state read immediately before abort, so it never mutates on an index entry that has gone stale; already-resolved candidates are a no-op.
• Global kill switch plus staged company-level enablement, not all-or-nothing. A scheduled job that mutates payment state across every company needs both an instant stop and gradual exposure, with per-candidate failure isolation so one bad record can't halt the sweep for everyone.
• Treat payment execution and downstream bookkeeping as independently-failing systems, not one atomic unit. No transaction spans the credit-card execution service and the downstream QuickBooks-booking path, so idempotent booking, a reconcilable retry/crash path, and never rolling back on a bookkeeping failure all have to hold independently — consistency is assembled from those three properties, not from a shared transaction.
• Validate the flow, not just the unit. The defect lived in how state was interpreted across paths, so the remediation was verified with regression coverage for mixed-outcome batches plus production-like end-to-end validation and runtime observability, not by unit tests of any single path.

Trade-offs

Keeping one fresh base-table read inside abort() costs a DynamoDB read per aborted candidate on top of the GSI query — more than trusting the index outright — but a stale-index abort on a live Payment Instruction is a worse bug to carry than that read is to pay for, and removing the redundant earlier fetch paid most of the cost back. A sharded GSI trades some query complexity, fan-out across shards instead of one partition, for avoiding a hot-partition bottleneck as in-flight volume grows. Company-level feature-flagging adds the complexity of two live behaviors to reason about during rollout, in exchange for gradual exposure and an instant stop.

Impact

Implemented behavior. The batch validator rejects unsupported Credit Card + RTP combinations before persistence or downstream dispatch; the pending-approval abort path is payment-state-aware; downstream QuickBooks booking for credit-card payments now triggers off an earlier, idempotent lifecycle event; the batch-payment polling API exposes a canonical failure taxonomy and a typed retry-eligibility flag for failed credit-card payments, with persisted failure type taking precedence over a legacy payment-status fallback and the two fields kept internally consistent; the Payment Instruction expiration TTL is runtime-refreshable; the scheduled sweep discovers and aborts stale in-flight Payment Instructions through the sharded GSI, with an authoritative-state re-read before each abort; and failed credit-card Payment Instructions are filtered out of Payments UI/API reads.

Validation. The validation and abort-handling changes are covered by a focused test suite plus the full module regression suite. The failure-taxonomy and retry-eligibility mapping is covered by a regression suite over the full policy matrix, including type precedence, legacy fallback, and unknown or incomplete failure data. The TTL configuration has a paired configuration change and unit coverage. The scheduled job and GSI are covered by a targeted automated test suite spanning scheduler guards, distributed-job behavior, GSI query and multi-shard aggregation semantics, and concurrent-abort and failure-isolation scenarios, and the GSI was provisioned and query-verified end to end at a representative scale. Representative-scale verification is not production-scale evidence. The read-path filter is validated against the controller contract. The cross-service payment-state fix is covered by targeted regression tests, including mixed-outcome batches, and by production-like end-to-end validation; it is deployed and running in production.

Safeguards. The sweep is gated behind a global kill switch and company-level feature-flag gating, with per-candidate failure isolation so one bad Payment Instruction cannot halt a run.

Ownership boundary. This is payment-correctness hardening, backend retry-policy and failure-taxonomy ownership for payment recovery, lifecycle automation, and DynamoDB access-pattern design inside an existing platform, not ownership of the payment-batch architecture, the Credit Card or RTP systems, the Credit Card Online Execution flow, downstream persistence and dispatch, the base Payment Instruction table, or the reschedule UX. The retry-eligibility work is a payments-domain business rule about which failed payments may be rescheduled, not message-delivery retry or at-least-once processing. No transaction-volume, latency, revenue, failed-payment-reduction, recovery-rate, incident-severity, or incident-reduction figures were measured or are claimed.

Lessons

Discovery data is not authoritative state. It's easy to write a scan-then-act job that's correct in the common case and wrong exactly when timing gets unlucky; the fix — one fresh persisted-state read immediately before the mutation — is cheap once you're looking for it.

Performance and correctness weren't in tension once each read had a defined purpose: the index can carry enough projected data to make most reads unnecessary, as long as exactly one fresh read stays right before the state change that can't tolerate staleness. The GSI work was also where I built hands-on DynamoDB depth — sparse keys, shard distribution, projection strategy, query-time trade-offs — reasoning about the database directly rather than through an abstraction.

The original ticket asked for a scheduled full-table scan. Getting to the sharded-GSI, revalidate-before-abort design meant questioning that starting point and reasoning about how the index would actually be queried and where staleness could enter the wider payment-state system — not just closing the ticket as written.

"The payment succeeded" and "the books are updated" are two separate facts owned by two separate services, not one fact that happens to have two write sites — the downstream booking fix only got simple once I stopped assuming otherwise and let the consuming side, not the transaction boundary, absorb the duplicate events and partial failures.

Local correctness was not enough. The behavior depended on a state transition and on how downstream consumers interpreted it across asynchronous boundaries, so tests that passed path by path gave false confidence. What closed the gap was a test that carried the real context, plus runtime evidence that the flow behaved correctly end to end.