Owning a Kafka Routing Layer on a Greenfield Banking Platform

Problem

UMTB's banking platform needed a single, consistent way for backend services to consume and route Kafka events, one that handled failures predictably instead of leaving each service to reinvent retry and dead-letter logic on its own. Without it, every team was on track to build its own ad-hoc consumer logic, with inconsistent retry behavior and no shared visibility into failures.

Context

The platform was greenfield: a new banking system with no existing conventions to inherit. I was one of the engineers building the platform's shared infrastructure layer, working alongside a principal architect who set overall technical direction but wasn't embedded in the day-to-day design of every component.

Constraints

• Regulated banking environment: message loss or silent failure was not acceptable.
• No existing internal precedent for how Kafka consumption should be structured on this platform.
• Multiple downstream services with different failure-handling needs, all needing to route through the same layer.
• No dedicated senior engineer assigned to guide the router's design: the research and the decisions were mine to make.

My Role

I owned the Kafka router end-to-end: architecture, implementation, and rollout. I researched the design independently, made the core structural decisions myself, and kept the principal architect and tech lead updated as the design took shape, not as a single after-the-fact approval, but as an ongoing feedback loop that let me catch gaps early and refine the approach against the platform's broader architecture.

Architecture / Approach

The router sits between raw Kafka topics and the services that consume them. It classifies failures by error type and routes accordingly: transient failures go through retry with backoff, and failures that exceed the retry budget are routed to a dead-letter queue rather than blocking the consumer or silently dropping the message. Consumers get a consistent contract regardless of which topic they're reading from, so failure handling doesn't have to be reinvented per service.

Key Decisions

• Error-type-based routing over one-size-fits-all retry. Different failure classes (transient vs. permanent) needed different handling, so the router classifies before deciding whether to retry or dead-letter.
• Centralizing the router rather than a shared library per service. A single owned component meant one place to fix bugs and one place to observe failure patterns across the platform, instead of N independent copies drifting apart.
• Continuous alignment over a single design review. Rather than presenting a finished design once, I surfaced the approach incrementally to the principal architect and tech lead, which caught misalignments while they were still cheap to fix.

Trade-offs

Owning the design independently meant moving faster than a committee-reviewed process would have allowed, but it also meant the burden of catching my own blind spots fell on me, mitigated by the continuous-alignment habit above rather than a formal review gate. Centralizing routing in one component also creates a shared dependency: a bug in the router has platform-wide blast radius, which is why the surrounding test coverage (Testcontainers-based integration tests, including a Kafka test-infrastructure issue I diagnosed and fixed independently) mattered as much as the routing logic itself.

Impact

The router now handles millions of monthly events across the platform with automated error-type-based routing and dead-letter-queue management. It became one of the components new engineers were walked through during onboarding, and the pattern it established (classify, retry with backoff, dead-letter on exhaustion) became the platform's default answer to "how do we consume Kafka safely here."

Lessons

Architectural ownership without a senior engineer in the loop doesn't have to mean isolation. Keeping stakeholders continuously informed, rather than deferring the conversation to a single review, turned out to be what let me move independently and catch issues early, instead of having to choose between speed and safety.