Stop picking Kafka when a plain RabbitMQ queue will do

6 min read

Stop choosing Kafka when a simple RabbitMQ queue does the job. I have watched three UK engineering teams in the last 18 months burn four to six weeks wiring up Kafka, Zookeeper or KRaft, schema registries, and a second observability stack just to move a few thousand messages a day between two services. One of them was a fintech in Shoreditch moving webhook events. Another was a team inside HMRC's supplier ecosystem processing batch validations overnight. A third was a Monzo-style challenger bank copycat trying to replicate the event bus from a conference talk. None of them needed ordered, replayable, partitioned log semantics. They needed a queue.

Kafka is brilliant. It is also overkill for roughly 70% of the workloads I see advertised in London job specs. When your throughput is measured in hundreds of messages per second, not hundreds of thousands, and you do not need to replay three days of traffic into a new consumer, you are paying a complexity tax for capability you will never use. RabbitMQ, or even AWS SQS behind a thin wrapper, will get you to production faster, with fewer on-call pages, and with a senior engineer who actually understands the system instead of six contractors copy-pasting Confluent blog posts.

What Kafka actually gives you, and when it matters

Kafka is a distributed, partitioned, append-only log. That sentence matters. It is not a queue in the traditional sense. Messages are not removed when a consumer reads them; they are retained for a configurable window and consumers track their own offset. This unlocks three real capabilities: replay, fan-out to many independent consumer groups, and very high throughput via partitioning.

Replay matters if you are rebuilding a data warehouse from upstream events, like the analytics pipelines inside BBC iPlayer or the fraud detection feeds at Revolut. Fan-out matters if ten different teams each need their own copy of the same event stream. Partitioned throughput matters if you are processing six-figure messages per second during a Black Friday peak at a retailer like Ocado or a Tesco bank holiday.

If none of those three things describe your workload, you are renting a bulldozer to plant tulips. RabbitMQ gives you per-message acknowledgement, dead letter exchanges, delayed messages, and a management UI you can hand to a junior without a two-week training plan. That is the right tool for roughly nine out of ten UK backend roles I have looked at in the past year.

The hidden cost of running Kafka in the UK

Kafka is operationally heavy. You need at least three broker nodes for quorum, which in AWS us-east-1 or eu-west-2 means roughly 1,500 to 2,500 GBP a month just for the brokers before you add storage, cross-AZ transfer, and the Schema Registry or Karapace sidecar. Add a platform engineer at 90k to 120k GBP a year to keep it healthy, plus a second on-call rota because Kafka incidents are not for the faint-hearted.

RabbitMQ in comparison runs comfortably on a single small EC2 instance or an RDS-style managed offering like CloudAMQP, starting around 80 to 200 GBP a month. The mental model is one your average mid-level developer already understands. The on-call burden is roughly a third. That is not a rounding error when you are a 40-person Series B startup in King's Cross trying to hit runway.

How UK teams end up picking the wrong tool

Three patterns I see repeatedly in CVs and interview prep conversations. First, conference-driven architecture. Someone watched a Monzo engineering blog post about their debit card event stream and decided their internal admin tool needs the same rigour. It does not. Monzo processes millions of authorisation events a day; your staff onboarding flow processes twelve.

Second, CV-driven architecture. Engineers pick Kafka because they want Kafka on their CV before their next job hop. That is a rational personal decision and a terrible engineering decision. A London platform engineer with deep Kafka experience can command 110k to 140k GBP; one with deep RabbitMQ plus SQS plus Postgres LISTEN/NOTIFY experience commands the same band and is far more useful at most UK employers. Choose the tool that matches the problem, not the one that flatters your CV.

Third, recruiter-led architecture. Job specs at Revolut, Wise, and a handful of fintechs in Manchester list Kafka because the hiring manager copy-pasted last year's spec. Candidates then design their side projects around Kafka to match. We end up with a generation of UK engineers who think "event-driven" automatically means "Kafka," when 80% of the time the same outcome is achievable with RabbitMQ topics, SNS, or even Postgres outbox tables read by a worker.

Decision framework you can apply this week

Ask four questions before you write a single line of Terraform. One: do I need to replay events from more than one hour ago? If no, a queue is fine. Two: do I need more than five independent consumer groups reading the same stream? If no, a queue is fine. Three: is my peak throughput above 10,000 messages per second sustained? If no, a queue is fine. Four: is ordering across all messages critical, or ordering within a partition key? If the latter, RabbitMQ with consistent hashing on a single queue handles it.

If you answered yes to two or more, then yes, start a Kafka POC. Otherwise, default to RabbitMQ or SQS and spend the saved engineering weeks on something that actually moves revenue, like the checkout latency fix your PM has been begging for since Q1.

There is a real cost to be honest about: switching from RabbitMQ to Kafka later is painful but well-understood. Switching from Kafka to RabbitMQ after you have built ten microservices around topics and consumer groups is a six-month migration that nobody wants to staff. Start small, upgrade only when the data justifies it.

What this means for your next UK role

If you are interviewing at a UK scale-up or enterprise and the team is proudly running Kafka for low-volume workloads, that is a yellow flag about their engineering judgement and a red flag about your upcoming on-call rota. Ask in the interview what their daily message volume is, how many consumer groups they run, and whether they have ever had to replay more than a day's worth of traffic. If the answers are "around 2,000 a day," "one," and "no," you are about to inherit complexity for no reason.

Is it worth the move?

Yes, take the role if the team is honest about it and migrating down. No, reconsider if they double down on Kafka as a religion. A strong platform engineer in London can earn 95k to 135k GBP with a couple of years of RabbitMQ, SQS, and Postgres experience; the same person with Kafka plus Kubernetes plus Terraform earns 110k to 150k. The premium is real but small, and the on-call pain is much larger. Pick the team that picks the boring tool on purpose.

FAQ

Is Kafka faster than RabbitMQ?

Per broker, Kafka can sustain much higher throughput because it batches and appends to disk sequentially. For the workloads most UK teams run, neither is the bottleneck. RabbitMQ comfortably handles tens of thousands of messages per second on modest hardware; Kafka's advantage only kicks in at sustained high fan-out or long retention.

Will I regret using RabbitMQ if we grow?

Unlikely. The migration path from RabbitMQ to Kafka, or from RabbitMQ to SQS plus Kafka, is well-documented and incremental. Most teams I have seen grow past RabbitMQ add Kafka for a specific analytics or replay use case while keeping transactional queues on RabbitMQ or SQS. You do not have to choose once for the lifetime of the company.

Should I still learn Kafka as a UK engineer?

Yes, learn it. Understand the log abstraction, consumer groups, and exactly-once semantics. Do not, however, reach for it by default. The most valuable UK engineers I have worked with at places like the BBC, Monzo, and various HMRC suppliers know when not to use the trendy tool. That judgement is what separates a 75k mid-level role from a 120k staff role, far more than any single piece of tech on your CV.