You set up Kafka, launched your first producer and consumer, and messages started flowing. A week later, you discover that consumers are lagging by millions of messages, and after a broker restart, some data is missing. Sound familiar? These are typical pitfalls that almost everyone encounters when starting with stream processing. In this article, we'll explore how Kafka works internally, what pitfalls await on the path from a local experiment to production, and how to choose a learning path so you don't lose months.
Why Kafka is not just a message queue
Many come to Kafka with experience in RabbitMQ or ActiveMQ and think it's just a faster queue. This is a dangerous misconception. Kafka is a distributed log where messages are not deleted after reading but are stored for a configured retention period. Consumers manage their own offsets, can re-read data, and multiple independent applications can read the same topic without interfering with each other.
This architecture enables real-time stream processing but requires understanding how partitions, replication, and consumer groups work. If you treat Kafka as a queue, you'll quickly run into balancing and message loss issues.
Key difference: in a classic queue, a message disappears after acknowledgment; in Kafka, it lives until the retention period expires. This changes everything: from reprocessing strategy to application design.
What you actually need to master to work with Kafka
The list of technologies around Kafka is huge: Kafka Streams, KSQL, Kafka Connect, Schema Registry, monitoring, security. But not all are equally important at the start. Here's what really affects your ability to build reliable streaming systems:
- Pub/sub basics and broker architecture — without this, you can't diagnose performance and fault tolerance issues.
- Producer and Consumer APIs — in Java and Python. Ability to configure acks, idempotence, manage offsets manually.
- Kafka Streams — a stream processing library embedded in the application. Allows joins, aggregations, windowed operations without external clusters.
- KSQL — a SQL-like interface for stream processing. Convenient for rapid prototyping but doesn't replace Streams in complex cases.
- Kafka Connect — integration with external systems (databases, storage) via ready-made connectors.
- Schema Registry — managing data schema evolution. Critical for producer-consumer compatibility.
Don't try to learn everything at once. Start with the core: broker, producers, consumers, partitions. Then add Streams or KSQL for processing. Connect and Schema Registry are the next step when integration with external systems arises.
Real-world case: how to avoid losing messages on failure
Imagine: an online store processes orders via Kafka. A producer sends an "order created" event, a consumer deducts the item from inventory. Everything works until a broker fails. After restart, some orders are lost, and some are processed twice.
Let's analyze what went wrong and how to fix it.
Step 1. Producer configuration. By default, acks=1 — the producer waits for acknowledgment only from the partition leader. If the leader fails before replication, the message is lost. Solution: set acks=all and enable idempotence (enable.idempotence=true). This ensures the message is written to all in-sync replicas and won't be duplicated on retry.
Step 2. Consumer configuration. If the consumer commits the offset before processing the message, it will skip the message on failure. You need to commit after successful processing. But then reprocessing is possible — that's at-least-once. To avoid duplicates, you need idempotent processing on the consumer side (e.g., deduplication by order ID).
Step 3. Exactly-once semantics. Kafka Streams and transactional producer allow achieving exactly-once processing within Kafka. But this works only if both reading and writing go to Kafka. If you write to an external DB, you'll have to implement idempotence yourself or use the outbox pattern.
What happens if you ignore this? Lost orders mean direct financial losses and unhappy customers. Duplicates mean extra inventory deductions and manual reconciliation. Recovering data after the fact can take days.
Where Kafka breaks: failure modes you need to know by sight
Even a properly configured system can fail. Here are several scenarios that occur most often.
1. Partition skew
What happens: if the partitioning key is chosen poorly (e.g., a field with few unique values), all messages go to one partition. The rest sit idle, performance drops, and consumers in other partitions are underutilized.
How to spot early: monitor partition size metrics and write speed to each partition. If one partition grows significantly faster than others — that's a problem.
What to do: choose the right key (e.g., user ID with high cardinality) or use round-robin without a key.
2. Consumer lag grows like a snowball
What happens: the consumer can't keep up with processing messages. The lag (difference between the last written and read message) increases. At some point, retention will delete old messages, and data will be lost irrevocably.
How to spot: set up lag monitoring (e.g., via Prometheus and Grafana). If lag is steadily growing — it's time to scale consumers or optimize processing.
What to do: add partitions (careful: this changes keys), increase the number of consumers in the group, optimize processing code.
3. Incompatible data schema change
What happens: the producer starts sending data in a new format, and consumers aren't ready. Without Schema Registry, this leads to deserialization errors and consumer crashes.
How to spot: use Schema Registry with compatibility checking. Set a compatibility policy (e.g., BACKWARD) so new schemas don't break old consumers.
What to do: if it already happened — roll back the schema or update consumers. In the future, use Schema Registry and test changes on staging.
4. Broker loss and topic unavailability
What happens: when a broker that was the partition leader fails, a leader re-election occurs. If replication didn't complete, messages may be lost. If min.insync.replicas is not configured, a producer with acks=all may get an error on write.
How to spot: monitor cluster state, number of in-sync replicas (ISR). Set alerts for ISR shrinkage.
What to do: properly configure default.replication.factor (usually 3) and min.insync.replicas (usually 2). Regularly check cluster health.
Learning paths: self-study, courses, practice
Once you realize Kafka is serious, the question arises: how to learn? There are three main paths, each with its own trade-offs.
| Path | Time | Money | Outcome |
|---|---|---|---|
| Self-study (docs, articles, videos) | 3–6 months to confident level | Minimal (only hardware or cloud costs) | Deep understanding, but possible gaps and mistakes no one corrects |
| Online course | 2–4 months | From a few hundred to several thousand dollars | Structured knowledge, feedback, analysis of typical mistakes |
| On-the-job practice (internship, project) | 6–12 months | Free (you're paid to learn) | Real experience, but depends on tasks and mentor |
Self-study is cheaper but requires discipline. The main risk is you won't know you did something wrong until you hit a production issue. For example, you set acks=1 and think everything is reliable until you lose data.
Courses provide structure and feedback. A good Kafka course includes not only theory but also practical assignments with error analysis. However, not all courses are equal: some just retell documentation, others provide real scenarios. Look for ones with hands-on work on a real cluster and analysis of typical mistakes.
On-the-job practice is the fastest path if you're lucky with the project. But a novice is rarely trusted with production cluster administration right away. Usually you start by writing producers and consumers under supervision.
Non-obvious insight: the best result comes from a combination: a course for structure and practice on a real project (even a pet project) for reinforcement. Without practice, knowledge evaporates in a month.
Who is suited for a career in stream processing, and who isn't
Kafka is not for everyone. Here are signs it might interest you:
- You enjoy understanding distributed systems, you're ready to read logs and metrics.
- You want to work with real-time data, not just batch processing.
- You're interested in infrastructure and reliability, you're not afraid to debug complex failures.
Who shouldn't go: if you prefer quick results without deep immersion, if debugging network issues and reading documentation annoys you, if you want only to write code and not touch infrastructure — stream processing may seem boring and complex.
Also honestly assess your prerequisites: solid Java or Python skills, understanding of networks (TCP/IP, TLS), databases. Without this, the start will be very tough.
Bottom line
Apache Kafka and stream processing is a field where specialists who can build reliable real-time systems are valued. But the path from novice to confident engineer is thorny: many pitfalls, mistakes are costly, and information is abundant but scattered. Self-study can take months of trial and error.
If you want to shorten this path and gain structured knowledge with analysis of real scenarios, check out the course Apache Kafka and Stream Processing. It covers key topics — from basics to exactly-once and multi-DC replication — and provides practical skills needed for a Kafka Administrator role. You can start on the course page.