Kafka Consumer Lag Monitoring
Summary
There are many things that can be causing the Kafka Consumer Lag, including: • big jump in traffic resulting in producing way more Kafka messages • various software bugs and issues resulting in slow processing Of course, the above list is just an example, but let’s look at one of the mentioned causes a bit deeper, the one about issues with the pipeline elements. Similar to the above point, you need to find and fix the issues in your code if they are the ones that are responsible for the Kafka Consumer lag. As developers know all that, I know, but unit tests, pair programming, and code reviews really help find issues and correct them. Luckily you can mitigate such issues, for example, by using the Parallel Consumer or trying to repartition the Kafka topic even more to isolate the noisy data source. Under the hood, the open source Sematext agent collects a few Kafka metrics with various offsets from which these rates are computed.
