Concurrently Process a Single Kafka Partition | Object Partners

▤ Summary

The next step was to write a simple consumer program that would deserialize the Avro and convert the field we needed to match on. Four hours and over 300 million records later, I found the data we were looking for (it was dropped by an intermediate process in the next step of the pipeline). Why not split the partition into segments and let multiple consumers scan through their own offset range? This isn’t something I’d get too carried away with and create 1000 groups to split up a partition as that might overwhelm Kafka, though I don’t have any concrete facts to back that up. The KafkaConsumer documentation does state that any number of groups can subscribe to a topic, so maybe it’s not that big of a concern.

▦ Linked Companies

Object Partners

$10M to $25M

GitHub, Inc.

$1M to $5M