High CPU usage during bootstrapping if broker accepts tcp connection but does not reply

[JochenABC]

Problem

When running this command in terminal 1:

nc -l -p 9092

... and this python script in terminal 2:

import kafka

if __name__ == '__main__':
    kafka.client.KafkaClient(bootstrap_servers='127.0.0.1:9092')

... it will draw 100% cpu for 40 seconds (connection timeout reached)

Expected behaviour:

Script draws a reasonable amount of CPU

Actual behaviour:

Script draws as much cpu as it gets.

Additional information

When enabling debug output for kafka module I get:

Bootstrapping cluster metadata from [('127.0.0.1', 9092, <AddressFamily.AF_INET: 2>)]
Attempting to bootstrap via node at 127.0.0.1:9092
<BrokerConnection node_id=bootstrap host=127.0.0.1/127.0.0.1 port=9092>: creating new socket
<BrokerConnection node_id=bootstrap host=127.0.0.1/127.0.0.1 port=9092>: setting socket option (6, 1, 1)
<BrokerConnection node_id=bootstrap host=127.0.0.1/127.0.0.1 port=9092>: connecting to 127.0.0.1:9092
<BrokerConnection node_id=bootstrap host=127.0.0.1/127.0.0.1 port=9092>: established TCP connection
<BrokerConnection node_id=bootstrap host=127.0.0.1/127.0.0.1 port=9092>: Connection complete.
Node bootstrap connected
<BrokerConnection node_id=bootstrap host=127.0.0.1/127.0.0.1 port=9092> Request 1: MetadataRequest_v0(topics=[])

---> hangs until connection timeout is reached

Why is this case relevant?

We run kafka as a statefulset in our kubernetes cluster. If it dies for some reason the kubernetes service will still be there, so tcp connections to service:9092 are still established. However there is no reply since kafka is dead. So all of our client processes will draw 100% cpu.

Version used

kafka-python module: Latest version from master (git+https://github.com/dpkp/kafka-python@3ff3d75004f94fd55fa089297d3e2376e33ccda7)

[tvoinarovskyi]

Seems legit. We do:

            future = bootstrap.send(metadata_request)
            while not future.is_done:
                bootstrap.recv()

Just went through the code, but it should be this. We don't backoff on blocking read exceptions here