Consumer blocks indefinitely in `poll` when assigned partitions and brokers are unavailable

[nihb]

Firstly: thanks (!) for writing this library.
I'm not sure if this is user error, a bug or a feature, but the following blocks indefinitely on the poll() command (no broker needed) in python2 and python3 on 1.3.5, 1.4.3, 1.4.5 as well as on 1.4.6.dev:

from kafka import KafkaConsumer, TopicPartition
# N.B. "yyy" does not resolve to any hostname on my network
consumer = KafkaConsumer(bootstrap_servers='yyy:9092', api_version='0.10.1')
consumer.assign([TopicPartition('zzz', 0)])
r_poll = consumer.poll(timeout_ms=1000)

I've also tested it by replacing "yyy" above with local hosts whose dns entries are defined but which have been turned off with the same results (blocks indefinitely). Adding group_id to KafkaConsumer also doesn't seem to help.

P.S. Removing the api_version='0.10.1' allows the KafkaConsumer() call to return immediately with no brokers available in 1.4.6.dev, blocks indefinitely on KafkaConsumer() in 1.4.5, returns immediately in 1.3.5.

[dpkp]

This is the same issue as #1346 / KAFKA-6397

[nihb]

Hi dpkp,
Thanks (!) for the quick response. So, if I understand correctly, it sounds like this is a fundamental issue with api_version=0.10.1.
I'll continue with my ugly workaround for the moment (using SIGALRM to break out of poll(), etc.), then try to get off this api altogether.
Cheers.

[dpkp]

No, this is not specific to 0.10.1 or any particular API version. It is a "bug" in the client code that does not pass the poll timeout into all internal method calls. This has also been a bug in the Java client. If/when it gets fixed, your consumer.poll call would return empty after the 1 sec timeout you are passing.

…

On Mon, Mar 25, 2019, 9:51 PM nihb ***@***.***> wrote: Hi dpkp, Thanks (!) for the quick response. So, if I understand correctly, it sounds like this is a fundamental issue with api_version=0.10.1. I'll continue with my ugly workaround for the moment (using SIGALRM to break out of poll(), etc.), then try to get off this api altogether. Cheers. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1764 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAzetKZet387GBFKNc9CQnRGndPiJHvPks5vaadIgaJpZM4cKfPP> .

[nihb]

Hi dpkp,
Thanks (!) for the clarification. So in the example above, the use of api_version=0.10.1 seems to have poked this bug due to the call to KafkaConsumer() returning immediately followed by a call to poll() where things block, whereas when the api_version isn't forced to be 0.10.1 the KafkaConsumer() function raises an error before poll() gets called complaining that no brokers are available, the program exits and we don't get stuck with poll() blocking indefinitely.
Since I'd like to avoid poking this problem as much as possible, it sounds like KafkaConsumer() here might be my friend. So a quick question: if api_version=0.10.1 does the initial connection to the brokers occur in poll(), whereas in later APIs KafkaConsumer() already tries to start up some kind of dialogue (e.g. metadata) with them?

[dpkp]

Without specifying an api_version we will do some initial magic "poking" and try to guess the broker version. The error you are seeing is from this version check, which is synchronous.

…

On Tue, Mar 26, 2019, 7:58 PM nihb ***@***.***> wrote: Hi dpkp, Thanks (!) for the clarification. So in the example above, the use of api_version=0.10.1 seems to have poked this bug due to the call to KafkaConsumer() returning immediately followed by a call to poll() where things block, whereas when the api_version isn't forced to be 0.10.1 the KafkaConsumer() function raises an error before poll() gets called complaining that no brokers are available, the program exits and we don't get stuck with poll() blocking indefinitely. Since I'd like to avoid poking this problem as much as possible, it sounds like KafkaConsumer() here might be my friend. So a quick question: if api_version=0.10.1 does the initial connection to the brokers occur in poll(), whereas in later APIs KafkaConsumer() already tries to start up some kind of dialogue (e.g. metadata) with them? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1764 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAzetDUq7SIF8VV6dcgdx1Ib59fx3HHaks5vat5qgaJpZM4cKfPP> .

[jeffwidman]

Closing as dupe of #1346