2019-03-15 14:37:31,466 INFO [kafka.coordinator] Successfully joined group <redacted-consumer-group> with generation 9 
2019-03-15 14:37:31,467 INFO [kafka.consumer.subscription_state] Updated partition assignment: [TopicPartition(topic='<redacted-topic-name>', partition=3)] 
2019-03-15 14:37:31,471 INFO [kafka.coordinator.consumer] Setting newly assigned partitions {TopicPartition(topic='<redacted-topic-name>', partition=3)} for group <redacted-consumer-group>
2019-03-15 14:37:31,496 INFO [kafka.conn] <BrokerConnection node_id=2 host=<redacted-broker-2>:9092 <connecting> [IPv4 ('10.100.0.140', 9092)]>: connecting to <redacted-broker-2>:9092 [('10.100.0.140', 9092) IPv4] 
2019-03-15 14:37:31,557 INFO [kafka.conn] <BrokerConnection node_id=2 host=<redacted-broker-2>:9092 <connecting> [IPv4 ('10.100.0.140', 9092)]>: Connection complete. 
2019-03-15 14:37:32,138 INFO [kafka.conn] <BrokerConnection node_id=0 host=<redacted-broker-0>:9092 <connecting> [IPv4 ('10.100.0.141', 9092)]>: connecting to <redacted-broker-0>:9092 [('10.100.0.141', 9092) IPv4] 
2019-03-15 14:37:32,159 INFO [kafka.conn] <BrokerConnection node_id=0 host=<redacted-broker-0>:9092 <connecting> [IPv4 ('10.100.0.141', 9092)]>: Connection complete. 
2019-03-15 14:37:32,160 INFO [kafka.conn] <BrokerConnection node_id=bootstrap host=10.100.0.140:9092 <connected> [IPv4 ('10.100.0.140', 9092)]>: Closing connection.  
2019-03-15 22:37:32,681 INFO [kafka.conn] <BrokerConnection node_id=1 host=<redacted-broker-1>:9092 <connecting> [IPv4 ('10.100.0.143', 9092)]>: connecting to <redacted-broker-1>:9092 [('10.100.0.143', 9092) IPv4] 
2019-03-15 22:37:32,682 INFO [kafka.conn] <BrokerConnection node_id=1 host=<redacted-broker-1>:9092 <connecting> [IPv4 ('10.100.0.143', 9092)]>: Connection complete. 
2019-03-15 22:51:32,689 INFO [kafka.client] Closing idle connection 1, last active 540001 ms ago 
2019-03-15 22:51:32,690 INFO [kafka.conn] <BrokerConnection node_id=1 host=<redacted-broker-1>:9092 <connected> [IPv4 ('10.100.0.143', 9092)]>: Closing connection.  
2019-03-15 23:12:57,527 WARNING [kafka.client] Protocol out of sync on <kafka.conn.BrokerConnection object at 0x7f904d004978>, closing 
2019-03-15 23:12:57,528 INFO [kafka.conn] <BrokerConnection node_id=2 host=<redacted-broker-2>:9092 <connected> [IPv4 ('10.100.0.140', 9092)]>: Closing connection. KafkaConnectionError: Socket EVENT_READ without in-flight-requests 
2019-03-15 23:12:57,528 WARNING [kafka.client] Node 2 connection failed -- refreshing metadata 
2019-03-15 23:12:57,546 INFO [kafka.conn] <BrokerConnection node_id=0 host=<redacted-broker-0>:9092 <connecting> [IPv4 ('10.100.0.141', 9092)]>: connecting to <redacted-broker-0>:9092 [('10.100.0.141', 9092) IPv4] 
2019-03-15 23:12:57,548 INFO [kafka.conn] <BrokerConnection node_id=0 host=<redacted-broker-0>:9092 <connecting> [IPv4 ('10.100.0.141', 9092)]>: Connection complete. 
2019-03-15 23:18:02,568 WARNING [kafka.client] <BrokerConnection node_id=2 host=<redacted-broker-2>:9092 <disconnected> [IPv4 ('10.100.0.140', 9092)]> timed out after 305000 ms. Closing connection. 
2019-03-15 23:18:02,569 WARNING [kafka.conn] <BrokerConnection node_id=2 host=<redacted-broker-2>:9092 <disconnected> [IPv4 ('10.100.0.140', 9092)]>: Duplicate close() with error: [Error 7] RequestTimedOutError: Request timed out after 305000 ms
2019-03-15 23:18:02,569 WARNING [kafka.client] <BrokerConnection node_id=2 host=<redacted-broker-2>:9092 <disconnected> [IPv4 ('10.100.0.140', 9092)]> timed out after 305000 ms. Closing connection. 
2019-03-15 23:18:02,569 WARNING [kafka.conn] <BrokerConnection node_id=2 host=<redacted-broker-2>:9092 <disconnected> [IPv4 ('10.100.0.140', 9092)]>: Duplicate close() with error: [Error 7] RequestTimedOutError: Request timed out after 305000 ms 
2019-03-15 23:18:03,071 WARNING [kafka.client] <BrokerConnection node_id=2 host=<redacted-broker-2>:9092 <disconnected> [IPv4 ('10.100.0.140', 9092)]> timed out after 305000 ms. Closing connection. 
2019-03-15 23:18:03,071 WARNING [kafka.conn] <BrokerConnection node_id=2 host=<redacted-broker-2>:9092 <disconnected> [IPv4 ('10.100.0.140', 9092)]>: Duplicate close() with error: [Error 7] RequestTimedOutError: Request timed out after 305000 ms 
2019-03-15 23:18:03,072 WARNING [kafka.client] <BrokerConnection node_id=2 host=<redacted-broker-2>:9092 <disconnected> [IPv4 ('10.100.0.140', 9092)]> timed out after 305000 ms. Closing connection. 
2019-03-15 23:18:03,072 WARNING [kafka.conn] <BrokerConnection node_id=2 host=<redacted-broker-2>:9092 <disconnected> [IPv4 ('10.100.0.140', 9092)]>: Duplicate close() with error: [Error 7] RequestTimedOutError: Request timed out after 305000 ms 
2019-03-15 23:18:03,574 WARNING [kafka.client] <BrokerConnection node_id=2 host=<redacted-broker-2>:9092 <disconnected> [IPv4 ('10.100.0.140', 9092)]> timed out after 305000 ms. Closing connection. 
2019-03-15 23:18:03,575 WARNING [kafka.conn] <BrokerConnection node_id=2 host=<redacted-broker-2>:9092 <disconnected> [IPv4 ('10.100.0.140', 9092)]>: Duplicate close() with error: [Error 7] RequestTimedOutError: Request timed out after 305000 ms

... these 2 repeating for 2 days until we killed the worker manually