produce loss data

[caseware66]

Description

When we try to write 10 messages in a Kafka cluster, we do not find the expected account as a result. We have 1 OR 2 OR 3...messages.
Do you have an idea ?

How to reproduce

import confluent_kafka
import time
import sys

producer_timings = {}
consumer_timings = {}

def error_cb(kafka_error):
print("ERR", kafka_error.name(), kafka_error.str(), file=sys.stderr)

def delivery_cb(err, msg):
if err:
print('message failed:', err, file=sys.stderr)

def calculate_thoughput(timing, n_messages=1000000, msg_size=100):
print("Processed {0} messsages in {1:.2f} seconds".format(n_messages, timing))
print("{0:.2f} MB/s".format((msg_size * n_messages) / timing / (1024*1024)))
print("{0:.2f} Msgs/s".format(n_messages / timing))

def confluent_kafka_producer_performance():
msg_count = 100
msg_size = 100
msg_payload = ('kafkatest' * 20).encode()[:msg_size]
topic = "short_name_new"

prod_config = {
    "error_cb": error_cb,
    "on_delivery": delivery_cb,
    "bootstrap.servers": "10.20.7.12:9092,10.20.7.14:9092,10.20.7.18:9092",
    "group.id": "python_inector",
    "retry.backoff.ms": 3000,
    "retries": 5,
    "default.topic.config": {"request.required.acks": "all"},
    "max.in.flight.requests.per.connection": 1,
    "queue.buffering.max.messages": 100000,
    "batch.num.messages": 100,
    "message.max.bytes": 2000000,
    "debug":"broker,topic,metadata"
}

producer = confluent_kafka.Producer(**prod_config)

producer_start = time.time()

for i in range(msg_count):
    while True:
        try:
            producer.produce(topic, value=msg_payload)
            producer.poll(0)
            break
        except BufferError as e:
            print(e, file=sys.stderr)
            producer.poll(1)

producer.flush()

return time.time() - producer_start

producer_timings["confluent_kafka_producer"] = confluent_kafka_producer_performance()
calculate_thoughput(producer_timings["confluent_kafka_producer"])

Checklist

Please provide the following information:

• confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()):
  confluent-kafka-python : ('1.4.1', 17039616)
  librdkafka version:('1.4.0', 17039615)

• Apache Kafka broker version:
  confluent_kafka 5.2.2

• Client configuration: {...}

• Operating system:

• Provide client logs (with 'debug': '..' as necessary)
  %7|1588299512.083|BROKER|rdkafka#producer-1| [thrd:app]: 10.20.7.12:9092/bootstrap: Added new broker with NodeId -1
  %7|1588299512.083|BRKMAIN|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Enter main broker thread
  %7|1588299512.083|CONNECT|rdkafka#producer-1| [thrd:app]: 10.20.7.12:9092/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
  %7|1588299512.083|BRKMAIN|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Enter main broker thread
  %7|1588299512.083|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v1.4.0 (0x10400ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_plain,sasl_scram,plugins,sasl_oauthbearer, GCC GXX PKGCONFIG INSTALL GNULD LDS LIBDL PLUGINS ZLIB SSL HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0xe)
  %7|1588299512.083|CONNECT|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Received CONNECT op
  %7|1588299512.083|STATE|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
  %7|1588299512.083|CONNECT|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: broker in state TRY_CONNECT connecting
  %7|1588299512.083|STATE|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
  %7|1588299512.083|TOPIC|rdkafka#producer-1| [thrd:app]: New local topic: short_name_new
  %7|1588299512.083|TOPPARNEW|rdkafka#producer-1| [thrd:app]: NEW short_name_new [-1] 0xaa8ce0 (at rd_kafka_topic_new0:438)
  %7|1588299512.083|CONNECT|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Connecting to ipv4#10.20.7.12:9092 (plaintext) with socket 7
  %7|1588299512.083|CONNECT|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Connected to ipv4#10.20.7.12:9092
  %7|1588299512.083|CONNECTED|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Connected (kafkatest, refactoring, and bug fixes #1)
  %7|1588299512.083|FEATURE|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion
  %7|1588299512.083|STATE|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY
  %7|1588299512.083|PROTOERR|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Protocol parse failure for ApiVersion v3 at 3/6 (rd_kafka_handle_ApiVersion:1942) (incorrect broker.version.fallback?)
  %7|1588299512.083|PROTOERR|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: ApiArrayCnt -1 out of range
  %7|1588299512.083|APIVERSION|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: ApiVersionRequest v3 failed due to UNSUPPORTED_VERSION: retrying with v0
  %7|1588299512.084|FEATURE|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq
  %7|1588299512.084|STATE|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Broker changed state APIVERSION_QUERY -> UP
  %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: Hinted cache of 1/1 topic(s) being queried
  %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: Requesting metadata for 1/1 topics: connected
  %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:10.20.7.12:9092/bootstrap]: 10.20.7.12:9092/bootstrap: Request metadata for 1 topic(s): connected
  %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: ===== Received metadata (for 1 requested topics): connected =====
  %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: ClusterId: Prq_BeAAT4693Lia4LGWVw, ControllerId: 0
  %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: 3 brokers, 1 topics
  %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: Broker #0/3: kafka-1:9092 NodeId 0
  %7|1588299512.084|BROKER|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Added new broker with NodeId 0
  %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: Broker kafkatest, refactoring, and bug fixes #1/3: kafka-3:9092 NodeId 2
  %7|1588299512.084|BRKMAIN|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Enter main broker thread
  %7|1588299512.084|BROKER|rdkafka#producer-1| [thrd:main]: kafka-3:9092/2: Added new broker with NodeId 2
  %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: Broker Use ReadTheDocs theme for building docs. #2/3: kafka-2:9092 NodeId 1
  %7|1588299512.084|BRKMAIN|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Enter main broker thread
  %7|1588299512.084|BROKER|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Added new broker with NodeId 1
  %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: Topic #0/1: short_name_new with 3 partitions
  %7|1588299512.084|STATE|rdkafka#producer-1| [thrd:main]: Topic short_name_new changed state unknown -> exists
  %7|1588299512.084|PARTCNT|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition count changed from 0 to 3
  %7|1588299512.084|TOPPARNEW|rdkafka#producer-1| [thrd:main]: NEW short_name_new [0] 0x7fe9400059c0 (at rd_kafka_topic_partition_cnt_update:747)
  %7|1588299512.084|TOPPARNEW|rdkafka#producer-1| [thrd:main]: NEW short_name_new [1] 0x7fe940005f20 (at rd_kafka_topic_partition_cnt_update:747)
  %7|1588299512.084|TOPPARNEW|rdkafka#producer-1| [thrd:main]: NEW short_name_new [2] 0x7fe940006480 (at rd_kafka_topic_partition_cnt_update:747)
  %7|1588299512.084|BRKMAIN|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Enter main broker thread
  %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 0 Leader 2
  %7|1588299512.084|BRKDELGT|rdkafka#producer-1| [thrd:main]: short_name_new [0]: delegate to broker kafka-3:9092/2 (rktp 0x7fe9400059c0, term 0, ref 2)
  %7|1588299512.084|BRKDELGT|rdkafka#producer-1| [thrd:main]: short_name_new [0]: delegating to broker kafka-3:9092/2 for partition with 0 messages (0 bytes) queued
  %7|1588299512.084|BRKMIGR|rdkafka#producer-1| [thrd:main]: Migrating topic short_name_new [0] 0x7fe9400059c0 from (none) to kafka-3:9092/2 (sending PARTITION_JOIN to kafka-3:9092/2)
  %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 1 Leader 0
  %7|1588299512.084|BRKDELGT|rdkafka#producer-1| [thrd:main]: short_name_new [1]: delegate to broker kafka-1:9092/0 (rktp 0x7fe940005f20, term 0, ref 2)
  %7|1588299512.084|TOPBRK|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Topic short_name_new [0]: joining broker (rktp 0x7fe9400059c0, 0 message(s) queued)
  %7|1588299512.084|FETCHADD|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Added short_name_new [0] to active list (1 entries, opv 0, 0 messages queued): joining
  %7|1588299512.084|BRKDELGT|rdkafka#producer-1| [thrd:main]: short_name_new [1]: delegating to broker kafka-1:9092/0 for partition with 0 messages (0 bytes) queued
  %7|1588299512.084|BRKMIGR|rdkafka#producer-1| [thrd:main]: Migrating topic short_name_new [1] 0x7fe940005f20 from (none) to kafka-1:9092/0 (sending PARTITION_JOIN to kafka-1:9092/0)
  %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 2 Leader 1
  %7|1588299512.084|BRKDELGT|rdkafka#producer-1| [thrd:main]: short_name_new [2]: delegate to broker kafka-2:9092/1 (rktp 0x7fe940006480, term 0, ref 2)
  %7|1588299512.084|TOPBRK|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Topic short_name_new [1]: joining broker (rktp 0x7fe940005f20, 0 message(s) queued)
  %7|1588299512.084|FETCHADD|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Added short_name_new [1] to active list (1 entries, opv 0, 0 messages queued): joining
  %7|1588299512.084|BRKDELGT|rdkafka#producer-1| [thrd:main]: short_name_new [2]: delegating to broker kafka-2:9092/1 for partition with 0 messages (0 bytes) queued
  %7|1588299512.084|BRKMIGR|rdkafka#producer-1| [thrd:main]: Migrating topic short_name_new [2] 0x7fe940006480 from (none) to kafka-2:9092/1 (sending PARTITION_JOIN to kafka-2:9092/1)
  %7|1588299512.084|PARTCNT|rdkafka#producer-1| [thrd:main]: Partitioning 100 unassigned messages in topic short_name_new to 3 partitions
  %7|1588299512.084|TOPBRK|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Topic short_name_new [2]: joining broker (rktp 0x7fe940006480, 0 message(s) queued)
  %7|1588299512.084|FETCHADD|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Added short_name_new [2] to active list (1 entries, opv 0, 0 messages queued): joining
  %7|1588299512.084|TOPPAR|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: short_name_new [1] 1 message(s) queued but broker not up
  %7|1588299512.084|TOPPAR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: short_name_new [0] 3 message(s) queued but broker not up
  %7|1588299512.084|STATE|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Broker changed state INIT -> TRY_CONNECT
  %7|1588299512.084|STATE|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Broker changed state INIT -> TRY_CONNECT
  %7|1588299512.084|CONNECT|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: broker in state TRY_CONNECT connecting
  %7|1588299512.084|UAS|rdkafka#producer-1| [thrd:main]: 100/100 messages were partitioned in topic short_name_new
  %7|1588299512.084|STATE|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Broker changed state TRY_CONNECT -> CONNECT
  %7|1588299512.084|CONNECT|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: broker in state TRY_CONNECT connecting
  %7|1588299512.084|STATE|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Broker changed state TRY_CONNECT -> CONNECT
  %7|1588299512.084|TOPPAR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: short_name_new [2] 18 message(s) queued but broker not up
  %7|1588299512.084|STATE|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Broker changed state INIT -> TRY_CONNECT
  %7|1588299512.085|CONNECT|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: broker in state TRY_CONNECT connecting
  %7|1588299512.085|STATE|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Broker changed state TRY_CONNECT -> CONNECT
  %7|1588299512.084|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: 1/1 requested topic(s) seen in metadata
  %7|1588299512.085|CLUSTERID|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: ClusterId update "" -> "Prq_BeAAT4693Lia4LGWVw"
  %7|1588299512.085|CONTROLLERID|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: ControllerId update -1 -> 0
  %7|1588299512.085|CONNECT|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Connecting to ipv4#10.20.7.18:9092 (plaintext) with socket 14
  %7|1588299512.085|CONNECT|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Connecting to ipv4#10.20.7.12:9092 (plaintext) with socket 15
  %7|1588299512.085|TOPPAR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: short_name_new [0] 36 message(s) queued but broker not up
  %7|1588299512.085|TOPPAR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: short_name_new [0] 36 message(s) queued but broker not up
  %7|1588299512.085|TOPPAR|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: short_name_new [1] 30 message(s) queued but broker not up
  %7|1588299512.085|CONNECT|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Connecting to ipv4#10.20.7.14:9092 (plaintext) with socket 16
  %7|1588299512.085|TOPPAR|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: short_name_new [1] 30 message(s) queued but broker not up
  %7|1588299512.085|TOPPAR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: short_name_new [2] 34 message(s) queued but broker not up
  %7|1588299512.085|TOPPAR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: short_name_new [2] 34 message(s) queued but broker not up
  %7|1588299512.085|CONNECT|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Connected to ipv4#10.20.7.12:9092
  %7|1588299512.085|CONNECTED|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Connected (kafkatest, refactoring, and bug fixes #1)
  %7|1588299512.085|FEATURE|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Updated enabled protocol features +ApiVersion to ApiVersion
  %7|1588299512.085|STATE|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Broker changed state CONNECT -> APIVERSION_QUERY
  %7|1588299512.085|TOPPAR|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: short_name_new [1] 30 message(s) queued but broker not up
  %7|1588299512.085|CONNECT|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Connected to ipv4#10.20.7.18:9092
  %7|1588299512.085|CONNECTED|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Connected (kafkatest, refactoring, and bug fixes #1)
  %7|1588299512.085|FEATURE|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Updated enabled protocol features +ApiVersion to ApiVersion
  %7|1588299512.085|STATE|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Broker changed state CONNECT -> APIVERSION_QUERY
  %7|1588299512.085|TOPPAR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: short_name_new [0] 36 message(s) queued but broker not up
  %7|1588299512.085|CONNECT|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Connected to ipv4#10.20.7.14:9092
  %7|1588299512.085|CONNECTED|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Connected (kafkatest, refactoring, and bug fixes #1)
  %7|1588299512.085|FEATURE|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Updated enabled protocol features +ApiVersion to ApiVersion
  %7|1588299512.085|STATE|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Broker changed state CONNECT -> APIVERSION_QUERY
  %7|1588299512.085|TOPPAR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: short_name_new [2] 34 message(s) queued but broker not up
  %7|1588299512.085|PROTOERR|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Protocol parse failure for ApiVersion v3 at 3/6 (rd_kafka_handle_ApiVersion:1942) (incorrect broker.version.fallback?)
  %7|1588299512.085|PROTOERR|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: ApiArrayCnt -1 out of range
  %7|1588299512.085|APIVERSION|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: ApiVersionRequest v3 failed due to UNSUPPORTED_VERSION: retrying with v0
  %7|1588299512.085|TOPPAR|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: short_name_new [1] 30 message(s) queued but broker not up
  %7|1588299512.085|FEATURE|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq
  %7|1588299512.085|STATE|rdkafka#producer-1| [thrd:kafka-1:9092/0]: kafka-1:9092/0: Broker changed state APIVERSION_QUERY -> UP
  %7|1588299512.085|METADATA|rdkafka#producer-1| [thrd:kafka-1:9092/0]: Skipping metadata refresh of 1 topic(s): connected: already being requested
  %7|1588299512.086|PROTOERR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Protocol parse failure for ApiVersion v3 at 3/6 (rd_kafka_handle_ApiVersion:1942) (incorrect broker.version.fallback?)
  %7|1588299512.086|PROTOERR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: ApiArrayCnt -1 out of range
  %7|1588299512.086|APIVERSION|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: ApiVersionRequest v3 failed due to UNSUPPORTED_VERSION: retrying with v0
  %7|1588299512.086|TOPPAR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: short_name_new [0] 36 message(s) queued but broker not up
  %7|1588299512.086|FEATURE|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq
  %7|1588299512.086|STATE|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: Broker changed state APIVERSION_QUERY -> UP
  %7|1588299512.086|METADATA|rdkafka#producer-1| [thrd:kafka-3:9092/2]: Skipping metadata refresh of 1 topic(s): connected: already being requested
  %7|1588299512.086|PROTOERR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Protocol parse failure for ApiVersion v3 at 3/6 (rd_kafka_handle_ApiVersion:1942) (incorrect broker.version.fallback?)
  %7|1588299512.086|PROTOERR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: ApiArrayCnt -1 out of range
  %7|1588299512.086|APIVERSION|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: ApiVersionRequest v3 failed due to UNSUPPORTED_VERSION: retrying with v0
  %7|1588299512.086|TOPPAR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: short_name_new [2] 34 message(s) queued but broker not up
  %7|1588299512.086|FEATURE|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq
  %7|1588299512.086|STATE|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: Broker changed state APIVERSION_QUERY -> UP
  %7|1588299512.086|METADATA|rdkafka#producer-1| [thrd:kafka-2:9092/1]: Skipping metadata refresh of 1 topic(s): connected: already being requested
  %7|1588299512.086|REQERR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
  %7|1588299512.086|BROKERUA|rdkafka#producer-1| [thrd:kafka-3:9092/2]: short_name_new [0]: broker unavailable: produce: Broker: Not leader for partition
  %7|1588299512.086|FASTQUERY|rdkafka#producer-1| [thrd:kafka-3:9092/2]: Starting fast leader query
  %7|1588299512.086|REQERR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
  %7|1588299512.086|BROKERUA|rdkafka#producer-1| [thrd:kafka-2:9092/1]: short_name_new [2]: broker unavailable: produce: Broker: Not leader for partition
  %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: Requesting metadata for 1/1 topics: partition leader query
  %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Request metadata for 1 topic(s): partition leader query
  %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: ===== Received metadata (for 1 requested topics): partition leader query =====
  %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: ClusterId: Prq_BeAAT4693Lia4LGWVw, ControllerId: 0
  %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: 3 brokers, 1 topics
  %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Broker #0/3: kafka-1:9092 NodeId 0
  %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Broker kafkatest, refactoring, and bug fixes #1/3: kafka-3:9092 NodeId 2
  %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Broker Use ReadTheDocs theme for building docs. #2/3: kafka-2:9092 NodeId 1
  %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Topic #0/1: short_name_new with 3 partitions
  %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 0 Leader 2
  %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 1 Leader 0
  %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 2 Leader 1
  %7|1588299513.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: 1/1 requested topic(s) seen in metadata
  %7|1588299515.087|REQERR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
  %7|1588299515.087|BROKERUA|rdkafka#producer-1| [thrd:kafka-3:9092/2]: short_name_new [0]: broker unavailable: produce: Broker: Not leader for partition
  %7|1588299515.087|FASTQUERY|rdkafka#producer-1| [thrd:kafka-3:9092/2]: Starting fast leader query
  %7|1588299515.088|REQERR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
  %7|1588299515.088|BROKERUA|rdkafka#producer-1| [thrd:kafka-2:9092/1]: short_name_new [2]: broker unavailable: produce: Broker: Not leader for partition
  %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: Requesting metadata for 1/1 topics: partition leader query
  %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: Request metadata for 1 topic(s): partition leader query
  %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: ===== Received metadata (for 1 requested topics): partition leader query =====
  %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: ClusterId: Prq_BeAAT4693Lia4LGWVw, ControllerId: 0
  %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: 3 brokers, 1 topics
  %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: Broker #0/3: kafka-1:9092 NodeId 0
  %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: Broker kafkatest, refactoring, and bug fixes #1/3: kafka-3:9092 NodeId 2
  %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: Broker Use ReadTheDocs theme for building docs. #2/3: kafka-2:9092 NodeId 1
  %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: Topic #0/1: short_name_new with 3 partitions
  %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 0 Leader 2
  %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 1 Leader 0
  %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 2 Leader 1
  %7|1588299516.082|METADATA|rdkafka#producer-1| [thrd:main]: 10.20.7.12:9092/bootstrap: 1/1 requested topic(s) seen in metadata
  %7|1588299518.089|REQERR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
  %7|1588299518.089|BROKERUA|rdkafka#producer-1| [thrd:kafka-3:9092/2]: short_name_new [0]: broker unavailable: produce: Broker: Not leader for partition
  %7|1588299518.089|FASTQUERY|rdkafka#producer-1| [thrd:kafka-3:9092/2]: Starting fast leader query
  %7|1588299518.089|REQERR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
  %7|1588299518.089|BROKERUA|rdkafka#producer-1| [thrd:kafka-2:9092/1]: short_name_new [2]: broker unavailable: produce: Broker: Not leader for partition
  %7|1588299519.082|METADATA|rdkafka#producer-1| [thrd:main]: Requesting metadata for 1/1 topics: partition leader query
  %7|1588299519.082|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Request metadata for 1 topic(s): partition leader query
  %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: ===== Received metadata (for 1 requested topics): partition leader query =====
  %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: ClusterId: Prq_BeAAT4693Lia4LGWVw, ControllerId: 0
  %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: 3 brokers, 1 topics
  %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker #0/3: kafka-1:9092 NodeId 0
  %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker kafkatest, refactoring, and bug fixes #1/3: kafka-3:9092 NodeId 2
  %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker Use ReadTheDocs theme for building docs. #2/3: kafka-2:9092 NodeId 1
  %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Topic #0/1: short_name_new with 3 partitions
  %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 0 Leader 2
  %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 1 Leader 0
  %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 2 Leader 1
  %7|1588299519.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: 1/1 requested topic(s) seen in metadata
  %7|1588299521.090|REQERR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
  %7|1588299521.090|BROKERUA|rdkafka#producer-1| [thrd:kafka-3:9092/2]: short_name_new [0]: broker unavailable: produce: Broker: Not leader for partition
  %7|1588299521.090|FASTQUERY|rdkafka#producer-1| [thrd:kafka-3:9092/2]: Starting fast leader query
  %7|1588299521.091|REQERR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
  %7|1588299521.091|BROKERUA|rdkafka#producer-1| [thrd:kafka-2:9092/1]: short_name_new [2]: broker unavailable: produce: Broker: Not leader for partition
  %7|1588299522.082|METADATA|rdkafka#producer-1| [thrd:main]: Requesting metadata for 1/1 topics: partition leader query
  %7|1588299522.082|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Request metadata for 1 topic(s): partition leader query
  %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: ===== Received metadata (for 1 requested topics): partition leader query =====
  %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: ClusterId: Prq_BeAAT4693Lia4LGWVw, ControllerId: 0
  %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: 3 brokers, 1 topics
  %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker #0/3: kafka-1:9092 NodeId 0
  %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker kafkatest, refactoring, and bug fixes #1/3: kafka-3:9092 NodeId 2
  %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker Use ReadTheDocs theme for building docs. #2/3: kafka-2:9092 NodeId 1
  %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Topic #0/1: short_name_new with 3 partitions
  %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 0 Leader 2
  %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 1 Leader 0
  %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 2 Leader 1
  %7|1588299522.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: 1/1 requested topic(s) seen in metadata
  %7|1588299524.092|REQERR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
  %7|1588299524.092|BROKERUA|rdkafka#producer-1| [thrd:kafka-3:9092/2]: short_name_new [0]: broker unavailable: produce: Broker: Not leader for partition
  %7|1588299524.092|FASTQUERY|rdkafka#producer-1| [thrd:kafka-3:9092/2]: Starting fast leader query
  %7|1588299524.093|REQERR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
  %7|1588299524.093|BROKERUA|rdkafka#producer-1| [thrd:kafka-2:9092/1]: short_name_new [2]: broker unavailable: produce: Broker: Not leader for partition
  %7|1588299525.083|METADATA|rdkafka#producer-1| [thrd:main]: Requesting metadata for 1/1 topics: partition leader query
  %7|1588299525.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Request metadata for 1 topic(s): partition leader query
  %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: ===== Received metadata (for 1 requested topics): partition leader query =====
  %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: ClusterId: Prq_BeAAT4693Lia4LGWVw, ControllerId: 0
  %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: 3 brokers, 1 topics
  %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Broker #0/3: kafka-1:9092 NodeId 0
  %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Broker kafkatest, refactoring, and bug fixes #1/3: kafka-3:9092 NodeId 2
  %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Broker Use ReadTheDocs theme for building docs. #2/3: kafka-2:9092 NodeId 1
  %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Topic #0/1: short_name_new with 3 partitions
  %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 0 Leader 2
  %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 1 Leader 0
  %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 2 Leader 1
  %7|1588299525.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: 1/1 requested topic(s) seen in metadata
  %7|1588299527.093|REQERR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
  %7|1588299527.093|BROKERUA|rdkafka#producer-1| [thrd:kafka-3:9092/2]: short_name_new [0]: broker unavailable: produce: Broker: Not leader for partition
  %7|1588299527.093|FASTQUERY|rdkafka#producer-1| [thrd:kafka-3:9092/2]: Starting fast leader query
  %7|1588299527.094|REQERR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
  %7|1588299527.094|BROKERUA|rdkafka#producer-1| [thrd:kafka-2:9092/1]: short_name_new [2]: broker unavailable: produce: Broker: Not leader for partition
  %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: Requesting metadata for 1/1 topics: partition leader query
  %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Request metadata for 1 topic(s): partition leader query
  %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: ===== Received metadata (for 1 requested topics): partition leader query =====
  %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: ClusterId: Prq_BeAAT4693Lia4LGWVw, ControllerId: 0
  %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: 3 brokers, 1 topics
  %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker #0/3: kafka-1:9092 NodeId 0
  %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker kafkatest, refactoring, and bug fixes #1/3: kafka-3:9092 NodeId 2
  %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker Use ReadTheDocs theme for building docs. #2/3: kafka-2:9092 NodeId 1
  %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Topic #0/1: short_name_new with 3 partitions
  %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 0 Leader 2
  %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 1 Leader 0
  %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 2 Leader 1
  %7|1588299528.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: 1/1 requested topic(s) seen in metadata
  %7|1588299530.095|REQERR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
  %7|1588299530.095|BROKERUA|rdkafka#producer-1| [thrd:kafka-3:9092/2]: short_name_new [0]: broker unavailable: produce: Broker: Not leader for partition
  %7|1588299530.095|FASTQUERY|rdkafka#producer-1| [thrd:kafka-3:9092/2]: Starting fast leader query
  %7|1588299530.097|REQERR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
  %7|1588299530.097|BROKERUA|rdkafka#producer-1| [thrd:kafka-2:9092/1]: short_name_new [2]: broker unavailable: produce: Broker: Not leader for partition
  %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: Requesting metadata for 1/1 topics: partition leader query
  %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Request metadata for 1 topic(s): partition leader query
  %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: ===== Received metadata (for 1 requested topics): partition leader query =====
  %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: ClusterId: Prq_BeAAT4693Lia4LGWVw, ControllerId: 0
  %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: 3 brokers, 1 topics
  %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker #0/3: kafka-1:9092 NodeId 0
  %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker kafkatest, refactoring, and bug fixes #1/3: kafka-3:9092 NodeId 2
  %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Broker Use ReadTheDocs theme for building docs. #2/3: kafka-2:9092 NodeId 1
  %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: Topic #0/1: short_name_new with 3 partitions
  %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 0 Leader 2
  %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 1 Leader 0
  %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 2 Leader 1
  %7|1588299531.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-2:9092/1: 1/1 requested topic(s) seen in metadata
  %7|1588299533.096|REQERR|rdkafka#producer-1| [thrd:kafka-3:9092/2]: kafka-3:9092/2: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
  %7|1588299533.096|BROKERUA|rdkafka#producer-1| [thrd:kafka-3:9092/2]: short_name_new [0]: broker unavailable: produce: Broker: Not leader for partition
  %7|1588299533.096|FASTQUERY|rdkafka#producer-1| [thrd:kafka-3:9092/2]: Starting fast leader query
  %7|1588299533.099|REQERR|rdkafka#producer-1| [thrd:kafka-2:9092/1]: kafka-2:9092/1: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
  %7|1588299533.099|BROKERUA|rdkafka#producer-1| [thrd:kafka-2:9092/1]: short_name_new [2]: broker unavailable: produce: Broker: Not leader for partition
  %7|1588299534.083|METADATA|rdkafka#producer-1| [thrd:main]: Requesting metadata for 1/1 topics: partition leader query
  %7|1588299534.083|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Request metadata for 1 topic(s): partition leader query
  %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: ===== Received metadata (for 1 requested topics): partition leader query =====
  %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: ClusterId: Prq_BeAAT4693Lia4LGWVw, ControllerId: 0
  %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: 3 brokers, 1 topics
  %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Broker #0/3: kafka-1:9092 NodeId 0
  %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Broker kafkatest, refactoring, and bug fixes #1/3: kafka-3:9092 NodeId 2
  %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Broker Use ReadTheDocs theme for building docs. #2/3: kafka-2:9092 NodeId 1
  %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: Topic #0/1: short_name_new with 3 partitions
  %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 0 Leader 2
  %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 1 Leader 0
  %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: Topic short_name_new partition 2 Leader 1
  %7|1588299534.084|METADATA|rdkafka#producer-1| [thrd:main]: kafka-1:9092/0: 1/1 requested topic(s) seen in metadata

• Provide broker log excerpts

• Critical issue

[edenhill]

Your cluster state is messed up, the partition leader for partition 0 as reported in Metadata is broker 2, but broker 2 responds to Produce requests with the error saying it is not the leader.
Suggest you look into the health of the Kafka cluster.