2.1.0

2.1.0 (Mar 15, 2025)

Support Kafka Broker 2.1 API Baseline

• Add baseline leader_epoch support for ListOffsets v4 / FetchRequest v10 (#2511)
• Support OffsetFetch v5 / OffsetCommit v6 (2.1 baseline) (#2505)
• Support 2.1 baseline consumer group apis (#2503)
• Support FindCoordinatorRequest v2 in consumer and admin client (#2502)
• Support ListOffsets v3 in consumer (#2501)
• Support Fetch Request/Response v6 in consumer (#2500)
• Add support for Metadata Request/Response v7 (#2497)
• Implement Incremental Fetch Sessions / KIP-227 (#2508)
• Implement client-side connection throttling / KIP-219 (#2510)
• Add KafkaClient.api_version(operation) for best available from api_versions (#2495)

Consumer

• Timeout coordinator poll / ensure_coordinator_ready / ensure_active_group (#2526)
• Add optional timeout_ms kwarg to remaining consumer/coordinator methods (#2544)
• Check for coordinator.poll failure in KafkaConsumer
• Only mark coordinator dead if connection_delay > 0 (#2530)
• Delay group coordinator until after bootstrap (#2539)
• KAFKA-4160: Ensure rebalance listener not called with coordinator lock (#1438)
• Call default_offset_commit_callback after _maybe_auto_commit_offsets_async (#2546)
• Remove legacy/v1 consumer message iterator (#2543)
• Log warning when attempting to list offsets for unknown topic/partition (#2540)
• Add heartbeat thread id to debug logs on start
• Add inner_timeout_ms handler to fetcher; add fallback (#2529)

Producer

• KafkaProducer: Flush pending records before close() (#2537)
• Raise immediate error on producer.send after close (#2542)
• Limit producer close timeout to 1sec in del; use context managers to close in test_producer
• Use NullLogger in producer atexit cleanup
• Attempt to fix metadata race condition when partitioning in producer.send (#2523)
• Remove unused partial KIP-467 implementation (ProduceResponse batch error details) (#2524)

AdminClient

• Implement perform leader election (#2536)
• Support delete_records (#2535)

Networking

• Call ApiVersionsRequest during connection, prior to Sasl Handshake (#2493)
• Fake api_versions for old brokers, rename to ApiVersionsRequest, and handle error decoding (#2494)
• Debug log when skipping api_versions request with pre-configured api_version
• Only refresh metadata if connection fails all dns records (#2532)
• Support connections through SOCKS5 proxies (#2531)
• Fix OverflowError when connection_max_idle_ms is 0 or inf (#2538)
• socket.setblocking for eventlet/gevent compatibility
• Support custom per-request timeouts (#2498)
• Include request_timeout_ms in request debug log
• Support client.poll with future and timeout_ms
• mask unused afi var
• Debug log if check_version connection attempt fails

SASL Modules

• Refactor Sasl authentication with SaslMechanism abstract base class; support SaslAuthenticate (#2515)
• Add SSPI (Kerberos for Windows) authentication mechanism (#2521)
• Support AWS_MSK_IAM authentication (#2519)
• Cleanup sasl mechanism configuration checks; fix gssapi bugs; add sasl_kerberos_name config (#2520)
• Move kafka.oauth.AbstractTokenProvider -> kafka.sasl.oauth.AbstractTokenProvider (#2525)

Testing

• Bump default python to 3.13 in CI tests (#2541)
• Update pytest log_format: use logger instead of filename; add thread id
• Improve test_consumer_group::test_group logging before group stabilized (#2534)
• Limit test duration to 5mins w/ pytest-timeout
• Fix external kafka/zk fixtures for testing (#2533)
• Disable zookeeper admin server to avoid port conflicts
• Set default pytest log level to debug
• test_group: shorter timeout, more logging, more sleep
• Cache servers/dist in github actions workflow (#2527)
• Remove tox.ini; update testing docs
• Use thread-specific client_id in test_group
• Fix subprocess log warning; specify timeout_ms kwarg in consumer.poll tests
• Only set KAFKA_JVM_PERFORMANCE_OPTS in makefile if unset; add note re: 2.0-2.3 broker testing
• Add kafka command to test.fixtures; raise FileNotFoundError if version not installed

Documentation

• Improve ClusterMetadata docs re: node_id/broker_id str/int types
• Document api_version_auto_timeout_ms default; override in group tests

Fixes

• Signal close to metrics expire_loop
• Add kafka.util timeout_ms_fn
• fixup TopicAuthorizationFailedError construction
• Fix lint issues via ruff check (#2522)
• Make the "mock" dependency optional (only used in Python < 3.3). (#2518)

2.0.6

2.0.6 (Mar 4, 2025)

Networking

• Improve error handling in client._maybe_connect (#2504)
• Client connection / maybe_refresh_metadata changes (#2507)
• Improve too-large timeout handling in client poll
• Default client.check_version timeout to api_version_auto_timeout_ms (#2496)

Fixes

• Decode and skip transactional control records in consumer (#2499)
• try / except in consumer coordinator __del__

Testing

• test_conn fixup for py2

Project Maintenance

• Add 2.0 branch for backports

2.0.5

2.0.5 (Feb 25, 2025)

Networking

• Remove unused client bootstrap backoff code
• 200ms timeout for client.poll in ensure_active_group and admin client

Fixes

• Admin client: check_version only if needed, use node_id kwarg for controller
• Check for -1 controller_id in admin client
• Only acquire coordinator lock in heartbeat thread close if not self thread

Testing

• Also sleep when waiting for consumers in test_describe_consumer_group_exists
• Refactor sasl_integration test_client - wait for node ready; use send future
• Add timeout to test_kafka_consumer
• Add error str to assert_message_count checks
• Retry on error in test fixture create_topic_via_metadata
• Fixup variable interpolation in test fixture error

Documentation

• Update compatibility docs
• Include client_id in BrokerConnection str output

Project Maintenance

• Add make targets servers/*/api_versions and servers/*/messages

2.0.4

2.0.4 (Feb 21, 2025)

Networking

• Check for wakeup socket errors on read and close and reinit to reset (#2482)
• Improve client networking backoff / retry (#2480)
• Check for socket and unresolved futures before creating selector in conn.check_version (#2477)
• Handle socket init errors, e.g., when IPv6 is disabled (#2476)

Fixes

• Avoid self-join in heartbeat thread close (#2488)

Error Handling

• Always log broker errors in producer.send (#2478)
• Retain unrecognized broker response error codes with dynamic error class (#2481)
• Update kafka.errors with latest types (#2485)

Compatibility

• Do not validate snappy xerial header version and compat fields (for redpanda) (#2483)

Documentation

• Added missing docstrings in admin/client.py (#2487)

Testing

• Update kafka broker test matrix; test against 3.9.0 (#2486)
• Add default resources for new kafka server fixtures (#2484)
• Drop make test-local; add PYTESTS configuration var
• Fix pytest runs when KAFKA_VERSION is not set

Project Maintenance

• Migrate to pyproject.toml / PEP-621
• Remove old travis files; update compatibility tests link to gha

2.0.3

2.0.3 (Feb 12, 2025)

Improvements

• Add optional compression libs to extras_require (#2123, #2387)
• KafkaConsumer: Exit poll if consumer is closed (#2152)
• Support configuration of custom kafka client for Admin/Consumer/Producer (#2144)
• Core Protocol: Add support for flexible versions (#2151)
• (Internal) Allow disabling thread wakeup in _send_request_to_node (#2335)
• Change loglevel of cancelled errors to info (#2467)
• Strip trailing dot off hostname for SSL validation. (#2472)
• Log connection close(error) at ERROR level (#2473)
• Support DescribeLogDirs admin api (#2475)

Compatibility

• Support for python 3.12 (#2379, #2382)
• Kafka 2.5 / 2.6 (#2162)
• Try collections.abc imports in vendored selectors34 (#2394)
• Catch OSError when checking for gssapi import for windows compatibility (#2407)
• Update vendored six to 1.16.0 (#2398)

Documentation

• Update usage.rst (#2308, #2334)
• Fix typos (#2319, #2207, #2178)
• Fix links to the compatibility page (#2295, #2226)
• Cleanup install instructions for optional libs (#2139)
• Update license_file to license_files (#2462)
• Update some RST documentation syntax (#2463)
• Add .readthedocs.yaml; update copyright date (#2474)

Fixes

• Use isinstance in builtin crc32 (#2329)
• Use six.viewitems instead of six.iteritems to avoid encoding problems in StickyPartitionAssignor (#2154)
• Fix array encoding TypeError: object of type 'dict_itemiterator' has no len() (#2167)
• Only try to update sensors fetch lag if the unpacked list contains elements (#2158)
• Avoid logging errors during test fixture cleanup (#2458)
• Release coordinator lock before calling maybe_leave_group (#2460)
• Dont raise RuntimeError for dead process in SpawnedService.wait_for() (#2461)
• Cast the size of a MemoryRecordsBuilder object (#2438)
• Fix DescribeConfigsResponse_v1 config_source (#2464)
• Fix base class of DescribeClientQuotasResponse_v0 (#2465)
• Update socketpair w/ CVE-2024-3219 fix (#2468)

Testing

• Transition CI/CD to GitHub Workflows (#2378, #2392, #2381, #2406, #2419, #2418, #2417, #2456)
• Refactor Makefile (#2457)
• Use assert_called_with in client_async tests (#2375)
• Cover sticky assignor's metadata method with tests (#2161)
• Update fixtures.py to check "127.0.0.1" for auto port assignment (#2384)
• Use -Djava.security.manager=allow for Java 23 sasl tests (#2469)
• Test with Java 23 (#2470)
• Update kafka properties template; disable group rebalance delay (#2471)

2.0.2

2.0.2 (Sep 29, 2020)

Consumer

• KIP-54: Implement sticky partition assignment strategy (aynroot / PR #2057)
• Fix consumer deadlock when heartbeat thread request timeout (huangcuiyang / PR #2064)

Compatibility

• Python 3.8 support (Photonios / PR #2088)

Cleanups

• Bump dev requirements (jeffwidman / PR #2129)
• Fix crc32c deprecation warning (crc32c==2.1) (jeffwidman / PR #2128)
• Lint cleanup (jeffwidman / PR #2126)
• Fix initialization order in KafkaClient (pecalleja / PR #2119)
• Allow installing crc32c via extras (mishas / PR #2069)
• Remove unused imports (jameslamb / PR #2046)

Admin Client

• Merge _find_coordinator_id methods (jeffwidman / PR #2127)
• Feature: delete consumergroups (swenzel / PR #2040)
• Allow configurable timeouts in admin client check version (sunnyakaxd / PR #2107)
• Enhancement for Kafka Admin Client's "Describe Consumer Group" (Apurva007 / PR #2035)

Protocol

• Add support for zstd compression (gabriel-tincu / PR #2021)
• Add protocol support for brokers 1.1.0 - 2.5.0 (gabriel-tincu / PR #2038)
• Add ProduceRequest/ProduceResponse v6/v7/v8 (gabriel-tincu / PR #2020)
• Fix parsing NULL header values (kvfi / PR #2024)

Tests

• Add 2.5.0 to automated CI tests (gabriel-tincu / PR #2038)
• Add 2.1.1 to build_integration (gabriel-tincu / PR #2019)

Documentation / Logging / Errors

• Disable logging during producer object gc (gioele / PR #2043)
• Update example.py; use threading instead of multiprocessing (Mostafa-Elmenbawy / PR #2081)
• Fix typo in exception message (haracejacob / PR #2096)
• Add kafka.structs docstrings (Mostafa-Elmenbawy / PR #2080)
• Fix broken compatibility page link (anuragrana / PR #2045)
• Rename README to README.md (qhzxc0015 / PR #2055)
• Fix docs by adding SASL mention (jeffwidman / #1990)

2.0.1

2.0.1 (Feb 19, 2020)

Admin Client

• KAFKA-8962: Use least_loaded_node() for AdminClient.describe_topics() (jeffwidman / PR #2000)
• Fix AdminClient topic error parsing in MetadataResponse (jtribble / PR #1997)

2.0.0

2.0.0 (Feb 10, 2020)

This release includes breaking changes for any application code that has not
migrated from older Simple-style classes to newer Kafka-style classes.

Deprecation

• Remove deprecated SimpleClient, Producer, Consumer, Unittest (jeffwidman / PR #1196)

Admin Client

• Use the controller for topic metadata requests (TylerLubeck / PR #1995)
• Implement list_topics, describe_topics, and describe_cluster (TylerLubeck / PR #1993)
• Implement eq and hash for ACL objects (TylerLubeck / PR #1955)
• Fixes KafkaAdminClient returning IncompatibleBrokerVersion when passing an api_version (ian28223 / PR #1953)
• Admin protocol updates (TylerLubeck / PR #1948)
• Fix describe config for multi-broker clusters (jlandersen / PR #1869)

Miscellaneous Bugfixes / Improvements

• Enable SCRAM-SHA-256 and SCRAM-SHA-512 for sasl (swenzel / PR #1918)
• Fix slots usage and use more slots (carsonip / PR #1987)
• Optionally return OffsetAndMetadata from consumer.committed(tp) (dpkp / PR #1979)
• Reset conn configs on exception in conn.check_version() (dpkp / PR #1977)
• Do not block on sender thread join after timeout in producer.close() (dpkp / PR #1974)
• Implement methods to convert a Struct object to a pythonic object (TylerLubeck / PR #1951)

Test Infrastructure / Documentation / Maintenance

• Update 2.4.0 resource files for sasl integration (dpkp)
• Add kafka 2.4.0 to CI testing (vvuibert / PR #1972)
• convert test_admin_integration to pytest (ulrikjohansson / PR #1923)
• xfail test_describe_configs_topic_resource_returns_configs (dpkp / Issue #1929)
• Add crc32c to README and docs (dpkp)
• Improve docs for reconnect_backoff_max_ms (dpkp / PR #1976)
• Fix simple typo: managementment -> management (timgates42 / PR #1966)
• Fix typos (carsonip / PR #1938)
• Fix doc import paths (jeffwidman / PR #1933)
• Update docstring to match conn.py's (dabcoder / PR #1921)
• Do not log topic-specific errors in full metadata fetch (dpkp / PR #1980)
• Raise AssertionError if consumer closed in poll() (dpkp / PR #1978)
• Log retriable coordinator NodeNotReady, TooManyInFlightRequests as debug not error (dpkp / PR #1975)
• Remove unused import (jeffwidman)
• Remove some dead code (jeffwidman)
• Fix a benchmark to Use print() function in both Python 2 and Python 3 (cclauss / PR #1983)
• Fix a test to use ==/!= to compare str, bytes, and int literals (cclauss / PR #1984)
• Fix benchmarks to use pyperf (carsonip / PR #1986)
• Remove unused/empty .gitsubmodules file (jeffwidman / PR #1928)
• Remove deprecated ConnectionError (jeffwidman / PR #1816)

1.4.7

1.4.7 (Sep 30, 2019)

This release is focused on KafkaConsumer performance, Admin Client
improvements, and Client concurrency. The KafkaConsumer iterator implementation
has been greatly simplified so that it just wraps consumer.poll(). The prior
implementation will remain available for a few more releases using the optional
KafkaConsumer config: legacy_iterator=True . This is expected to improve
consumer throughput substantially and help reduce heartbeat failures / group
rebalancing.

Major thanks to @carsonip @Baisang @iv-m @davidheitman @cardy31 @ulrikjohansson @iAnomaly @Wayde2014 @ossdev07 @commanderdishwasher @justecorruptio @melor @rustyrothwurt @sachiin @jacky15 and @rikonen for submitting PRs; thanks as well to everyone that
submitted bug reports and issues, and to @jeffwidman and @tvoinarovskyi for code
reviews, comments, testing, debugging, and helping to maintain kafka-python!

Client

• Send socket data via non-blocking IO with send buffer (@dpkp / PR #1912)
• Rely on socket selector to detect completed connection attempts (@dpkp / PR #1909)
• Improve connection lock handling; always use context manager (@melor @dpkp / PR #1895)
• Reduce client poll timeout when there are no in-flight requests (@dpkp / PR #1823)

KafkaConsumer

• Do not use wakeup when sending fetch requests from consumer (@dpkp / PR #1911)
• Wrap consumer.poll() for KafkaConsumer iteration (@dpkp / PR #1902)
• Allow the coordinator to auto-commit on old brokers (@justecorruptio / PR #1832)
• Reduce internal client poll timeout for (legacy) consumer iterator interface (@dpkp / PR #1824)
• Use dedicated connection for group coordinator (@dpkp / PR #1822)
• Change coordinator lock acquisition order (@dpkp / PR #1821)
• Make partitions_for_topic a read-through cache (@Baisang / PR #1781,#1809)
• Fix consumer hanging indefinitely on topic deletion while rebalancing (@commanderdishwasher / PR #1782)

Miscellaneous Bugfixes / Improvements

• Fix crc32c avilability on non-intel architectures (@ossdev07 / PR #1904)
• Load system default SSL CAs if ssl_cafile is not provided (@iAnomaly / PR #1883)
• Catch py3 TimeoutError in BrokerConnection send/recv (@dpkp / PR #1820)
• Added a function to determine if bootstrap is successfully connected (@Wayde2014 / PR #1876)

Admin Client

• Add ACL api support to KafkaAdminClient (@ulrikjohansson / PR #1833)
• Add sasl_kerberos_domain_name config to KafkaAdminClient (@jeffwidman / PR #1852)
• Update security_protocol config documentation for KafkaAdminClient (@cardy31 / PR #1849)
• Break FindCoordinator into request/response methods in KafkaAdminClient (@jeffwidman / PR #1871)
• Break consumer operations into request / response methods in KafkaAdminClient (@jeffwidman / PR #1845)
• Parallelize calls to _send_request_to_node() in KafkaAdminClient (@davidheitman / PR #1807)

Test Infrastructure / Documentation / Maintenance

• Add Kafka 2.3.0 to test matrix and compatibility docs (dpkp / PR #1915)
• Convert remaining KafkaConsumer tests to pytest (@jeffwidman / PR #1886)
• Bump integration tests to 0.10.2.2 and 0.11.0.3 (@jeffwidman / #1890)
• Cleanup handling of KAFKA_VERSION env var in tests (@jeffwidman / PR #1887)
• Minor test cleanup (@jeffwidman / PR #1885)
• Use socket.SOCK_STREAM in test assertions (@iv-m / PR #1879)
• Sanity test for consumer.topics() and consumer.partitions_for_topic() (@Baisang / PR #1829)
• Cleanup seconds conversion in client poll timeout calculation (@jeffwidman / PR #1825)
• Remove unused imports (@jeffwidman / PR #1808)
• Cleanup python nits in RangePartitionAssignor (@jeffwidman / PR #1805)
• Update links to kafka consumer config docs (@jeffwidman)
• Fix minor documentation typos (@carsonip / PR #1865)
• Remove unused/weird comment line (@jeffwidman / PR #1813)
• Update docs for api_version_auto_timeout_ms (@jeffwidman / PR #1812)

1.4.6

1.4.6 (Apr 2, 2019)

This is a patch release primarily focused on bugs related to concurrency,
SSL connections and testing, and SASL authentication.
Major thanks to @pt2pham , @isamaru , @braedon , @gingercookiemage , for submitting
PRs to help fix many of these issues. And major thanks to everyone that
submitted bug reports and issues. And thanks always to @jeffwidman and @tvoinarovskyi for code
reviews, comments, testing, debugging, and helping to maintain this project!

Client Concurrency Issues (Race Conditions / Deadlocks)

• Fix race condition in protocol.send_bytes (isamaru / PR #1752)
• Do not call state_change_callback with lock (dpkp / PR #1775)
• Additional BrokerConnection locks to synchronize protocol/IFR state (dpkp / PR #1768)
• Send pending requests before waiting for responses (dpkp / PR #1762)
• Avoid race condition on client._conns in send() (dpkp / PR #1772)
• Hold lock during client.check_version (dpkp / PR #1771)

Producer Wakeup / TimeoutError

• Dont wakeup during maybe_refresh_metadata -- it is only called by poll() (dpkp / PR #1769)
• Dont do client wakeup when sending from sender thread (dpkp / PR #1761)

SSL - Python3.7 Support / Bootstrap Hostname Verification / Testing

• Wrap SSL sockets after connecting for python3.7 compatibility (dpkp / PR #1754)
• Allow configuration of SSL Ciphers (dpkp / PR #1755)
• Maintain shadow cluster metadata for bootstrapping (dpkp / PR #1753)
• Generate SSL certificates for local testing (dpkp / PR #1756)
• Rename ssl.keystore.location and ssl.truststore.location config files (dpkp)
• Reset reconnect backoff on SSL connection (dpkp / PR #1777)

SASL - OAuthBearer support / api version bugfix

• Fix 0.8.2 protocol quick detection / fix SASL version check (dpkp / PR #1763)
• Update sasl configuration docstrings to include supported mechanisms (dpkp)
• Support SASL OAuthBearer Authentication (pt2pham / PR #1750)

Miscellaneous Bugfixes

• Dont force metadata refresh when closing unneeded bootstrap connections (dpkp / PR #1773)
• Fix possible AttributeError during conn._close_socket (dpkp / PR #1776)
• Return connection state explicitly after close in connect() (dpkp / PR #1778)
• Fix flaky conn tests that use time.time (dpkp / PR #1758)
• Add py to requirements-dev (dpkp)
• Fixups to benchmark scripts for py3 / new KafkaFixture interface (dpkp)