Should Kafka stop logging during shutdown?

[mauhcs]

I keep having this issue with kafka:

Exception ignored in: <bound method KafkaProducer.__del__ of <kafka.producer.kafka.KafkaProducer object at 0x7f03d57c1518>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/kafka/producer/kafka.py", line 400, in __del__
  File "/usr/local/lib/python3.5/dist-packages/kafka/producer/kafka.py", line 413, in close
  File "/usr/lib/python3.5/logging/__init__.py", line 1279, in info
  File "/usr/lib/python3.5/logging/__init__.py", line 1415, in _log
  File "/usr/lib/python3.5/logging/__init__.py", line 1425, in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1487, in callHandlers
  File "/usr/lib/python3.5/logging/__init__.py", line 855, in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1047, in emit
  File "/usr/lib/python3.5/logging/__init__.py", line 1037, in _open
NameError: name 'open' is not defined

If you go to the lines 400 and 413 in the Kafka repo (you can use the link below), you will see that kafka is logging during shutdown:

kafka-python/kafka/producer/kafka.py

Line 399 in b1ae45c

def __del__(self):

This discussion in Python bug tracker suggests that logging during shutdown should be avoided:
https://docs.python.org/3/library/weakref.html

I made the following simple example to reproduce the issue. It appears when I am trying to define my own logger (setting the filename in the basicConfig):

#!/usr/bin/python3
'''
Demonstrate issue:
'''
from kafka import KafkaProducer
import logging

# set up logging to file - see previous section for more details
logging.basicConfig(level=logging.INFO,
                    filename='/tmp/my_log_file.log')

prod = KafkaProducer(bootstrap_servers="localhost")

def main():
    prod.send("test",b"my message.")

if __name__ == "__main__":
    main()

prod.close() #Edit from comments

Copy the code and save in a file called test.py and in the terminal run: python -m "import test"
You will get the issue I got. (if you have a kafka instance running in the localhost)

The problem with this is that I cannot define my own loggings and gracefully terminate the program.

Should we add some try/except in de close method in kafka?

Or I am missing something?

EDIT: I understand that the error 'comes' from me instantiating a producer outside any function call whatsoever. That does not seem to be a good idea and in itself should be avoided, but it is really confusing having this error poping up in this case.

[tvoinarovskyi]

You must close the producer yourself, if you don't, some messages may not be delivered...
Call producer.close() before shutdown.

[mauhcs]

Thank you for the feedback
I still get the error.
Let me add the close in the example.

[jeffwidman]

This discussion in Python bug tracker suggests that logging during shutdown should be avoided:
https://docs.python.org/3/library/weakref.html

It looks like you pasted the wrong link, do you mind updating, as I'm curious to read the discussion...

And I do agree that the error is confusing, I'm not sure what the right fix is, but be nice to not have something that inscrutable.

[dpkp]

I haven't used weakref.finalize before -- it might be a nice way to manage the cleanup functions.

[tvoinarovskyi]

@dpkp I am a bit confused by the __del__ part. Why would we need to close the producer on interpreter shutdown? I thought, that __del__ was mostly to avoid having opened connections if a user did not call close() properly. If the interpreter is closing up we don't need any safe closings, just let it die, no?