Opt out serialization/deserialization for heap type

[corona10]

BPO	41052
Nosy	@pitrou, @vstinner, @serhiy-storchaka, @corona10, @shihai1991
PRs	bpo-41052: Opt out serialization/deserialization for _random.Random #21002 [3.9] bpo-41052: Opt out serialization/deserialization for _random.Random (GH-21002). #21030 bpo-41052: Optout serialization/deserialization for blake2s/b #22189 bpo-41052: Fix pickling heap types implemented in C with protocols 0 and 1 #22870 [3.9] bpo-41052: Fix pickling heap types implemented in C with protocols 0 and 1 (GH-22870). #22963

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-10-25.16:09:30.699>
created_at = <Date 2020-06-20.13:17:29.949>
labels = ['expert-C-API', 'type-bug', '3.9', '3.10']
title = 'Opt out serialization/deserialization for heap type'
updated_at = <Date 2020-10-27.04:22:53.436>
user = 'https://github.com/corona10'

bugs.python.org fields:

activity = <Date 2020-10-27.04:22:53.436>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-10-25.16:09:30.699>
closer = 'serhiy.storchaka'
components = ['C API']
creation = <Date 2020-06-20.13:17:29.949>
creator = 'corona10'
dependencies = []
files = []
hgrepos = []
issue_num = 41052
keywords = ['patch']
message_count = 10.0
messages = ['371937', '371992', '371993', '376951', '379214', '379223', '379251', '379532', '379585', '379735']
nosy_count = 5.0
nosy_names = ['pitrou', 'vstinner', 'serhiy.storchaka', 'corona10', 'shihai1991']
pr_nums = ['21002', '21030', '22189', '22870', '22963']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41052'
versions = ['Python 3.9', 'Python 3.10']

[corona10]

See https://bugs.python.org/issue40077#msg371813

We noticed that heap type has different behavior about serialization/deserialization.

Basically it can occur the regression issues.

Two things needed.

1. opt out serialization/deserialization for converted modules.
2. Add unit tests to check whether their serialization is blocked.
3. If the module is already ported to 3.9 backport patch is needed.
   Long term

• Add the object.reduce() and/or update pickle can be smarter

[corona10]

New changeset 6989af0 by Dong-hee Na in branch 'master':
bpo-41052: Opt out serialization/deserialization for _random.Random (GH-21002)
6989af0

[corona10]

New changeset 814b07b by Dong-hee Na in branch '3.9':
[3.9] bpo-41052: Opt out serialization/deserialization for _random.Random (GH-21002). (GH-21030)
814b07b

[serhiy-storchaka]

There are other heap types implemented in C in the stdlib and third-party libraries for which pickling with protocols 0 and 1 works incorrectly. It would be better to fix copyreg._reduce_ex() instead of disabling pickling for these types one by one.

From PEP-307:

Let D be the class on the object to be pickled. First, find the nearest base class that is implemented in C (either as a built-in type or as a type defined by an extension class). Call this base class B, and the class of the object to be pickled D. Unless B is the class 'object', instances of class B must be picklable, either by having built-in support (as defined in the above three bullet points), or by having a non-default __reduce__ implementation. B must not be the same class as D (if it were, it would mean that D is not implemented in Python).

The problem is with determining which class is implemented in C. The current code implies that heap types are implemented in Python, and static types are implemented in C. It is not always true, because some heap types can be implemented in C.

[corona10]

It would be better to fix copyreg._reduce_ex() instead of disabling pickling for these types one by one

I agree :)

[pitrou]

At some point, pickle protocols 0 and 1 should be deprecated and slated for removal. Protocol 2 is 17 years ago already, and protocol 3 has become the default in Python 3.

That would probably be a better use of contributor time than trying to make heap types compatible with those obsolete protocols.

[serhiy-storchaka]

I came to the same conclusion after trying to fix it. But we cannot just do it now. We have to fix bugs in Python 3.9 and support protocols 0 and 1 for some deprecation period.

Also protocols 0 and 1 are used in some third-party implementations for other programming languages, so they can be used for interoperability with other languages.

The proposed patch adds additional check. It uses the fact that the __new__ attribute for classes implemented in C is a bultin function with __self__ pointing to this class (it is created in tp_new_wrapper()).