From ecd5e7ba7226eb43b8e7f61b6bf7fc180baa6cb2 Mon Sep 17 00:00:00 2001 From: Cong Ma Date: Sat, 27 Mar 2021 16:46:58 +0200 Subject: [PATCH 1/3] bpo-43605: Improve the documentation to exec() and eval() - Add links to the relevant section in Language Reference about dynamic execution's limitation with respect to namespaces. - For eval(), move some explanatory text into a "Note" box. - Make the first paragraph of eval's doc consistent about what the first argument accepts. - For exec(), remove the text about the "globals" optional argument having to be a dict but not an instance of a subclass of dict. This is no longer true -- the code calls PyDict_Check(), not PyDict_CheckExact(). - Put quotes around the ``__builtins__`` in the text: clarify that in the context it means a string key in the dict passed to eval/exec as the globals dict. Otherwise, since the identifier __builtins__ refers to a module, it can be confusing and misleading. - Reordering some paragraphs so that overall the structure is improved. - Re-wrap some long lines in RST source. --- Doc/library/functions.rst | 133 ++++++++++++++++++++++---------------- 1 file changed, 76 insertions(+), 57 deletions(-) diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 5cb1df93702d68..4a8b22b68639ed 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -496,44 +496,38 @@ are always available. They are listed here in alphabetical order. .. function:: eval(expression[, globals[, locals]]) - The arguments are a string and optional globals and locals. If provided, - *globals* must be a dictionary. If provided, *locals* can be any mapping - object. - - The *expression* argument is parsed and evaluated as a Python expression - (technically speaking, a condition list) using the *globals* and *locals* - dictionaries as global and local namespace. If the *globals* dictionary is - present and does not contain a value for the key ``__builtins__``, a - reference to the dictionary of the built-in module :mod:`builtins` is - inserted under that key before *expression* is parsed. That way you can - control what builtins are available to the executed code by inserting your - own ``__builtins__`` dictionary into *globals* before passing it to - :func:`eval`. If the *locals* dictionary is omitted it defaults to the - *globals* dictionary. If both dictionaries are omitted, the expression is - executed with the *globals* and *locals* in the environment where - :func:`eval` is called. Note, *eval()* does not have access to the - :term:`nested scopes ` (non-locals) in the enclosing - environment. - - The return value is the result of - the evaluated expression. Syntax errors are reported as exceptions. Example: + This function supports the dynamic evaluation of Python expression. The + first argument can be a string or a code object. The optional arguments + specify the globals and locals respectively. If provided, *globals* must be + a dictionary. If provided, *locals* can be any mapping object. + + In the most common case, the *expression* argument is a string, and it is + parsed and evaluated as a Python expression (see the section + ":ref:`expression-input`" in the Language Reference). The leading and + trailing spaces, tabs, and newlines are stripped. + + The evaluation is performed using the *globals* and *locals* dictionaries as + the global and local namespaces. If the *globals* dictionary is present and + does not contain a value for the key ``"__builtins__"``, a reference to the + dictionary of the built-in module :mod:`builtins` is inserted under that key + before *expression* is parsed. That way, you can control what builtins are + available to the executed code by inserting your own ``"__builtins__"`` + dictionary into *globals* before passing it to :func:`eval`. If the + *locals* dictionary is omitted, it defaults to the *globals* dictionary. If + both dictionaries are omitted, the expression is executed with the globals + and locals in the environment where :func:`eval` is called. + + The return value is the result of the evaluated expression. Syntax errors + are reported as exceptions. For example: >>> x = 1 >>> eval('x+1') 2 - This function can also be used to execute arbitrary code objects (such as - those created by :func:`compile`). In this case pass a code object instead - of a string. If the code object has been compiled with ``'exec'`` as the - *mode* argument, :func:`eval`\'s return value will be ``None``. - - Hints: dynamic execution of statements is supported by the :func:`exec` - function. The :func:`globals` and :func:`locals` functions - returns the current global and local dictionary, respectively, which may be - useful to pass around for use by :func:`eval` or :func:`exec`. - - If the given source is a string, then leading and trailing spaces and tabs - are stripped. + The :func:`eval` function can also be used to execute code objects, such as + those created by :func:`compile`. In this case, pass a code object instead + of a string as the first argument. If the code object has been compiled + with ``'exec'`` as the *mode* argument, the return value will be ``None``. See :func:`ast.literal_eval` for a function that can safely evaluate strings with expressions containing only literals. @@ -543,36 +537,58 @@ are always available. They are listed here in alphabetical order. Raises an :ref:`auditing event ` ``exec`` with the code object as the argument. Code compilation events may also be raised. + .. _eval_limitation_dynamic: + .. note:: + + Dynamic evaluation at run-time is not equivalent to embedding the + expression in a Python program and having it compiled as a part of the + whole program. In particular, :func:`eval` does not have access to the + :term:`nested scopes ` (non-locals) in the enclosing + environment (see the section ":ref:`dynamic-features`" in the Language + Reference chapter on name binding). Of note are some expressions, such + as :term:`lambdas `, :ref:`comprehensions `, and + :term:`generator expressions `, which create an + inner scope of their own. The interaction between these expressions and + :func:`eval` can be explicitly controlled by the parameters *globals* and + *locals* in the aforementioned manner. + + The built-in functions :func:`globals` and :func:`locals` return the + current global and local dictionary, respectively, which may be useful to + pass around for use as the second and third argument to :func:`eval`. + + Dynamic execution of *statements* is supported by the :func:`exec` + function (see below). + + .. index:: builtin: exec .. function:: exec(object[, globals[, locals]]) This function supports dynamic execution of Python code. *object* must be - either a string or a code object. If it is a string, the string is parsed as - a suite of Python statements which is then executed (unless a syntax error - occurs). [#]_ If it is a code object, it is simply executed. In all cases, - the code that's executed is expected to be valid as file input (see the - section "File input" in the Reference Manual). Be aware that the - :keyword:`nonlocal`, :keyword:`yield`, and :keyword:`return` - statements may not be used outside of - function definitions even within the context of code passed to the - :func:`exec` function. The return value is ``None``. - - In all cases, if the optional parts are omitted, the code is executed in the - current scope. If only *globals* is provided, it must be a dictionary - (and not a subclass of dictionary), which - will be used for both the global and the local variables. If *globals* and - *locals* are given, they are used for the global and local variables, - respectively. If provided, *locals* can be any mapping object. Remember - that at module level, globals and locals are the same dictionary. If exec - gets two separate objects as *globals* and *locals*, the code will be - executed as if it were embedded in a class definition. + either a string or a code object. If it is a string, the string is parsed + as a suite of Python statements which is then executed (unless a syntax + error occurs). [#]_ If it is a code object, it is simply executed. In all + cases, the code that's executed is expected to be valid as file input (see + the section ":ref:`file-input`" in the Language Reference). Be aware that + the :keyword:`nonlocal`, :keyword:`yield`, and :keyword:`return` statements + may not be used outside of function definitions even within the context of + code passed to the :func:`exec` function. The return value is ``None``. + + In all cases, if the optional arguments are omitted, the code is executed in + the current scope. If only *globals* is provided, it must be a dictionary, + which will be used for both the global and the local variables. If + *globals* and *locals* are given, they are used for the global and local + variables, respectively. If provided, *locals* can be any mapping object. + Remember that at module level, globals and locals are the same dictionary. + If exec gets two separate objects as *globals* and *locals*, the code will + be executed as if it were embedded in a class definition. If the *globals* dictionary does not contain a value for the key - ``__builtins__``, a reference to the dictionary of the built-in module + ``"__builtins__"``, a reference to the dictionary of the built-in module :mod:`builtins` is inserted under that key. That way you can control what builtins are available to the executed code by inserting your own - ``__builtins__`` dictionary into *globals* before passing it to :func:`exec`. + ``"__builtins__"`` dictionary into *globals* before passing it to + :func:`exec`. .. audit-event:: exec code_object exec @@ -581,9 +597,12 @@ are always available. They are listed here in alphabetical order. .. note:: - The built-in functions :func:`globals` and :func:`locals` return the current - global and local dictionary, respectively, which may be useful to pass around - for use as the second and third argument to :func:`exec`. + Like :func:`eval`, :func:`exec` is :ref:`limited by constraints + particular to dynamic code execution and namespaces + `. The built-in functions :func:`globals` and + :func:`locals` return the current global and local dictionary, + respectively, which may be useful to pass around for use as the second + and third argument to :func:`exec`. .. note:: From 4ffc9cdc0ea9cfe0220f8dd1b556244a730511b9 Mon Sep 17 00:00:00 2001 From: Cong Ma Date: Mon, 29 Mar 2021 10:16:17 +0200 Subject: [PATCH 2/3] Further revise eval/exec documentations for consistency and accuracy. The documentations for eval/exec are further modified for consistency and accuracy. - For both functions, the documentation mostly follows the following structure: description of arguments, default values for optional arguments, special treatment of the "__builtins__" key, and notices about limitations or other noteworthy material. Previously, the description about the special "__builtins__" key is intermingled with the description about default values for missing arguments. Now, these are in separate paragraphs. Also, it is stressed that the special key is inserted in-place. - In the notice about limitations to dynamic eval/exec w.r.t. scopes, also mention the assignment expression succinctly. - Remove the description that exec() with both globals and locals behave "as if" the source string is embedded in a class block. Again, a class block inside a function can access the non-locals, but this is not available to exec(). - Instead of mentioning passing the return values of globals() and locals() around, we now say that the programmer can construct their own namespaces based on their return values. The reason is that currently the semantics of locals() is not very well defined, and in the light of PEP 558 it may change in the future. Explicitly constructing new namespaces is more predictable than using the "raw" return value of locals(). In the future, this part of the docs can be further updated to keep up with the new definition of locals(). - Omit the footnote to exec() about the parser only accepting Unix-style LF newlines. This piece of information is outdated. - Other small fixes for grammar. --- Doc/library/functions.rst | 127 +++++++++++++++++++------------------- 1 file changed, 63 insertions(+), 64 deletions(-) diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 4a8b22b68639ed..0a308c809123a7 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -496,26 +496,29 @@ are always available. They are listed here in alphabetical order. .. function:: eval(expression[, globals[, locals]]) - This function supports the dynamic evaluation of Python expression. The + This function supports the dynamic evaluation of Python expressions. The first argument can be a string or a code object. The optional arguments - specify the globals and locals respectively. If provided, *globals* must be - a dictionary. If provided, *locals* can be any mapping object. + specify the global and local namespaces respectively. If provided, + *globals* must be a dictionary. If provided, *locals* can be any + :term:`mapping` object. In the most common case, the *expression* argument is a string, and it is parsed and evaluated as a Python expression (see the section ":ref:`expression-input`" in the Language Reference). The leading and trailing spaces, tabs, and newlines are stripped. - The evaluation is performed using the *globals* and *locals* dictionaries as - the global and local namespaces. If the *globals* dictionary is present and - does not contain a value for the key ``"__builtins__"``, a reference to the - dictionary of the built-in module :mod:`builtins` is inserted under that key - before *expression* is parsed. That way, you can control what builtins are - available to the executed code by inserting your own ``"__builtins__"`` - dictionary into *globals* before passing it to :func:`eval`. If the - *locals* dictionary is omitted, it defaults to the *globals* dictionary. If - both dictionaries are omitted, the expression is executed with the globals - and locals in the environment where :func:`eval` is called. + The evaluation is performed in the environment specified by the arguments + *globals* and *locals*. If both are omitted, by default it uses the + environment where :func`eval` is called. If only the *globals* argument is + given, the local namespace defaults to *globals*. + + Before evaluation, the special key ``"__builtins__"`` is searched for in the + global namespace that is explicitly or impcitly specified for :func:`eval`. + If this key is not present, by default a reference to the dictionary of the + built-in module :mod:`builtins` is inserted *in-place* under that key, so + that built-in identifiers resolve to their usual built-in implementations. + By overriding the value for the key ``"__builtins__"`` in *globals*, you + can control what builtins are available to the expression being evaluated. The return value is the result of the evaluated expression. Syntax errors are reported as exceptions. For example: @@ -541,20 +544,24 @@ are always available. They are listed here in alphabetical order. .. note:: Dynamic evaluation at run-time is not equivalent to embedding the - expression in a Python program and having it compiled as a part of the - whole program. In particular, :func:`eval` does not have access to the - :term:`nested scopes ` (non-locals) in the enclosing - environment (see the section ":ref:`dynamic-features`" in the Language - Reference chapter on name binding). Of note are some expressions, such - as :term:`lambdas `, :ref:`comprehensions `, and - :term:`generator expressions `, which create an - inner scope of their own. The interaction between these expressions and - :func:`eval` can be explicitly controlled by the parameters *globals* and - *locals* in the aforementioned manner. + expression at the same place in a Python program and having it compiled + as a part of the whole program; this is explained in the section + ":ref:`dynamic-features`" in the Language Reference. + + In particular, :func:`eval` does not have access to the :term:`nested + scopes ` (non-locals) in the enclosing environment. Of + note are expressions such as :term:`lambdas `, + :ref:`comprehensions `, and :term:`generator expressions + ` which create an inner scope of their own. Since + Python 3.8, the action of an assignment expression (see :pep:`572`) also + depends on scope information determined at compile-time. Interaction + between these expressions and :func:`eval` can be explicitly controlled + by the arguments *globals* and *locals* in the aforementioned manner. The built-in functions :func:`globals` and :func:`locals` return the - current global and local dictionary, respectively, which may be useful to - pass around for use as the second and third argument to :func:`eval`. + current global and local dictionaries respectively, which may be useful + for constructing namespaces passed as the second and third arguments to + :func:`eval`. Dynamic execution of *statements* is supported by the :func:`exec` function (see below). @@ -564,31 +571,29 @@ are always available. They are listed here in alphabetical order. .. function:: exec(object[, globals[, locals]]) - This function supports dynamic execution of Python code. *object* must be - either a string or a code object. If it is a string, the string is parsed - as a suite of Python statements which is then executed (unless a syntax - error occurs). [#]_ If it is a code object, it is simply executed. In all - cases, the code that's executed is expected to be valid as file input (see - the section ":ref:`file-input`" in the Language Reference). Be aware that - the :keyword:`nonlocal`, :keyword:`yield`, and :keyword:`return` statements - may not be used outside of function definitions even within the context of - code passed to the :func:`exec` function. The return value is ``None``. - - In all cases, if the optional arguments are omitted, the code is executed in - the current scope. If only *globals* is provided, it must be a dictionary, - which will be used for both the global and the local variables. If - *globals* and *locals* are given, they are used for the global and local - variables, respectively. If provided, *locals* can be any mapping object. - Remember that at module level, globals and locals are the same dictionary. - If exec gets two separate objects as *globals* and *locals*, the code will - be executed as if it were embedded in a class definition. - - If the *globals* dictionary does not contain a value for the key - ``"__builtins__"``, a reference to the dictionary of the built-in module - :mod:`builtins` is inserted under that key. That way you can control what - builtins are available to the executed code by inserting your own - ``"__builtins__"`` dictionary into *globals* before passing it to - :func:`exec`. + This function supports the dynamic execution of Python code. *object* must + be either a string or a code object. If it is a string, the string is + parsed as a suite of Python statements which is then executed (unless a + syntax error occurs). If it is a code object, it is simply executed. In + all cases, the code being executed is expected to be valid as file input + (see the section ":ref:`file-input`" in the Language Reference). The return + value of :func:`exec` is ``None``. + + If provided, *globals* must be a dictionary. If provided, *locals* can be + any mapping object. + + The execution is performed in the environment specified by the arguments + *globals* and *locals*. If both are omitted, by default it uses the + environment where :func`exec` is called. If only the *globals* argument is + given, the local namespace defaults to *globals*. + + Before execution, the special key ``"__builtins__"`` is searched for in the + global namespace that is explicitly or impcitly specified for :func:`exec`. + If this key is not present, by default a reference to the dictionary of the + built-in module :mod:`builtins` is inserted *in-place* under that key, so + that built-in identifiers resolve to their usual built-in implementations. + By overriding the value for the key ``"__builtins__"`` in *globals*, you + can control what builtins are available to the code being executed. .. audit-event:: exec code_object exec @@ -597,19 +602,19 @@ are always available. They are listed here in alphabetical order. .. note:: - Like :func:`eval`, :func:`exec` is :ref:`limited by constraints - particular to dynamic code execution and namespaces - `. The built-in functions :func:`globals` and - :func:`locals` return the current global and local dictionary, - respectively, which may be useful to pass around for use as the second - and third argument to :func:`exec`. + Like :func:`eval`, :func:`exec` is limited by constraints particular to + :ref:`dynamic code execution and namespaces `. + The built-in functions :func:`globals` and :func:`locals` return the + current global and local dictionaries respectively, which may be useful + for constructing namespaces passed as the second and third arguments to + :func:`exec`. .. note:: The default *locals* act as described for function :func:`locals` below: modifications to the default *locals* dictionary should not be attempted. - Pass an explicit *locals* dictionary if you need to see effects of the - code on *locals* after function :func:`exec` returns. + You can explicitly pass a distinct *locals* dictionary to :func:`exec` to + observe the effects of executing the code on the local namespace. .. function:: filter(function, iterable) @@ -1974,9 +1979,3 @@ are always available. They are listed here in alphabetical order. .. versionchanged:: 3.9 When the command line options :option:`-E` or :option:`-I` are being used, the environment variable :envvar:`PYTHONCASEOK` is now ignored. - -.. rubric:: Footnotes - -.. [#] Note that the parser only accepts the Unix-style end of line convention. - If you are reading the code from a file, make sure to use newline conversion - mode to convert Windows or Mac-style newlines. From 6504b9cb12ad8c65c6a6770aa3e0611af0701eb9 Mon Sep 17 00:00:00 2001 From: Cong Ma Date: Mon, 29 Mar 2021 13:20:39 +0200 Subject: [PATCH 3/3] Fix invalid RST syntax. --- Doc/library/functions.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 0a308c809123a7..514b3fad91ff1a 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -509,7 +509,7 @@ are always available. They are listed here in alphabetical order. The evaluation is performed in the environment specified by the arguments *globals* and *locals*. If both are omitted, by default it uses the - environment where :func`eval` is called. If only the *globals* argument is + environment where :func:`eval` is called. If only the *globals* argument is given, the local namespace defaults to *globals*. Before evaluation, the special key ``"__builtins__"`` is searched for in the @@ -584,7 +584,7 @@ are always available. They are listed here in alphabetical order. The execution is performed in the environment specified by the arguments *globals* and *locals*. If both are omitted, by default it uses the - environment where :func`exec` is called. If only the *globals* argument is + environment where :func:`exec` is called. If only the *globals* argument is given, the local namespace defaults to *globals*. Before execution, the special key ``"__builtins__"`` is searched for in the