[C API] Deprecate Global Configuration Variables like Py_IgnoreEnvironment

[vstinner]

PEP 587 "Python Initialization Configuration" got implemented in Python 3.8: https://peps.python.org/pep-0587/ It adds a new PyConfig API which is more complete and reliable than the legacy API to initialize Python: Py_Initialize() with a scattered collection of global variables and some "Set" functions like Py_SetPath().

In Python 3.11, I deprecated functions like Py_SetPath() and Py_SetStandardStreamEncoding(): https://docs.python.org/dev/c-api/init.html#c.Py_SetPath

I now propose to deprecate global configuration variables like Py_IgnoreEnvironment: PyConfig members should be used instead.

PR: #93943 merged as commit 0ff626f

[malemburg]

@vstinner: I don't know where this deprecation process has been discussed, but I didn't see it.

While it does make sense, given that we have the PyConfig structure now, the PRs you put in place literally skip the deprecation process: e.g. code setting one of the Py_XYZFlag variables will no longer work after the PyConfig struct has been set up, since the PRs redirect the lookup to PyConfig and completely ignore the global flags.

A better way would be to query both values and use the global var in case it is set. The gobal vars could be set to -1 during init to make this possible.

The global vars could then then be removed in a later release.

[vstinner]

skip the deprecation process

It seems like there is misunderstanding. This issue is only about marking global configuration variables as deprecated, it should not change the behavior of Python. This issue is not about scheduling the removal of these variables. That will be discussed later.

I don't know where this deprecation process has been discussed

It's discussed here.

When I implemented PEP 587 PyConfig, I spent a significant time on writing unit tests (Lib/test/test_embed.py and Programs/_testembed.c) to test that using the old way (like global config vars) still works. You might want to look at Python/preconfig.c and Python/initconfig.c to see the details.

A better way would be to query both values and use the global var in case it is set.

That's basically how it works. PyConfig_InitPythonConfig() is backward compatible with Python 3.7, whereas PyConfig_InitIsolatedConfig() ignores global config vars.

[vstinner]

test_init_global_config is a good example of unit test for the feature.

[malemburg]

I'll have a look, thanks.

Note that these flags can be adjusted after the interpreter has been initialized, e.g. to turn on optimization or increase verbosity, so I doubt that the logic as-is will work by just having PyConfig_InitPython*Config() pick up the current global flag values.

[malemburg]

Here's the analysis from reading the code:

Python/initconfig.c:

- config_get_global_vars() copies the global vars into the PyConfig struct, but only if config->_config_init != _PyConfig_INIT_COMPAT
  - this appears to be the only connection between the global vars and PyConfig
- it is called by _PyConfig_Read(), which is called by PyConfig_Read()

Programs/_bootstrap_python.c:

- only needed for building deep frozen modules
- calls PyConfig_Read() as part of main() during interpreter startup
  - this appears to duplicate the config read done later by PyConfig_Read()
- calls Py_InitializeFromConfig() as part of main() during interpreter startup

Modules/main.c:

- calls Py_InitializeFromConfig() as part of pymain_init()
- pymain_init() is (eventually) called by main() in Programs/python.c

Python/frozenmain.c:

- only needed for building frozen apps
- calls Py_InitializeFromConfig() as part of Py_FrozenMain() during interpreter startup of frozen apps

Python/pylifecycle.c:

- calls _PyConfig_Read() as part of _PyInterpreterState_SetConfig()
  - _PyInterpreterState_SetConfig() is only used by tests
- calls _PyConfig_Read() as part of pyinit_core()
- pyinit_core() is part of Py_InitializeFromConfig()

Python/pystate.c:

- _Py_GetConfig() is the helper used to access the PyConfig struct
  - this does not recheck the global vars

I may have missed some logic, but from the above, it seems that PyConfig is indeed only initialized from the global vars during interpreter startup.

Subsequent changes to the global flags are not taken into account by the various calls to _Py_GetConfig(), so any changes after interpreter init are ignored. This effectively causes the flags to be disabled. The only part which probably still works is when embedding the interpreter and setting up these global vars before initializing the interpreter.

To me, this looks like the deprecation has been skipped :-) and since this is not intended, I would suggest that the global flags are not replaced everywhere with calls to _Py_GetConfig(), but instead queried to check whether they have been altered first and only then have _Py_GetConfig() override them in case they have not been set.

This can be had by initializing the global vars to -1 (they are currently set to 0). If they are -1, _Py_GetConfig() is used. Otherwise the global var value is used (and perhaps a deprecation warning raised).

[malemburg]

Related to this, the other way around also doesn't appear to work, i.e. setting PyConfig.parser_debug doesn't result in Py_DebugFlag to be set.

In this particular case, the PyConfig member also appears to be misnamed. The debug flag happens to only be used in the parser (at the moment), but its global flag name certainly has a more general meaning than "parser_debug" and can be used for more than just the Python parser, e.g. to enable debugging code in Python extensions.

[malemburg]

Looking at the PyConfig API, it seems that an API for setting integer PyConfig flags is missing. Is that on purpose, or an oversight (since there are APIs for setting strings) ?

Also, if I read the PEP 587 and code correctly, we're losing the possibility to e.g. change the verbosity or optimize flags from a C extension in a running Python interpreter. PEP 587 only appears to address config changes prior to starting the interpreter. Is that intended ?

For PyRun I will have to find a way around these limitations, since it's using Python to setup some of these flags. Previously, I could modify the global vars directly, but now I will have to go deep into the internals via _PyInterpreterState_GetConfig() and then directly set the values in the config struct. Updating sys.flags won't be possible, since make_flags() from Python/sysmodule.c is not available as public API.

[malemburg]

I also could not find a public PyConfig API for reading the current config. Why is that ?

There's lots of useful information in the PyConfig struct, which extensions could use as well and won't be available elsewhere anymore once the global vars are gone.

[vstinner]

The PyConfig API is designed to initialize Python. Once Python is initialized, it's not expected to be modified, whereas some variables initialized from PyConfig can be modified on purpose. For example, you should not modify PyConfig.module_search_paths, but sys.path. Once Python is initialized, modifying PyConfig.module_search_paths has no effect. Well, PyConfig has many members, maybe some members should be treated differently.

I also could not find a public PyConfig API for reading the current config. Why is that ?

So far, nobody asked for the feature. If you want to access PyConfig, please open an issue for that.

There's lots of useful information in the PyConfig struct, which extensions could use as well and won't be available elsewhere anymore once the global vars are gone.

Which members are useful? As I wrote, many PyConfig members are copied somewhere else. For example, sys.flags contains a few of them.

Related to this, the other way around also doesn't appear to work, i.e. setting PyConfig.parser_debug doesn't result in Py_DebugFlag to be set.

With PyConfig_InitPythonConfig(), Py_DebugFlag is used to initialize PyConfig.parser_debug. Once Python is initialized, global configuration variables are set from their related PyConfig members. After Python is initialized, global configuration variables are no longer used.

A few global config variables are still used, and I'm trying to fix that for consistency. The remaining code is the most complicated to fix. But most global configuration variables are ignored since Python 3.8.

That's why I would like to deprecate them. Better communicate that they are no longer used.

Also, if I read the PEP 587 and code correctly, we're losing the possibility to e.g. change the verbosity or optimize flags from a C extension in a running Python interpreter. PEP 587 only appears to address config changes prior to starting the interpreter. Is that intended ?

Again, there is no public C API to modify the configuration once Python is initialized. So far, nobody requested this feature.

To change the optimize flags, use the optimize parameter of the compile() functions and functions of py_compile and compileall modules.

Technically, I implemented _testinternalcapi.get_config() and _testinternalcapi.set_config() which can be used to get and set any PyConfig member. But it's an internal C API, since there was no request to have a public API.

now I will have to go deep into the internals via _PyInterpreterState_GetConfig() and then directly set the values in the config struct. Updating sys.flags won't be possible, since make_flags() from Python/sysmodule.c is not available as public API.

I don't think that it was possible to modify sys.flags previously. But there were ways to have an inconsistent configuration :-) When I implemented PyConfig, I also tried to prevent inconsistencies.

Technically, _testinternalcapi.set_config() does update sys.flags, so it's technically possible to modify sys.flags.bytes_warning for example.

$ ./python
Python 3.12.0a0 (heads/is_type_signed:1bebc6416e, May 25 2022, 22:10:52) [GCC 12.1.1 20220507 (Red Hat 12.1.1-1)] on linux
>>> import _testinternalcapi
>>> import sys
>>> sys.flags.bytes_warning
0

>>> config=_testinternalcapi.get_config()
>>> config['bytes_warning']
0

>>> config['bytes_warning']=1
>>> config=_testinternalcapi.set_config(config)
>>> sys.flags.bytes_warning
1

[vstinner]

Implemented in #93943 I close the issue.

[malemburg]

@vstinner I'm sorry, but you have not addressed my concerns. The changes you are making are not deprecating global variables. Instead you are disabling their use altogether without any deprecation in 3.12. This is against our published deprecation policy.

Global Python variables have been in use for more than two decades and it was always possible to change their values at runtime, not only before initializing the interpreter. This functionality is now gone and it's not possible to make those changes via the PyConfig API. Not only is that API not providing ways to change the values of e.g. the debug, optimize or verbose flags (to name a few), the design doesn't even consider that runtime changes are possible after initialization.

The fact that no one asked for any such APIs is merely because no one was aware of the consequences. The PyConfig API is a good one, but in order to replace the current global variables, it has to provide adequate replacements for allowing post-init changes to the configuration.

This makes it impossible to tweak the interpreter global settings by tools which use the frozen modules, since those do not use the C runtime startup system and have to set whatever settings are needed in Python after PyConfig initialization.

Additionally, Python C extensions can no longer adjust those parameters at runtime to e.g. force use of optimized byte code, raise verbosity to enable debugging imports or make use of the debug flag for their own debugging purposes (to name a few use cases).

Please reconsider and discuss this change more openly, e.g. on Discourse. Thanks.

[vstinner]

Global Python variables have been in use for more than two decades and it was always possible to change their values at runtime, not only before initializing the interpreter.

You're making assumptions which are no longer true. For example, Py_OptimizeFlag is no longer used since Python 3.8 released in 2019. It's not a recent change. PEP 587 (PyConfig) was implemented in Python 3.8. Moreover, there are (C API and Python) functions with an "optimize" parameter to choose the optimization level, there is no need to set it globally.

Modifying global configuration variables directly is bad since many values are only used to initialize Python, but then copied somewhere else. For example, Py_DontWriteBytecodeFlag is used to initialize sys.flags.dont_write_bytecode. The internal _PyInterpreterState_SetConfig() function fix this problem: not only it updates PyConfig of the interpreter, but it also updates sys.flags. There is no public API since as I already wrote, before you, nobody asked for the feature.

If you want to modify a specific parameter, you should come up with a specific use case and request a public API for that. I am not convinced that we need a generic function to change "any" configuration value. The API is too wide to give a generic solution. Just an example, modifying PyConfig.module_search_paths once Python is initialized is useless, since it's copied to sys.path and then import uses sys.path. It no longer uses PyConfig.

The change here is more about advertizing that: hey, in Python 3.12, you should no longer use global configuration variables, but the PyConfig API to initialize Python.

I closed the issue, again, if you consider that a new API is needed, please open a separated issue.

[malemburg]

On 20.06.2022 17:02, Victor Stinner wrote: Global Python variables have been in use for more than two decades and it was always possible to change their values at runtime, not only before initializing the interpreter. You're making assumptions which are no longer true. For example, Py_OptimizeFlag is no longer used since Python 3.8 released in 2019. It's not a recent change. PEP 587 (PyConfig) was implemented in Python 3.8.

Which only underlines that the change to disable changing global vars after initialization was probably not well communicated; otherwise, I would have spoken up earlier. The PEP only talks about "PyInterpreterState.config becomes the new reference configuration, replacing global configuration variables and other private variables." The PEP focuses on initialization, which happens once at startup. It doesn't mention removing the global variables or having code no longer use them at runtime. It's still a good idea to proceed in that direction, but the public C API will have to provide an alternative way to achieve the same effect as changing the global vars at runtime. E.g. if I want the import logic to start looking for and loading optimized code, there has to be a way to modify sys.flags.optimize (which is used by the import logic).

Moreover, there are (C API and Python) functions with an "optimize" parameter to choose the optimization level, there is no need to set it globally.

Not for explicitly compiling code in optimized mode, but there is a need to tell the import logic to look for optimized code instead of regular byte code. The setting is also needed to set the default for compiling imported source files, which are implicitly compiled.

Modifying global configuration variables directly is bad since many values are only used to initialize Python, but then copied somewhere else. For example, |Py_DontWriteBytecodeFlag| is used to initialize sys.flags.dont_write_bytecode. The internal |_PyInterpreterState_SetConfig()| function fix this problem: not only it updates PyConfig of the interpreter, but it also updates |sys.flags|. There is no public API since as I already wrote, before you, nobody asked for the feature.

Great, so we should make that a public API. I guess the right way to change a value would then be to first get the current config using _Py_GetConfig(), change the setting and then pass the config to _PyInterpreterState_SetConfig() to enable it.

If you want to modify a specific parameter, you should come up with a specific use case and request a public API for that. I am not convinced that we need a generic function to change "any" configuration value. The API is too wide to give a generic solution. Just an example, modifying |PyConfig.module_search_paths| once Python is initialized is useless, since it's copied to sys.path and then |import| uses |sys.path|. It no longer uses |PyConfig|.

I don't think having an API per setting is a good approach. It's better to expose the PyConfig struct and allow extensions to make direct changes via the above APIs.

The change here is more about advertizing that: hey, in Python 3.12, you should no longer use global configuration variables, but the PyConfig API to initialize Python. I closed the issue, again, if you consider that a /new/ API is needed, please open a separated issue.

Before opening up new issues, I'd like to get clear on a direction. We can continue here or open a discussion on Discourse (which I'd prefer, since that's read by more people than discussions on tickets).

…

-- Marc-Andre Lemburg http://www.malemburg.com/

[encukou]

It seems like there is misunderstanding. This issue is only about marking global configuration variables as deprecated, it should not change the behavior of Python. This issue is not about scheduling the removal of these variables. That will be discussed later.

Was it ever discussed?

[vstinner]

Was it ever discussed?

Well, I wrote a whole PEP to address this issue :-) https://peps.python.org/pep-0741/

[malemburg]

PEP 741 doesn't include any wording towards deprecating and removing the configuration globals. PEP 587 doesn't either (it only lists how the globals are mapped to the PyConfig versions).

This issue is the only "discussion" of the deprecation that ever took place, AFAIK.

FWIW: For eGenix PyRun, I will have to hack the PyConfig implementation for Python 3.12+ to again allow changes to the config at runtime, since PyRun does the command line parsing and much of the configuration in Python and thus after PyConfig has already been finalized. PyRun used to use the Python config globals for this, since many of them could be changed at runtime. The PyConfig API does not permit doing this to my knowledge.

I'll also investigate replacing the Python initialization in PyRun with the standard C one, but would really like to avoid this, since having the Python initialization in Python is so much easier to maintain (much like importlib is much easier to maintain than the previous C implementation).

[vstinner]

This issue is closed, I suggest to continue the discussion at: https://discuss.python.org/t/pep-741-python-configuration-c-api-second-version/45403/77