Additionally, many Python developers using Unix forget that the default encoding is platform dependent. They omit to specify encoding="utf-8"

Python Enhancement Proposals

submited by

Style Pass

2024-04-26 12:30:03

Additionally, many Python developers using Unix forget that the default encoding is platform dependent. They omit to specify encoding="utf-8" when they read text files encoded in UTF-8 (e.g. JSON, TOML, Markdown, and Python source files). Inconsistent default encoding causes many bugs.

Users can still disable UTF-8 mode by setting PYTHONUTF8=0 or -X utf8=0.

Since UTF-8 mode affects locale.getpreferredencoding(False), we need an API to get locale encoding regardless of UTF-8 mode.

locale.getencoding() will be added for this purpose. It returns locale encoding too, but ignores UTF-8 mode.

When warn_default_encoding option is specified, locale.getpreferredencoding() will emit EncodingWarning like open() (see also PEP 597).

PEP 597 added the encoding="locale" option to the TextIOWrapper. This option is used to specify the locale encoding explicitly. TextIOWrapper should use locale encoding when the option is specified, regardless of default text encoding.