UTF-8 by Default: What PEP 686 Changes in Python 3.15

Authors

K19G

grok-bot

Updated

October 31, 2026

UTF-8 by Default: What PEP 686 Changes in Python 3.15

Thesis

Until 3.14, open("notes.txt") meant “open in whatever encoding this machine’s locale says”. On most Linux and macOS systems that happened to be UTF-8. On Windows it was the ANSI code page (cp1252 in Western Europe), and on an old server it might be ISO-8859-1. Python 3.15 makes UTF-8 mode the default (PEP 686, Final). With no encoding= argument, text files, pipes, and the standard streams now use UTF-8 everywhere, whatever the locale. That removes a whole class of “works on my machine” bugs. It also moves some bugs around. Files written by older jobs in the locale encoding no longer decode. Output of locale-speaking subprocesses no longer decodes. And because the default now depends on the interpreter version, a library that supports 3.10–3.15 still produces different bytes on different interpreters. The fix in every case is the same: say which encoding you mean.

Version note: every listing below ran on CPython 3.15.0 (final, 2026-10-09) and CPython 3.14.8 on Linux x86-64, and the outputs are copied from those runs. Linux with a UTF-8 locale barely notices this change, so the lab builds a legacy ISO-8859-1 locale to reproduce what a Windows machine with a non-UTF-8 code page sees. The mechanism is the same one Windows uses: the locale encoding is not UTF-8. The Windows-specific notes are from the 3.15 documentation and were not run here.

Mental model

                              3.14 (locale mode)        3.15 (UTF-8 mode, default)
open(path)                    locale encoding           utf-8
subprocess.run(text=True)     locale encoding           utf-8
sys.stdin / sys.stdout        locale, strict            utf-8, surrogateescape
sys.getfilesystemencoding()   locale (Unix)             utf-8
locale.getpreferredencoding() locale                    'utf-8' (always)
locale.getencoding()          locale                    locale   ◀── unchanged: the real locale
open(path, encoding="locale") locale                    locale   ◀── unchanged: explicit opt-in
open(path, encoding="utf-8")  utf-8                     utf-8    ◀── unchanged: always correct

turn it off (per process, at startup only):  PYTHONUTF8=0   or   python -X utf8=0
find every implicit use:                      python -X warn_default_encoding  (EncodingWarning)

UTF-8 mode is not new. It has existed since 3.7 (PEP 540), and Python already switched it on automatically when the locale was C or POSIX. What 3.15 changes is the default. Everything an explicit encoding= argument says keeps meaning exactly what it said. Only the “I didn’t say” path moved.

Worked examples

The lab needs two interpreters and a legacy locale. localedef compiles locales from glibc’s sources (on Debian and Ubuntu they come with the locales package), and LOCPATH points glibc at a private directory, so no root access is needed:

$ mkdir -p loc
$ localedef -i en_US -f UTF-8      loc/en_US.utf8
$ localedef -i en_US -f ISO-8859-1 loc/en_US.iso88591
$ localedef -i de_AT -f ISO-8859-1 loc/de_AT.iso88591
$ export LOCPATH=$PWD/loc
$ clean() { env -i PATH="$PATH" LOCPATH="$LOCPATH" "$@"; }   # run with only the variables we name

clean starts each run from an empty environment, so nothing in your shell (LC_ALL, PYTHONUTF8, PYTHONIOENCODING) can leak in. Absolute paths in tracebacks and warnings are shortened to ... below.

Case 1: What each default actually picks

# probe.py — which encoding does each default actually pick?
import locale
import sys

print(f"Python {sys.version.split()[0]}  utf8_mode={sys.flags.utf8_mode}")
print(f"  locale.getencoding()           {locale.getencoding()}")
print(f"  locale.getpreferredencoding()  {locale.getpreferredencoding()}")
print(f"  sys.getfilesystemencoding()    {sys.getfilesystemencoding()}")
print(f"  sys.stdout                     {sys.stdout.encoding} / {sys.stdout.errors}")
with open("probe.txt", "w") as f:
    print(f"  open('probe.txt', 'w')         {f.encoding}")
with open("probe.txt", "w", encoding="locale") as f:
    print(f"  open(..., encoding='locale')   {f.encoding}")

On a UTF-8 Linux machine, the encodings are the same between versions. One error handler is not:

$ clean LANG=en_US.UTF-8 python3.14 probe.py
Python 3.14.8  utf8_mode=0
  locale.getencoding()           UTF-8
  locale.getpreferredencoding()  UTF-8
  sys.getfilesystemencoding()    utf-8
  sys.stdout                     utf-8 / strict
  open('probe.txt', 'w')         UTF-8
  open(..., encoding='locale')   UTF-8
$ clean LANG=en_US.UTF-8 python3.15 probe.py
Python 3.15.0  utf8_mode=1
  locale.getencoding()           UTF-8
  locale.getpreferredencoding()  utf-8
  sys.getfilesystemencoding()    utf-8
  sys.stdout                     utf-8 / surrogateescape
  open('probe.txt', 'w')         utf-8
  open(..., encoding='locale')   UTF-8

sys.stdout now uses surrogateescape instead of strict. Keep that in mind for the trap. Now the legacy locale, which is where the real change shows:

$ clean LANG=en_US.ISO-8859-1 python3.14 probe.py
Python 3.14.8  utf8_mode=0
  locale.getencoding()           ISO-8859-1
  locale.getpreferredencoding()  ISO-8859-1
  sys.getfilesystemencoding()    iso8859-1
  sys.stdout                     iso8859-1 / strict
  open('probe.txt', 'w')         ISO-8859-1
  open(..., encoding='locale')   ISO-8859-1
$ clean LANG=en_US.ISO-8859-1 python3.15 probe.py
Python 3.15.0  utf8_mode=1
  locale.getencoding()           ISO-8859-1
  locale.getpreferredencoding()  utf-8
  sys.getfilesystemencoding()    utf-8
  sys.stdout                     utf-8 / surrogateescape
  open('probe.txt', 'w')         utf-8
  open(..., encoding='locale')   ISO-8859-1

Same machine, same environment. 3.14 follows the locale everywhere. 3.15 uses UTF-8 everywhere except the two APIs whose job is to report the locale: locale.getencoding() and encoding="locale". locale.getpreferredencoding() is not one of them. In UTF-8 mode it always answers 'utf-8', so code that asks it “what is the system encoding?” gets the wrong answer.

On Windows, the 3.15 docs list one difference from this Linux picture. Console I/O and the filesystem encoding were already UTF-8 there before 3.15 (PEPs 528 and 529), so the change on Windows lands on open(), pipes and subprocess, which used the ANSI code page. That code page is still reachable when you need it, as encoding="locale" or the "mbcs" codec.

Case 2: A file from last year’s job no longer decodes

# export_report.py — a nightly job that never said which encoding it meant
rows = [("Wien", 12), ("Graz", 11), ("Klagenfurt am Wörthersee", 9)]
with open("report.csv", "w") as f:
    f.write("city;temp_c\n")
    for city, temp in rows:
        f.write(f"{city};{temp}\n")
# read_report.py — the consumer, also without an encoding
with open("report.csv") as f:
    for line in f:
        print(line.rstrip())

The exporter ran on 3.14 for years. The reader is upgraded to 3.15 first:

$ clean LANG=en_US.ISO-8859-1 python3.14 export_report.py
$ grep -a Klagenfurt report.csv | od -An -tx1
 4b 6c 61 67 65 6e 66 75 72 74 20 61 6d 20 57 f6
 72 74 68 65 72 73 65 65 3b 39 0a
$ clean LANG=en_US.ISO-8859-1 python3.15 read_report.py
Traceback (most recent call last):
  File ".../read_report.py", line 3, in <module>
    for line in f:
                ^
  File "<frozen codecs>", line 325, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 43: invalid start byte

ö was written as the single byte f6, which is ISO-8859-1. In UTF-8 that byte can only appear inside a multi-byte sequence, so the reader stops. That is the good direction, because it fails loudly. Upgrade the other side first and the failure is silent:

$ clean LANG=en_US.ISO-8859-1 python3.15 export_report.py
$ grep -a Klagenfurt report.csv | od -An -tx1
 4b 6c 61 67 65 6e 66 75 72 74 20 61 6d 20 57 c3
 b6 72 74 68 65 72 73 65 65 3b 39 0a
$ clean LANG=en_US.ISO-8859-1 python3.14 read_report.py | iconv -f ISO-8859-1 -t UTF-8
city;temp_c
Wien;12
Graz;11
Klagenfurt am Wörthersee;9

Now ö is the two bytes c3 b6. ISO-8859-1 decodes any byte, so 3.14 reads them as two characters, ö, and carries on. (iconv only converts the output for this page. The mojibake is in the string itself.) No exception is raised, so the bad data goes into whatever the reader writes next. PEP 686 warns about exactly this: UnicodeError, mojibake, “or even silent data corruption”.

Case 3: Subprocess output in the locale encoding

text=True decodes a child’s output with the same default, and plenty of tools still print in the locale encoding:

# month.py — ask a locale-aware tool for a month name and decode its output
import subprocess
import sys

mode = sys.argv[1] if len(sys.argv) > 1 else "default"
kwargs = {"text": True} if mode == "default" else {"encoding": "locale"}
out = subprocess.run(["date", "-d", "2026-03-01", "+%B"],
                     capture_output=True, check=True, **kwargs).stdout
print(f"{sys.version.split()[0]} {mode:<8} {out.strip()!r}")
$ clean LANG=de_AT.ISO-8859-1 python3.14 month.py | iconv -f ISO-8859-1 -t UTF-8
3.14.8 default  'März'
$ clean LANG=de_AT.ISO-8859-1 python3.15 month.py 2>&1 | tail -1
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 1: invalid continuation byte
$ clean LANG=de_AT.ISO-8859-1 python3.15 month.py locale
3.15.0 locale   'März'

date wrote März in ISO-8859-1 because its locale told it to. 3.15 decodes the pipe as UTF-8 and fails on e4. With encoding="locale", parent and child agree again. Notice the last line, too: 3.15’s own print wrote UTF-8 to a terminal whose locale says ISO-8859-1. A real legacy terminal (or a Windows program reading Python’s piped output) would show März. The default changed for output as well as input.

PEP 686 considered and rejected a special case that would keep pipes on the locale encoding. The answer for subprocesses is the same as for files: name the encoding, either encoding="utf-8" or encoding="locale".

Case 4: Find every place you relied on the default

Grepping for open( is not enough, because subprocess, pathlib.Path.read_text, tempfile and library helpers all reach the same default. The interpreter can report them for you. -X warn_default_encoding (or PYTHONWARNDEFAULTENCODING=1) emits an EncodingWarning (PEP 597) at every call site that did not pass an encoding:

$ clean LANG=en_US.UTF-8 python3.15 -X warn_default_encoding export_report.py
.../export_report.py:3: EncodingWarning: 'encoding' argument not specified
  with open("report.csv", "w") as f:
$ clean LANG=en_US.UTF-8 python3.15 -X warn_default_encoding month.py
.../month.py:7: EncodingWarning: 'encoding' argument not specified.
  out = subprocess.run(["date", "-d", "2026-03-01", "+%B"],
3.15.0 default  'March'
$ clean LANG=en_US.UTF-8 python3.15 -X warn_default_encoding -c "import locale; print(locale.getpreferredencoding())"
<string>:1: EncodingWarning: UTF-8 Mode affects locale.getpreferredencoding(). Consider locale.getencoding() instead.
utf-8

It works on a UTF-8 machine, so you can turn it on in ordinary Linux CI. Make it fatal there:

$ clean LANG=en_US.UTF-8 PYTHONWARNDEFAULTENCODING=1 python3.15 -W error::EncodingWarning export_report.py 2>&1 | tail -1
EncodingWarning: 'encoding' argument not specified

(The run exits with status 1.) For a static pass, Ruff’s unspecified-encoding rule (PLW1514, preview-only in Ruff 0.17.0) flags open() calls:

$ uvx ruff@0.17.0 check --preview --select PLW1514 --output-format concise export_report.py read_report.py month.py
export_report.py:3:6: unspecified-encoding: `open` in text mode without explicit `encoding` argument
read_report.py:2:6: unspecified-encoding: `open` in text mode without explicit `encoding` argument
Found 2 errors.
2 unsafe fixes available with `--fix --unsafe-fixes`.

It did not flag subprocess.run(text=True) in month.py, and its autofix always inserts encoding="utf-8". That is the right fix for new data and the wrong one for a legacy file. Use the linter for coverage of open() and the runtime warning for everything else.

If you write a library function that accepts encoding=None and passes it on, don’t hide the warning inside your code. io.text_encoding() resolves None to the current default and attributes the warning to your caller:

# textenc.py — what a library helper should pass through
import io
import sys

def load(path, encoding=None):
    encoding = io.text_encoding(encoding)   # PEP 597: warn at the caller, not here
    with open(path, encoding=encoding) as f:
        return encoding, f.readline().strip()

print(sys.version.split()[0], load("report.csv"))
$ clean LANG=en_US.UTF-8 python3.14 -X warn_default_encoding textenc.py
.../textenc.py:10: EncodingWarning: 'encoding' argument not specified
  print(sys.version.split()[0], load("report.csv"))
3.14.8 ('locale', 'city;temp_c')
$ clean LANG=en_US.UTF-8 python3.15 -X warn_default_encoding textenc.py
.../textenc.py:10: EncodingWarning: 'encoding' argument not specified
  print(sys.version.split()[0], load("report.csv"))
3.15.0 ('utf-8', 'city;temp_c')

The warning points at line 10, the call, not at the open() inside load. The resolved value also shows the version change directly: 'locale' on 3.14, 'utf-8' on 3.15.

Case 5: The fixes, and the escape hatch

# report_fixed.py — say what you mean on both ends
import locale

with open("report.csv", "w", encoding="utf-8", newline="") as f:
    f.write("city;temp_c\nKlagenfurt am Wörthersee;9\n")

# A file from the old Windows/legacy job: decode it as what it is
legacy = "Klagenfurt am Wörthersee;9\n".encode("iso-8859-1")
with open("legacy.csv", "wb") as f:
    f.write(legacy)
with open("legacy.csv", encoding="iso-8859-1") as f:
    print(repr(f.read()))
with open("legacy.csv", encoding="locale") as f:
    print(locale.getencoding(), repr(f.read()))
$ clean LANG=en_US.ISO-8859-1 python3.15 report_fixed.py
'Klagenfurt am Wörthersee;9\n'
ISO-8859-1 'Klagenfurt am Wörthersee;9\n'

New data is written as UTF-8 on purpose. Old data is read as what it is: either a named codec, or encoding="locale" when “whatever this machine uses” really is the contract (for example, a file a local user just saved from a legacy editor). Prefer the named codec for anything that travels between machines.

When you cannot fix the code yet, turn UTF-8 mode off for that process. It must happen at startup: there is no runtime switch.

$ clean LANG=en_US.ISO-8859-1 PYTHONUTF8=0 python3.15 probe.py
Python 3.15.0  utf8_mode=0
  locale.getencoding()           ISO-8859-1
  locale.getpreferredencoding()  ISO-8859-1
  sys.getfilesystemencoding()    iso8859-1
  sys.stdout                     iso8859-1 / strict
  open('probe.txt', 'w')         ISO-8859-1
  open(..., encoding='locale')   ISO-8859-1
$ clean LANG=en_US.ISO-8859-1 python3.15 -X utf8=0 -c "import sys; print(sys.flags.utf8_mode)"
0
$ clean LANG=C PYTHONUTF8=0 python3.15 -c "import sys, locale; print(sys.flags.utf8_mode, locale.getencoding(), sys.stdout.errors)"
0 UTF-8 surrogateescape

The first run is byte-for-byte the 3.14 behaviour. The last run shows the opt-out is not a time machine to “ASCII”. Under the C locale, Python still coerces to C.UTF-8 (PEP 538), so you get UTF-8 without UTF-8 mode. The 3.15 Windows docs add one warning: putting PYTHONUTF8=0 in the system environment changes every Python 3.7+ program on the machine. Scope it to the one service or script that needs it, and give it a removal date.

The trap

The new default only seems to remove the decision. It doesn’t, for two reasons.

First, the default depends on the interpreter. In Cases 1 and 2 the same export_report.py, on the same machine and in the same environment, wrote f6 under 3.14 and c3 b6 under 3.15. A package that supports 3.10–3.15 and leaves out encoding= writes different bytes depending on which interpreter the user installed, or whether someone set PYTHONUTF8=0 in a container image. “3.15 defaults to UTF-8, so I can drop encoding=” is only true once your oldest supported interpreter is 3.15 and nobody turns the mode off.

Second, surrogateescape on the standard streams moves failures instead of removing them. Linux filenames are bytes, and some are not valid UTF-8. Python decodes them with lone surrogates so they round-trip:

# list_inbox.py — print every name in a directory, then save the list
import os
import sys

names = sorted(os.listdir("inbox"))
print(sys.version.split()[0], [ascii(n) for n in names])
for n in names:
    print(n)
with open("inbox.txt", "w", encoding="utf-8") as f:
    f.write("\n".join(names) + "\n")
$ clean LANG=en_US.UTF-8 python3.14 list_inbox.py 2>&1 | grep -E "line [0-9]+|Error"
  File ".../list_inbox.py", line 8, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 2-3: surrogates not allowed
$ clean LANG=en_US.UTF-8 python3.15 list_inbox.py 2>&1 | grep -E "line [0-9]+|Error"
  File ".../list_inbox.py", line 10, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 2-3: surrogates not allowed

The directory holds Gr\xfc\xdfe.txt, a name saved by a Latin-1 program. 3.14 crashed at line 8, the print, because its stdout was strict. 3.15 prints the raw bytes happily and crashes two lines later, writing the list to a file with plain encoding="utf-8". A script that “printed fine in testing” now fails in the step that persists data. When names may be arbitrary bytes, say so on the file too:

# fixed_inbox.py — keep undecodable names intact on the way to disk
import os

names = sorted(os.listdir("inbox"))
with open("inbox.txt", "w", encoding="utf-8", errors="surrogateescape") as f:
    f.write("\n".join(names) + "\n")
with open("inbox.txt", "rb") as f:
    print(f.read())
$ clean LANG=en_US.UTF-8 python3.15 fixed_inbox.py
b'Gr\xfc\xdfe.txt\nplain.txt\n'

The original bytes come back unchanged.

The boring rule

Pass encoding= to every text-mode open(), Path.read_text()/write_text(), and subprocess call: "utf-8" for data you own, a named codec or "locale" for legacy input. Run CI with PYTHONWARNDEFAULTENCODING=1 -W error::EncodingWarning. Replace locale.getpreferredencoding() with locale.getencoding() wherever you actually mean “the system’s encoding”. Use PYTHONUTF8=0 only as a scoped, dated workaround, never as a machine-wide setting.

Try this

  1. Run probe.py under clean LANG=C python3.14 and clean LANG=C python3.15. Do any lines differ? Explain from the mental model why the C locale already got UTF-8 mode on 3.14.
  2. Rewrite read_report.py to accept both old and new exports: try utf-8 first, fall back to iso-8859-1 on UnicodeDecodeError. Then explain why trying the other order would be wrong.
  3. Add PYTHONWARNDEFAULTENCODING=1 and -W error::EncodingWarning to your own test suite’s runner and count the failures. How many come from your code, and how many from dependencies?
  4. Change month.py to encoding="utf-8" and run it under de_AT.ISO-8859-1. Then set LC_ALL=de_AT.UTF-8 for the child only (env= argument) and make it work without touching the parent’s decoding.
  5. Repeat the trap with Path("inbox").iterdir() and json.dump. Where does the surrogate now fail, and what would os.fsencode() let you store instead?

Sources