Python 3.15 (source build): OverflowError: Python int too large to convert to C ssize_t in Tokenizer._flush_specials on any spacy.load() — same recipe works on 3.14 #14047

Summary

Building spaCy 3.8.16 from source on CPython 3.15.0 (Windows x64) compiles and imports fine, but any spacy.load() of a model crashes during tokenizer deserialization:

File "spacy/tokenizer.pyx", line 787, in spacy.tokenizer.Tokenizer.from_disk
File "spacy/tokenizer.pyx", line 833, in spacy.tokenizer.Tokenizer.from_bytes
  self.rules = {}
File "spacy/tokenizer.pyx", line 128, in spacy.tokenizer.Tokenizer.rules.__set__
  self._flush_specials()
File "spacy/tokenizer.pyx", line 239, in spacy.tokenizer.Tokenizer._flush_specials
  del self._specials[k]
OverflowError: Python int too large to convert to C ssize_t

Since there are no cp315 wheels yet, building from source is currently the only way to try spaCy on 3.15 — and this is where it lands.

A/B test (controlled, same build recipe both sides)

On the 3.14 side I built the wheel (pip wheel spacy==3.8.16 --no-build-isolation), unzipped it to an isolated dir and loaded it via PYTHONPATH so the official site-packages install was untouched — the source-built copy loads models fine. Same commands on 3.15 crash. With the build recipe and dependency stack held identical, the remaining variable is the Python version itself.

Steps to reproduce (Windows x64, Python 3.15.0)

Toolchain: VS Build Tools 2022 (MSVC 14.44), Windows SDK 10.0.26100, LLVM (clang — blis's build looks for it), BLIS_ARCH=generic (clang rejects -mavx512pf in the skx kernels).

py -3.15 -m pip install cython numpy setuptools wheel
py -3.15 -m pip install murmurhash==1.0.15 cymem==2.0.13 preshed==3.0.13 --ignore-requires-python --no-build-isolation
# blis needs BLIS_ARCH=generic with clang on this box:
BLIS_ARCH=generic py -3.15 -m pip install blis==1.3.3 --ignore-requires-python --no-build-isolation
py -3.15 -m pip install thinc==8.3.13 --ignore-requires-python --no-build-isolation
py -3.15 -m pip install spacy==3.8.16 --ignore-requires-python --no-build-isolation
py -3.15 -m pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl
py -3.15 -c "import spacy; spacy.load('en_core_web_sm')"   # → OverflowError

Notes

• I could not honor the cython>=0.25,<3.0 build pin: NumPy 2.x's __init__.pxd hard-errors on Cython < 3.0 ("the NumPy Cython headers require Cython 3.0.0 or newer"), so on 3.15 Cython 3.x is forced. Tested both Cython 3.0.12 and 3.3.0 — identical crash; and the 3.14 side of the A/B also used Cython 3.0.12 (i.e. violating the pin the same way) and works, so the Cython version alone doesn't explain it.
• Dependency skew ruled out: an earlier run had preshed 3.0.10 on the 3.15 side; after upgrading to 3.0.13 (matching the working 3.14 stack) the crash is unchanged.
• The value that fails conversion is a key coming out of for k in self._specials (PreshMap iteration) in _flush_specials; on 3.15 the int→C conversion in the generated code raises where 3.14 doesn't. I'm not claiming a specific CPython change — just reporting the controlled difference.

Environment

• spaCy 3.8.16 (source), thinc 8.3.13, preshed 3.0.13, blis 1.3.3, cymem 2.0.13, murmurhash 1.0.15, Cython 3.0.12, numpy 2.5.3
• CPython 3.15.0 / 3.14.2, Windows 10 x64 (26200), MSVC 14.44.35207 + LLVM
• env: BLIS_ARCH=generic

Happy to run any further experiment you'd want on this box (it has the full source-build toolchain set up).