Skip to content

don't split German gender-inclusive colon forms - #14022

Closed
gyanu2507 wants to merge 1 commit into
explosion:masterfrom
gyanu2507:fix/de-gender-colon
Closed

gyanu2507 wants to merge 1 commit into
explosion:masterfrom
gyanu2507:fix/de-gender-colon

Conversation

@gyanu2507

Copy link
Copy Markdown

Fixes #14010

German infix [:<>=] splits gender-inclusive colon forms before anything else sees them:

nlp("Unsere Kund:innen und Mitarbeiter:innen sind zufrieden.")
# Kund / : / innen

Star and interpunct variants already stay whole (Kolleg*innen, Expert·innen). This drops : from that infix class and leaves <>= splitting alone, which is the patch we've been running in production.

eine:n / Ärzt:in stay one token as well. Comparisons like x<y still split.

Kund:innen and similar were broken on the letter-colon-letter infix.
Keep <>= splitting; drop only the colon from that class.

Co-authored-by: Cursor <cursoragent@cursor.com>
@gyanu2507

Copy link
Copy Markdown
Author

Closing this to narrow my open-source work to fewer projects. Anyone is welcome to pick the change up if it's useful.

@gyanu2507 gyanu2507 closed this Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

German tokenizer splits gender-inclusive colon forms (Kund:innen)

1 participant