GitHub repository
explosion/curated-tokenizers
Lightweight piece tokenization library
Repository metrics
- Stars · Nov 15, 2026
- 12
- Leaderboard status
- Outside the current top 1,000
Language Cython
Source: GitHubLB snapshots.
GitHub repository
Lightweight piece tokenization library
Source: GitHubLB snapshots.