GitHub repository
huggingface/tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
Repository metrics
- Stars · Nov 15, 2026
- 10,479
- Leaderboard status
- Outside the current top 1,000
Language Rust
Packages from this repository
- tokenizersPyPI205,008,768
Source: GitHubLB snapshots.