GitHub repository
vectorize-io/toktok
hyper-fast bpe tokenizer - drop in replacement for tiktoken
Repository metrics
- Stars · Nov 15, 2026
- 10
- Leaderboard status
- Outside the current top 1,000
Language Rust
Source: GitHubLB snapshots.
GitHub repository
hyper-fast bpe tokenizer - drop in replacement for tiktoken
Source: GitHubLB snapshots.