tokenizers license, foundation and stars
ML frameworks 路 Apache 2.0 (open source) 路 backed by Hugging Face 路 11,112 stars
馃挜 Fast State-of-the-Art Tokenizers optimized for Research and Production
License history
LICENSE fileApache 2.0 No license change recorded since the repository was created in 2019.
Latest release
No GitHub releases published; the project releases elsewhere (tags or its own site).
More in ml frameworks
| Project | Stars | |||
|---|---|---|---|---|
 TensorFlowGoogleApache 2.0 | Apache 2.0 | 200k | +538 | v2.21.0Mar 6, 2026 |
 TransformersHugging FaceApache 2.0 | Apache 2.0 | 167k | +652 | v5.17.0Sep 9, 2026 |
 whisperOpenAIMIT | MIT | 110k | - | - |
 PyTorchPyTorch FoundationBSD-style | BSD-style | 103k | +187 | v2.14.0Sep 2, 2026 |
 opencvApache 2.0 | Apache 2.0 | 91k | - | - |
 PaddleOCRApache 2.0 | Apache 2.0 | 90.2k | - | - |
 unslothApache 2.0 | Apache 2.0 | 76.7k | - | - |
 tesseractApache 2.0 | Apache 2.0 | 76.7k | - | - |
About and history
- Repository
- huggingface/tokenizers 路 created Nov 1, 2019
- Website
- huggingface.co/docs/tokenizers
- Language
- Rust
- Foundation
- None; run by Hugging Face
- Company
- Hugging Face 路 on Companies 路 on Acquisitions 路 on awards 路 on blogs 路 on Attested 路 on Tier Zero 路 on Mainstage 路 on Leaders 路 on marketplaces 路 on papers 路 on Paydays 路 on Releases 路 on Nines 路 on training 路 on Hotlist
- GitHub detects
- Apache-2.0
- Stars per week
- From the second weekly snapshot
Weekly snapshots
| Week of | Stars | |
|---|---|---|
| Sep 21, 2026 | 11,112 |
Changes
| What changed | |
|---|---|
| Sep 25, 2026 | TrackedAdded: huggingface/tokenizers |