Reposby Glyph
tokenizers license, foundation and stars
ML frameworks · Apache 2.0 (open source) · backed by Hugging Face · 11,228 stars
Fast State-of-the-Art Tokenizers optimized for Research and Production
License history
LICENSE fileApache 2.0 No license change recorded since the repository was created in 2019.
Latest release
No GitHub releases published; the project releases elsewhere (tags or its own site).
More in ml frameworks
TensorFlowAn Open Source Machine Learning Framework for Everyone.Google · +180 a week · v2.21.0, Mar 6, 2026
TransformersTransformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.Hugging Face · +337 a week · v5.17.0, Sep 9, 2026
whisperRobust Speech Recognition via Large-Scale Weak Supervision.OpenAI · +387 a week
PyTorchTensors and Dynamic neural networks in Python with strong GPU acceleration.PyTorch Foundation · +311 a week · v2.14.0, Sep 2, 2026
PaddleOCRTurn any PDF or image document into structured data for your AI.+333 a week
opencvOpen Source Computer Vision Library.+77 a week
unslothLocal UI to run and train LLMs and diffusion models.+407 a week
tesseractA machine learning library by tesseract-ocr, written in C++.+122 a week| Project | Stars | |||
|---|---|---|---|---|
 TensorFlowAn Open Source Machine Learning Framework for Everyone.GoogleApache 2.0 | Apache 2.0 | 201k | +180 | v2.21.0Mar 6, 2026 |
 TransformersTransformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.Hugging FaceApache 2.0 | Apache 2.0 | 169k | +337 | v5.17.0Sep 9, 2026 |
 whisperRobust Speech Recognition via Large-Scale Weak Supervision.OpenAIMIT | MIT | 112k | +387 | - |
 PyTorchTensors and Dynamic neural networks in Python with strong GPU acceleration.PyTorch FoundationBSD-style | BSD-style | 105k | +311 | v2.14.0Sep 2, 2026 |
 PaddleOCRTurn any PDF or image document into structured data for your AI.Apache 2.0 | Apache 2.0 | 92.2k | +333 | - |
 opencvOpen Source Computer Vision Library.Apache 2.0 | Apache 2.0 | 91.4k | +77 | - |
 unslothLocal UI to run and train LLMs and diffusion models.Apache 2.0 | Apache 2.0 | 79.2k | +407 | - |
 tesseractA machine learning library by tesseract-ocr, written in C++.Apache 2.0 | Apache 2.0 | 77.4k | +122 | - |
About and history
- Repository
- huggingface/tokenizers · created Nov 1, 2019
- Website
- huggingface.co/docs/tokenizers
- Language
- Rust
- Foundation
- None; run by Hugging Face
- Company
- Hugging Face · on Companies · on Acquisitions · on awards · on blogs · on Attested · on Tier Zero · on Mainstage · on Leaders · on marketplaces · on papers · on Paydays · on Releases · on Nines · on training · on Hotlist
- GitHub detects
- Apache-2.0
- Stars per week
- 19 (GitHub weekly snapshots)
Weekly snapshots
| Week of | Stars | |
|---|---|---|
| Nov 2, 2026 | 11,228 | |
| Oct 26, 2026 | 11,211 | |
| Oct 12, 2026 | 11,183 | |
| Oct 5, 2026 | 11,162 | |
| Sep 21, 2026 | 11,112 |