Open source for data and AI

Projects749Relicensed100

749 projects · 100 relicensed

tiktoken license, foundation and stars

ML frameworks · MIT (open source) · backed by OpenAI · 19,619 stars

GitHub

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

License history

LICENSE file

MIT No license change recorded since the repository was created in 2022.

Latest release

No GitHub releases published; the project releases elsewhere (tags or its own site).

More in ml frameworks

Apache 2.0201k starsTensorFlowAn Open Source Machine Learning Framework for Everyone.Google · +180 a week · v2.21.0, Mar 6, 2026Apache 2.0169k starsTransformersTransformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.Hugging Face · +337 a week · v5.17.0, Sep 9, 2026MIT112k starswhisperRobust Speech Recognition via Large-Scale Weak Supervision.OpenAI · +387 a weekBSD-style105k starsPyTorchTensors and Dynamic neural networks in Python with strong GPU acceleration.PyTorch Foundation · +311 a week · v2.14.0, Sep 2, 2026Apache 2.092.2k starsPaddleOCRTurn any PDF or image document into structured data for your AI.+333 a weekApache 2.091.4k starsopencvOpen Source Computer Vision Library.+77 a weekApache 2.079.2k starsunslothLocal UI to run and train LLMs and diffusion models.+407 a weekApache 2.077.4k starstesseractA machine learning library by tesseract-ocr, written in C++.+122 a week
ProjectStars
TensorFlowAn Open Source Machine Learning Framework for Everyone.GoogleApache 2.0201k
TransformersTransformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.Hugging FaceApache 2.0169k
whisperRobust Speech Recognition via Large-Scale Weak Supervision.OpenAIMIT112k
PyTorchTensors and Dynamic neural networks in Python with strong GPU acceleration.PyTorch FoundationBSD-style105k
PaddleOCRTurn any PDF or image document into structured data for your AI.Apache 2.092.2k
opencvOpen Source Computer Vision Library.Apache 2.091.4k
unslothLocal UI to run and train LLMs and diffusion models.Apache 2.079.2k
tesseractA machine learning library by tesseract-ocr, written in C++.Apache 2.077.4k
About and history
Repository
openai/tiktoken · created Dec 1, 2022
Language
Python
Foundation
None; run by OpenAI
Company
OpenAI · on Companies · on Acquisitions · on awards · on blogs · on Attested · on Downtime · on Mainstage · on Leaders · on Arena · on papers · on Allied · on Paydays · on predictions · on Metered · on quotes · on Releases · on Nines · on TechConf · on Titlewave · on training · on Hotlist
GitHub detects
MIT
Stars per week
49 (GitHub weekly snapshots)

Weekly snapshots

Week ofStars
Nov 2, 202619,619
Oct 26, 202619,549
Oct 12, 202619,485
Oct 5, 202619,425
Sep 21, 202619,323

Changes

Stars per weekNov 8, 2026

Gains about 49 stars a week, up from 45.

Stars per weekOct 29, 2026

Gains about 45 stars a week, down from 54.

Stars per weekOct 18, 2026

Gains about 54 stars a week, up from 51.

Stars per weekOct 9, 2026

Gains about 51 stars a week.

TrackedSep 25, 2026

Now tracked, from openai/tiktoken on GitHub.