from
Hacker News
Top
New
Tiktoken-rs: Batteries-included Rust library for working with GPT-4's tokenizer
by
zurawiki
on 3/18/23, 4:22 AM with 1 comments
by
zurawiki
on 3/18/23, 4:22 AM
In case others find this useful. Just updated for GPT-4, tiktoken-rs is a Rust crate you can use to tokenize text sent to and from OpenAI's LLM models