from Hacker News

Calculate the number of language model tokens for a string

by waldekm on 2/2/25, 8:47 AM with 3 comments

  • by sva_ on 2/5/25, 2:01 PM

      import tiktoken
      enc = tiktoken.encoding_for_model('o1-mini')
      len(enc.encode('some string'))