Specify a github or local repo, github pull request, arXiv or Sci-Hub paper, Youtube transcript or documentation URL on the web and scrape into a text file and clipboard for easier LLM ingestion

The ChatGPT Long Term Memory package is a powerful tool designed to empower your projects with the ability to handle a large number of simultaneous users and external sources.

A custom GPT based on [Zero To Hero](https://karpathy.ai/zero-to-hero.html) utilizing tiktoken with the intent to augment AI Transformer-model education and reverse engineer GPT models from scratch.

Predict the exact openai token usage of functions

🧰 The AutoTokenizer that TikToken always needed -- Load any tokenizer with TikToken now! ✨

The Tiktoken API is a tool that enables developers to calculate the token usage of their OpenAI API requests before sending them, allowing for more efficient use of tokens.

Calculate the token count for GPT-4, GPT-3.5, GPT-3, and GPT-2.

Count tokens in a text file.

Simple wrapper around tiktoken to use it in your favorite language.

Docker container to expose the OpenAI tokenizer as a REST service

Logseq Markdown Tokenizer is a Python application that tokenizes and estimates prices for one to many markdown files.

tiktoken fastapi for encoding and decoding tokens for OpenAI ChatGPT GPT-3 GPT-3.5 GPT-4 models

Extracts Google Sheets to JSONL for fine-tuning, estimates task costs with tiktoken.

Tiktoken-counter as standalone API

List of single token words for LLM usage

Este projeto consiste em uma aplicação que utiliza IA generativa para enviar recomendações de produtos com base nos perfis de compra de cada cliente

This project uses the CrewAI framework to automate stock analysis, enabling AI agents to collaborate and execute complex tasks efficiently. Example stock: Nvidia. Technologies include Python, CrewAI, Unstructured, PyOWM, Tools, Wikipedia, yFinance, SEC-API, tiktoken, faiss-cpu, python-dotenv, langchain-community, langchain-core, and OpenAI.

Count tokens to determine cost differential of ChatGPT Plus subscription and ChatGPT API

Text splitting example using Tiktoken

console based game based on a llm