GigaToken: ~1000x faster Language model tokenization

GigaToken is a new tokenizer that claims to be approximately 1000x faster than HuggingFace's tokenizers while serving as a drop-in replacement, capable of processing text data at gigabytes per second. It supports multiple CPU architectures and common tokenizer formats, with options for both a compatibility mode for existing HuggingFace and Tiktoken APIs and a native Gigatoken API for maximum performance. The tool is available via pip installation and is written in Rust to leverage multithreading and direct data processing for optimal speed.