Expand description
Hugging Face tokenizer.json parser.
Reconstructs a full BPE/ByteLevel GgufTokenizer from sibling tokenizer.json files,
preventing fallback to 256-byte dummy tokenizers when loading Safetensors models.
Functionsยง
- parse_
hf_ tokenizer_ json - Parse Hugging Face
tokenizer.jsoninto aGgufTokenizer.