Skip to main content

Module hf_json

Module hf_json 

Source
Expand description

Hugging Face tokenizer.json parser.

Reconstructs a full BPE/ByteLevel GgufTokenizer from sibling tokenizer.json files, preventing fallback to 256-byte dummy tokenizers when loading Safetensors models.

Functionsยง

parse_hf_tokenizer_json
Parse Hugging Face tokenizer.json into a GgufTokenizer.