Tokenizers

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v1.0.0-rc.3).

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

Post-processors

TODO — coming soon. Reference documentation on this page is not generated yet: the rc0 bindings do not expose all of these APIs. It comes back with them, along with the other bindings. See REQUIRED_FOR_V1.md for the full list.

Every post-processor is a template. BertProcessing, RobertaProcessing, ByteLevel and a Sequence of post-processors were spellings of one, so a tokenizer.json using any of them is rewritten into a TemplateProcessing when it is converted, and there is no separate class for them.

Python

Rust

Node

PostProcessor

tokenizers.processors.PostProcessor

TemplateProcessing

tokenizers.processors.TemplateProcessing

Update on GitHub

←Pre-tokenizers Trainers→