Tokenizers documentation
Decoders
You are viewing main version, which requires installation from source. If you'd like
regular pip install, checkout the latest stable version (v1.0.0-rc.3).
Decoders
TODO — coming soon. Reference documentation on this page is not generated yet: the rc0 bindings do not expose all of these APIs. It comes back with them, along with the other bindings. See
REQUIRED_FOR_V1.mdfor the full list.
Python
Rust
Node
DecodeStream
tokenizers.decoders.DecodeStream BPEDecoder
tokenizers.decoders.BPEDecoder ByteFallback
tokenizers.decoders.ByteFallback ByteLevel
tokenizers.decoders.ByteLevel CTC
tokenizers.decoders.CTC Fuse
tokenizers.decoders.Fuse Metaspace
tokenizers.decoders.Metaspace Replace
tokenizers.decoders.Replace Sequence
tokenizers.decoders.Sequence Strip
tokenizers.decoders.Strip WordPiece
tokenizers.decoders.WordPiece