Tokenizers

Input Sequences

Not supported by the rc0 bindings yet. rc0 loads a tokenizer.json and encodes and decodes with it — building a tokenizer from its components, editing one, saving one and training are not exposed. They are coming soon, along with the other bindings. See REQUIRED_FOR_V1.md for the full list.

Python

Rust

Node

These types represent all the different kinds of sequence that can be used as input of a Tokenizer. Globally, any sequence can be either a string or a list of strings, according to the operating mode of the tokenizer: raw text vs pre-tokenized.

TextInputSequence

tokenizers.TextInputSequence

A str that represents an input sequence

PreTokenizedInputSequence

tokenizers.PreTokenizedInputSequence

A pre-tokenized input sequence. Can be one of:

A List of str
A Tuple of str

alias of Union[List[str], Tuple[str]].

InputSequence

tokenizers.InputSequence

Represents all the possible types of input sequences for encoding. Can be:

When is_pretokenized=False: TextInputSequence
When is_pretokenized=True: PreTokenizedInputSequence

alias of Union[str, List[str], Tuple[str]].

Update on GitHub