Turns text into the input tensors of a model, with the tokenizer.json of a model repository.
Needs the optional tokenizers dependency: add {:tokenizers, "~> 0.5"} to your deps.
tokenizer = Transmuter.Tokenizer.load!({:hf, "Xenova/all-MiniLM-L6-v2"})
{inputs, _encodings} = Transmuter.Tokenizer.encode(tokenizer, ["Hello world"])
inputs["input_ids"]Summary
Functions
Sets the :length, :pad_direction, :truncation, :stride and :clean_up_tokenization_spaces options of load!/2.
Decodes token ids into text, without the special tokens. With skip_special_tokens: false, the text keeps them, such as the location tokens of Florence-2. With the :clean_up_tokenization_spaces option of load!/2, the text has no space before punctuation and English contractions.
Encodes inputs, a list of strings or {text, text_pair} tuples.
Returns the token ids of encoding, a Tokenizers.Encoding of tokenizer. The BPE model of M2M100 merges pieces that its vocabulary does not have, and such a piece is the unknown token, as in Python transformers. The other ids are those of the encoding.
Loads the tokenizer of a Transmuter.Repo or of a source ({:hf, repo} or a local directory).
Returns the token that pads inputs. Tokenizers without a pad token pad with the eos token, as Python transformers does.
Returns the id of token, or nil when the vocabulary does not have it.
Functions
Sets the :length, :pad_direction, :truncation, :stride and :clean_up_tokenization_spaces options of load!/2.
Decodes token ids into text, without the special tokens. With skip_special_tokens: false, the text keeps them, such as the location tokens of Florence-2. With the :clean_up_tokenization_spaces option of load!/2, the text has no space before punctuation and English contractions.
Encodes inputs, a list of strings or {text, text_pair} tuples.
Returns the input tensors and, for each input, the list of its Tokenizers.Encoding, one for each row of the tensors. An input has one row, or one row for each window with :stride. An encoding has the offsets, the word ids and the special tokens mask. The tensors are "input_ids", "attention_mask" and "token_type_ids", of type :s64 on Nx.BinaryBackend. "input_ids" has the ids of ids/2. Read the ids of an encoding with ids/2, not with Tokenizers.Encoding.get_ids/1.
Options
:add_special_tokens- adds the special tokens of the tokenizer, such as[CLS]and[SEP]. The default istrue. Usefalsefor text that already has them, such as the output of a chat template.:suffix- token ids that end each input, such as the end of a chat template. When an input reaches the largest length of:length, so that truncation can have cut it, these ids replace its last tokens, as sentence-transformers restores the end of a chat template. The default is[].
Returns the token ids of encoding, a Tokenizers.Encoding of tokenizer. The BPE model of M2M100 merges pieces that its vocabulary does not have, and such a piece is the unknown token, as in Python transformers. The other ids are those of the encoding.
Loads the tokenizer of a Transmuter.Repo or of a source ({:hf, repo} or a local directory).
Options
:length- the length of the encoded inputs.nil(the default) pads to the longest input. An integer pads to that length. A list of lengths pads to the smallest length that fits the longest input. Inputs longer than the largest length are truncated.:pad_direction-:right(the default) or:left. The servings of the tasks always pad on the right, also with a tokenizer from their:tokenizeroption (seeTransmuter.Serving.tokenizer/2).:truncation- the text of a pair that truncation shortens::longest_first(the default),:only_firstor:only_second.:stride-nil(the default) removes the tokens after the largest length. An integer keeps them in more windows of the largest length. Each window repeats the laststridetokens of the window before it. The stride is at most half of the largest length, as in the question answering pipeline of Python transformers.:clean_up_tokenization_spaces- whentrue,decode/3removes the space before punctuation and English contractions, asclean_up_tokenizationof Python transformers. The default isclean_up_tokenization_spacesoftokenizer_config.json, andfalsewhen the file does not have it.
Other options go to Transmuter.Repo.load!/2 when the first argument is a source.
Returns the token that pads inputs. Tokenizers without a pad token pad with the eos token, as Python transformers does.
Returns the id of token, or nil when the vocabulary does not have it.