# `Transmuter.Tokenizer`
[🔗](https://github.com/georgeguimaraes/transmuter/blob/v2.0.0/lib/transmuter/tokenizer.ex#L1)

Turns text into the input tensors of a model, with the `tokenizer.json` of a model repository.

Needs the optional `tokenizers` dependency: add `{:tokenizers, "~> 0.5"}` to your deps.

    tokenizer = Transmuter.Tokenizer.load!({:hf, "Xenova/all-MiniLM-L6-v2"})
    {inputs, _encodings} = Transmuter.Tokenizer.encode(tokenizer, ["Hello world"])
    inputs["input_ids"]

# `configure`

Sets the `:length`, `:pad_direction`, `:truncation`, `:stride` and `:clean_up_tokenization_spaces` options of `load!/2`.

# `decode`

Decodes token ids into text, without the special tokens. With `skip_special_tokens: false`, the text keeps them, such as the location tokens of Florence-2. With the `:clean_up_tokenization_spaces` option of `load!/2`, the text has no space before punctuation and English contractions.

# `encode`

Encodes `inputs`, a list of strings or `{text, text_pair}` tuples.

Returns the input tensors and, for each input, the list of its `Tokenizers.Encoding`, one for each row of the tensors. An input has one row, or one row for each window with `:stride`. An encoding has the offsets, the word ids and the special tokens mask. The tensors are `"input_ids"`, `"attention_mask"` and `"token_type_ids"`, of type `:s64` on `Nx.BinaryBackend`. `"input_ids"` has the ids of `ids/2`. Read the ids of an encoding with `ids/2`, not with `Tokenizers.Encoding.get_ids/1`.

## Options

  * `:add_special_tokens` - adds the special tokens of the tokenizer, such as `[CLS]` and `[SEP]`. The default is `true`. Use `false` for text that already has them, such as the output of a chat template.

  * `:suffix` - token ids that end each input, such as the end of a chat template. When an input reaches the largest length of `:length`, so that truncation can have cut it, these ids replace its last tokens, as sentence-transformers restores the end of a chat template. The default is `[]`.

# `ids`

Returns the token ids of `encoding`, a `Tokenizers.Encoding` of `tokenizer`. The BPE model of M2M100 merges pieces that its vocabulary does not have, and such a piece is the unknown token, as in Python transformers. The other ids are those of the encoding.

# `load!`

Loads the tokenizer of a `Transmuter.Repo` or of a source (`{:hf, repo}` or a local directory).

## Options

  * `:length` - the length of the encoded inputs. `nil` (the default) pads to the longest input. An integer pads to that length. A list of lengths pads to the smallest length that fits the longest input. Inputs longer than the largest length are truncated.

  * `:pad_direction` - `:right` (the default) or `:left`. The servings of the tasks always pad on the right, also with a tokenizer from their `:tokenizer` option (see `Transmuter.Serving.tokenizer/2`).

  * `:truncation` - the text of a pair that truncation shortens: `:longest_first` (the default), `:only_first` or `:only_second`.

  * `:stride` - `nil` (the default) removes the tokens after the largest length. An integer keeps them in more windows of the largest length. Each window repeats the last `stride` tokens of the window before it. The stride is at most half of the largest length, as in the question answering pipeline of Python transformers.

  * `:clean_up_tokenization_spaces` - when `true`, `decode/3` removes the space before punctuation and English contractions, as `clean_up_tokenization` of Python transformers. The default is `clean_up_tokenization_spaces` of `tokenizer_config.json`, and `false` when the file does not have it.

Other options go to `Transmuter.Repo.load!/2` when the first argument is a source.

# `pad_token`

Returns the token that pads inputs. Tokenizers without a pad token pad with the eos token, as Python transformers does.

# `token_to_id`

Returns the id of `token`, or `nil` when the vocabulary does not have it.

---

*Consult [api-reference.md](api-reference.md) for complete listing*