Builds a serving that predicts the masked word of a text.

The input is a string with [MASK] once, or a list of them. [MASK] becomes the mask token of the tokenizer, such as <mask>. The serving raises when the encoded text does not have exactly one mask token, for example when truncation removes it. The result is %{predictions: [%{score: score, token: token}]}, from the highest score.

The model gives what its ONNX export computes. An Optimum export of BigBird, such as that of google/bigbird-roberta-base, traces full attention. Python transformers changes BigBird to block sparse attention for a text of more than 704 tokens, so a longer text can give other tokens than Python. A text of more than 512 tokens also needs a larger :sequence_length (see the module documentation).

Options

• :top_k - the number of predictions. The default is 5.

See the module documentation for the other options.

Builds a serving that generates text with a decoder, such as a chat model.

The input is a string, %{text: string}, %{messages: messages} or a list of them. Messages are maps with :role and :content, and they go through the chat template of the model (see Transmuter.ChatTemplate). A map input can also have :seed, and a chat can have :tools. The result is %{results: [%{text: text, token_summary: %{input: count, output: count, padding: 0}}]}. The text has only the new tokens.

The settings come from generation_config.json, and the options below override them. Generation stops at an eos_token_id of the configuration or after :max_new_tokens.

The default variant is :q4 when the repository has it, because the 4-bit weights of a decoder are smaller and faster.

Options

• :max_new_tokens - the maximum number of new tokens. The default is 256.

• :strategy - :greedy or :sampling. The default is :sampling when do_sample in the configuration is true, else :greedy.

• :num_beams - the number of beams of beam search. The default is num_beams of the configuration, else 1. As in Python transformers, :strategy does not change it, so a model with num_beams: 4 in its configuration, such as Marian, runs 4 beams by default. Beam search takes strategy: :greedy, and it does not take :stream. The result is the best hypothesis. See Transmuter.Generation.

• :length_penalty and :early_stopping - the settings of beam search, as in Python transformers. The length of a hypothesis to the power of :length_penalty divides its score. :early_stopping is true, false or :never. The defaults are those of the configuration, else 1.0 and false.

• :temperature, :top_k, :top_p and :repetition_penalty - as in Python transformers. The defaults are 1.0, 50, 1.0 and 1.0.

• :no_repeat_ngram_size, :min_length, :min_new_tokens, :forced_bos_token_id and :forced_eos_token_id - as in Python transformers. :min_length counts the prompt and the new tokens, and :min_new_tokens counts only the new tokens. The n-grams include the prompt. The defaults are those of the configuration, else no processor. Transmuter.Generation tells how each one counts the tokens of a row.

• :seed - the seed of sampling for inputs without their own seed. The default is a new seed for each input.

• :template_options - other variables of the chat template, such as enable_thinking: false for Qwen3.

• :token_logprobs - true adds :token_logprobs to each result: the log-probability of each new token, without the eos token, so the list has output values of the token summary. The log-probability is that of the distribution that the step selects from: after the processors, the temperature, top-k and top-p, and for beam search the step of the beam of the token. These are the values of compute_transition_scores of Python transformers with normalize_logits=True (see Transmuter.Generation.generate/5). Python also gives a value for the step of the eos token, which the text does not have. A stream gives the values in its done event, so with :stream the option needs stream_done: true. Without the option, generation does not compute the values. The default is false.

• :compile - :batch_size, the number of inputs that generate together. The default is 1. The serving does not use :sequence_length, because it does not pad or truncate the prompt.

• :stream - true makes the serving return a stream of text chunks. The default is false. With :stream, the input must be one input, not a list.

• :stream_done - true adds {:done, %{token_summary: summary}} at the end of the stream. With token_logprobs: true, the map also has :token_logprobs, with the values of all tokens of the stream. The default is false.

The other options are those of the module documentation.

Batches

With compile: [batch_size: 4], the serving runs up to 4 inputs in one loop. The inputs come from a list, or from concurrent calls to Nx.Serving.batched_run/2. The prompts can have different lengths. Each row stops at its own eos token. When a batch has fewer inputs, padding rows fill it. A larger batch gives more tokens per second in total, but each input gets its tokens more slowly.

Each row samples with its own key from its seed, so the random numbers of a row do not depend on the batch. With greedy selection, and with sampling from a seed, an input gives the same text in a batch as alone. A batch can change the rounding of the logits, so in rare cases a token can be different.

Transmuter.Generation tells how each mode of the KV cache keeps the rows apart. A decoder that grows its cache, such as Xenova/gpt2 or Gemma 3, runs the prompts of each length as a separate group, one group after the other. Each step of a group runs all rows of the batch. So a batch is faster for these models only when prompts have the same length.

Prompts of up to 4096 tokens go into the same batch. A longer prompt goes into a batch with prompts of the same power-of-two width.

Streaming

With stream: true, Nx.Serving.run/2 and Nx.Serving.batched_run/2 return a stream. Consume the stream in the process that called the serving:

serving = Transmuter.Text.generation({:hf, "HuggingFaceTB/SmolLM2-135M-Instruct"}, stream: true)

serving
|> Nx.Serving.run(%{messages: [%{role: "user", content: "Write a haiku."}]})
|> Enum.each(&IO.write/1)

The chunks come out as the tokens arrive, and they join to the text of a serving without :stream. The serving decodes the text as Bumblebee and TextStreamer of Python transformers do. A chunk ends before the last space of the pending text, at a newline, or after a CJK character. So a chunk never has a part of a character, and a word does not lose the space that its next token gives.

With a batch, a stream gets the tokens of its row as each step of the loop ends. For a decoder that grows its cache, the rows of a later group get their first token when the groups before it finish.

Builds a serving that finds the answer to a question in a context, with an extractive question answering model.

The input is %{question: question, context: context} or a list of them. The result is %{results: [%{text: text, start: start, end: end, score: score}]}. The text is a part of the context, and start and end are byte offsets in the context. The results are empty when no window has an answer in the context.

The search follows the question answering pipeline of Python transformers. The score of a span of context tokens is the start probability of its first token times the end probability of its last token. A long context becomes windows of the largest sequence length, which is 384 tokens by default. The windows overlap by 128 tokens. Each window is one row of :batch_size. Spans with the same text add their scores, also from different windows, and the text with the highest score is the answer. Transmuter.Text.QuestionAnswering gives all the steps.

Options

• :max_answer_length - the maximum number of tokens in the answer. The default is 15.

See the module documentation for the other options.

Builds a serving that classifies text, such as sentiment analysis.

The input is a string, a {text, text_pair} tuple for models that compare two texts (such as cross-encoders for reranking), or a list of them. The result is %{predictions: [%{label: label, score: score}]}, from the highest score.

The labels come from id2label in config.json. As in Python transformers, the scores are the logits when problem_type is "regression", a sigmoid when the model has one label or problem_type is "multi_label_classification", and else a softmax.

Options

• :top_k - the number of predictions, or nil for all. The default is 5.

• :scores_function - :softmax, :sigmoid or :none for the logits.

See the module documentation for the other options.

Builds a serving that turns text into an embedding.

The input is a string or a list of strings. The result is %{embedding: tensor}, or a list of them for a list.

When the graph has the output sentence_embedding, as the sentence-transformers exports of EmbeddingGemma do, the embedding is that output. It includes the pooling, the Dense layers and the normalization of the model. Set :output_pool or :embedding_processor to pool the token embeddings instead.

Else, a sentence-transformers model gives its pooling and normalization in 1_Pooling/config.json and modules.json. When the repository does not have these files, they come from the base_model in its model card. When the Hub denies the base model, such as a gated model without a token, the serving logs a warning and does not use it. Without these files, the embedding is the mean of the token embeddings.

Options

• :output_pool - :mean_pooling, :cls_token_pooling, :last_token_pooling, or nil for the token embeddings.

• :embedding_processor - :l2_norm or nil.

See the module documentation for the other options.

Builds a serving that ranks documents by their relevance to a query, with a cross-encoder or a reranker, as CrossEncoder.rank of sentence-transformers.

The input is %{query: query, documents: documents} or a list of them. Each document is one row of :batch_size. The result is %{results: [%{index: index, score: score}]}, from the highest score. index is the position of the document in documents.

The serving reads the sentence-transformers files of the model: config_sentence_transformers.json, modules.json, sentence_bert_config.json, the configuration of each module and chat_template.jinja. When the repository does not have a config_sentence_transformers.json with the model type CrossEncoder, these files come from the base_model in its model card, as in text_embedding/2. The text and the score follow these files:

• The text. A model takes the query and the document as a pair of texts, and the prompt goes before the query. A model takes them as chat messages when modality_config in sentence_bert_config.json has "message", or, for a model without these files, when the repository has a chat template. The messages have the roles "query" and "document", after the prompt as a "system" message, and the chat template adds the special tokens. The serving raises when the template does not put the query and the document in the text. When truncation cuts a text, the end of the template replaces its last tokens.

• The score. A sequence classification model, such as Xenova/ms-marco-MiniLM-L-6-v2, gives the logit of its one label. A causal language model with a LogitScore module, such as onnx-community/Qwen3-Reranker-0.6B-ONNX, gives the logit of the true token minus the logit of the false token, at the last token of the text. A causal language model without these files scores the tokens "yes" and "no" with the chat template of the repository.

The model gives the logits of all positions, because the ONNX exports have no input to keep only the last position. For Qwen3-Reranker, a row of 512 tokens has 512 × 151,669 logits of type f32, which is 311 MB. Keep :batch_size and :sequence_length small for these models. The default sequence lengths truncate a text at 512 tokens. sentence-transformers truncates Qwen3-Reranker only at 40,960 tokens, so a longer text gets a different score.

Options

• :prompt - the instruction before the query. The default is the prompt of default_prompt_name in config_sentence_transformers.json, else no prompt. A sequence classification model gets it at the start of the query, and a causal language model as the system message.

• :scores_function - :sigmoid or :none for the logits. The default is activation_fn in config_sentence_transformers.json, else sentence_transformers.activation_fn or sbert_ce_default_activation_function in config.json, else :sigmoid.

• :top_k - the number of results, or nil (the default) for all.

See the module documentation for the other options.

Builds a serving that generates text from a source text with an encoder-decoder model, such as T5, Marian, NLLB or M2M100 for translation.

The input is a string, %{text: string} or a list of them. A map input can also have :seed. The result is %{results: [%{text: text, token_summary: %{input: count, output: count, padding: count}}]}. padding is the number of pad tokens that the source text gets.

The repository must have onnx/encoder_model.onnx and onnx/decoder_model_merged.onnx, as the exports of Xenova and onnx-community. The default variant is fp32. The settings come from generation_config.json, as in generation/2, with beam search when the configuration has num_beams.

serving = Transmuter.Text.text_to_text({:hf, "Xenova/t5-small"})
Nx.Serving.run(serving, "translate English to German: The house is wonderful.")
#=> %{results: [%{text: "Das Haus ist wunderbar.", token_summary: %{input: 11, output: 5, padding: 21}}]}

NLLB and M2M100 translate between many languages. Give the language tokens of the tokenizer in :source_language_token and :target_language_token:

serving =
  Transmuter.Text.text_to_text({:hf, "Xenova/nllb-200-distilled-600M"},
    source_language_token: "eng_Latn",
    target_language_token: "fra_Latn"
  )

Options

• :source_language_token - the token of the language of the source text, such as "eng_Latn" for NLLB or "__en__" for M2M100. It replaces the language token in the template of the post-processor of tokenizer.json, as src_lang of the tokenizer in Python transformers. The default is the token of tokenizer.json, English for NLLB and M2M100.

• :target_language_token - the token of the language of the result, such as "fra_Latn" or "__fr__". Its id is :forced_bos_token_id, as tgt_lang of the translation pipeline of transformers. The default is no forced token.

• :compile - :sequence_length, as for the other text tasks. The serving pads the source text to the smallest length that fits. :batch_size is the number of inputs that generate together, as in generation/2. Inputs of the same sequence length go into one batch.

• :max_new_tokens - the maximum number of new tokens. The default is that of the text2text-generation pipeline of transformers (see Transmuter.Generation.max_new_tokens/3): 256 for T5, and max_length minus 1 for the start token when the configuration gives a max_length other than 20, such as 511 for Marian.

• :strategy, :num_beams, :length_penalty, :early_stopping, :temperature, :top_k, :top_p, :repetition_penalty, :no_repeat_ngram_size, :min_length, :min_new_tokens, :forced_bos_token_id, :forced_eos_token_id, :seed, :token_logprobs, :stream and :stream_done - as in generation/2. The prompt of the decoder is decoder_start_token_id, so :min_length counts it, and the forced bos token is the first new token.

The other options are those of the module documentation, but :model does not apply.

Builds a serving that labels the tokens of a text, such as named entity recognition.

The input is a string or a list of strings. The result is %{entities: [%{start: start, end: end, label: label, score: score, phrase: phrase}]}. start and end are byte offsets in the text.

Options

• :aggregation - how tokens become entities, as in Python transformers:

  • nil (the default) - each token is an entity.
  • :same - adjacent tokens with the same label become one entity. A B- label begins a new entity.
  • :word_first, :word_average and :word_max - the tokens of a word first get one label: the label of the first token, the label of the average scores, or the label of the token with the highest score. Then adjacent words group as with :same.

• :ignored_labels - labels to leave out. The default is ["O"].

See the module documentation for the other options.

Builds a serving that classifies text with labels that you give, with a natural language inference (NLI) model.

The model compares the text with one hypothesis for each label, such as "This example is sports.". The score of a label comes from the entailment class of the model. The input is a string or a list of strings. The result is %{predictions: [%{label: label, score: score}]}, from the highest score.

The entailment class is the class in id2label of config.json that starts with "entail". As in Python transformers, the contradiction class is the last class when the entailment class is the first, else the first class. Each label adds one sequence to the batch, so a text with 5 labels uses 5 rows of :batch_size.

Options

• :hypothesis_template - a function that takes a label and returns its hypothesis. The default is &"This example is #{&1}.".

• :multi_label - false (the default) gives scores that add up to 1: a softmax of the entailment logits over the labels. true scores each label alone: a softmax over the contradiction and entailment logits of the label. A single label always gets the score of true, as in Python transformers.

• :top_k - the number of predictions, or nil for all. The default is 5.

See the module documentation for the other options.

Builds a serving that finds entities of the labels that you give, with a GLiNER model, such as onnx-community/gliner_small-v2.1.

The input is %{text: text, labels: labels} or a list of them. Each input can have its own labels. The result is %{entities: [%{start: start, end: end, label: label, score: score, phrase: phrase}]}, in the order of the text. start and end are byte offsets in the text, and phrase is the text of the entity. gliner in Python gives character offsets.

The serving follows gliner 0.2.29 (see Transmuter.Text.ZeroShotEntityRecognition). It reads gliner_config.json, and it runs the span models and the token models (span_mode "token_level") of GLiNER. Models with a labels encoder, a labels decoder or relations, streaming models, and other word splitters than "whitespace" raise. As gliner, the serving keeps the first max_len words of the text. It also keeps only the words that fit in the largest sequence length, which is 512 tokens by default. gliner does not truncate the tokens, so a longer text gets other entities. A full batch runs one more row, so that new texts do not compile the model again. With the default batch size of 1, each request runs 2 rows. The module documentation tells why.

Options

• :threshold - an entity needs a score more than this value. The default is 0.5.

• :flat_ner - true (the default) removes an entity that overlaps an entity with a higher score. false removes it only when the two entities overlap partly, so an entity can be inside another entity.

• :multi_label - true lets a span of words have more than one label. The default is false.

See the module documentation for the other options.