Pipelines

This guide shows how to run text models with the servings of Transmuter.Text, image models with the servings of Transmuter.Vision, and speech models with the servings of Transmuter.Audio. A serving takes text, images or audio and gives results. It does the tokenization or the preprocessing, the batching and the steps after the model, such as pooling and softmax.

Installation

The servings need the tokenizers package:

def deps do
  [
    {:transmuter, "~> 1.0"},
    {:emlx, "~> 0.5"},
    {:tokenizers, "~> 0.5"}
  ]
end

Chat templates need {:otera, "~> 0.1"}. Otera renders Jinja templates in pure Elixir, with the same output as apply_chat_template of Python transformers.

The examples in this guide do not show the backend options. Without them, the model runs on Nx.BinaryBackend, which is very slow. Add backend: EMLX.Backend, defn_options: [compiler: EMLX] on Apple silicon, or the EXLA options on other machines. See Backends and performance.

What a serving loads

A serving loads a Hugging Face repository: the ONNX file, tokenizer.json, and the configuration files. You do not give a file name:

serving = Transmuter.Text.text_classification({:hf, "Xenova/distilbert-base-uncased-finetuned-sst-2-english"})

A local directory with the same files also works:

serving = Transmuter.Text.text_classification("/models/sst2")

ONNX exports keep each file in variants, for example onnx/model.onnx (fp32) and onnx/model_q4.onnx (4-bit weights). The serving selects a variant in this order:

1. The :variant option, such as variant: :q4.
2. The dtype in the transformers.js_config block of config.json.
3. The default of the task: fp32 for encoders and :q4 for generation, when the repository has the file.
4. fp32, then the other variants.

Tasks

Each function also takes a list of inputs and then gives a list of results. The results have the shapes of Bumblebee. Bumblebee has no text-to-speech, so text_to_speech/2 gives the shape of the pipeline of Python transformers. Their values match the pipelines of Python transformers and sentence-transformers, and the tests of Transmuter check this for each task. The audio of text-to-speech depends on a random seed, so the tests compare the audio without noise.

Embeddings

serving = Transmuter.Text.text_embedding({:hf, "Xenova/bge-small-en-v1.5"})
%{embedding: embedding} = Nx.Serving.run(serving, "The cat sat on the mat.")

A sentence-transformers model gives its pooling, its normalization and its maximum length in 1_Pooling/config.json, modules.json and sentence_bert_config.json. ONNX exports often do not have these files. The serving then reads them from the base_model in the model card. For example, Xenova/bge-small-en-v1.5 gets CLS pooling and L2 normalization from BAAI/bge-small-en-v1.5. Without these files, the embedding is the mean of the token embeddings.

The serving raises an error when modules.json has a module that Transmuter does not run, such as a Dense layer. Set :output_pool and :embedding_processor to use the model without these modules.

Classification and reranking

serving = Transmuter.Text.text_classification({:hf, "Xenova/ms-marco-MiniLM-L-6-v2"})

Nx.Serving.run(serving, [
  {"How many people live in Berlin?", "Berlin has a population of 3,520,031 registered inhabitants."},
  {"How many people live in Berlin?", "New York City is famous for the Metropolitan Museum of Art."}
])

The labels come from id2label in config.json. The scores are a softmax, a sigmoid for a model with one label or with multi-label classification, and the logits for regression.

Reranking

serving = Transmuter.Text.text_ranking({:hf, "Xenova/ms-marco-MiniLM-L-6-v2"})

Nx.Serving.run(serving, %{
  query: "How many people live in Berlin?",
  documents: [
    "New York City is famous for the Metropolitan Museum of Art.",
    "Berlin has a population of 3,520,031 registered inhabitants."
  ]
})
#=> %{results: [%{index: 1, score: 8.76}, %{index: 0, score: -11.25}]}

The scores are those of CrossEncoder.rank of sentence-transformers. The serving reads the sentence-transformers files of the model, or of the base_model in the model card, as for embeddings. A causal language model with a LogitScore module, such as onnx-community/Qwen3-Reranker-0.6B-ONNX, scores the text of its chat template with the logits of "yes" and "no". These models give the logits of all positions, so use a small :batch_size and :sequence_length.

Zero-shot classification

serving =
  Transmuter.Text.zero_shot_classification(
    {:hf, "Xenova/distilbert-base-uncased-mnli"},
    ["urgent", "phone", "tablet"]
  )

Nx.Serving.run(serving, "My iphone is broken and I need it fixed today!")

You give the labels when you build the serving. A natural language inference (NLI) model compares the text with the hypothesis of each label, such as "This example is phone.". Set :hypothesis_template to change the hypothesis. The scores add up to 1. With multi_label: true, the model scores each label alone, so more than one label can have a high score.

Each label is one row of the batch. A text with 5 labels uses 5 rows of :batch_size.

Question answering

serving = Transmuter.Text.question_answering({:hf, "Xenova/distilbert-base-cased-distilled-squad"})

Nx.Serving.run(serving, %{
  question: "Where is the Eiffel Tower?",
  context: "The Eiffel Tower is a wrought-iron lattice tower in Paris."
})
#=> %{results: [%{text: "Paris", start: 52, end: 57, score: ...}]}

The answer is a part of the context, and start and end are byte offsets in the context. A long context becomes windows of 384 tokens that overlap by 128 tokens, as in Python transformers. A different :sequence_length in :compile changes the length of the windows. Each window is one row of the batch. Spans with the same text add their scores, also from different windows, and the text with the highest score is the answer.

Named entities

serving = Transmuter.Text.token_classification({:hf, "Xenova/bert-base-NER"}, aggregation: :same)
Nx.Serving.run(serving, "Hugging Face Inc. is a company based in New York City.")
#=> %{entities: [%{label: "ORG", phrase: "Hugging Face Inc", start: 0, end: 16, ...}, ...]}

start and end are byte offsets in the text, so binary_part(text, start, end - start) gives the span.

Entities of your labels

serving = Transmuter.Text.zero_shot_entity_recognition({:hf, "onnx-community/gliner_small-v2.1"})

Nx.Serving.run(serving, %{
  text: "Cristiano Ronaldo plays for Al Nassr.",
  labels: ["person", "organization"]
})
#=> %{entities: [%{label: "person", phrase: "Cristiano Ronaldo", start: 0, end: 17, ...}, ...]}

A GLiNER model finds the entities of the labels that you give, and each input can have its own labels. The steps follow gliner in Python. Span models and token models (such as knowledgator/gliner-pii-small-v1.0) both run. :threshold sets the lowest score, flat_ner: false keeps an entity inside another entity, and multi_label: true lets a span have more than one label.

Generation

serving =
  Transmuter.Text.generation({:hf, "onnx-community/Qwen3-0.6B-ONNX"},
    max_new_tokens: 512,
    template_options: [enable_thinking: false]
  )

Nx.Serving.run(serving, %{messages: [%{role: "user", content: "Write a haiku about Elixir."}]})

Messages go through the chat template of the repository (see Transmuter.ChatTemplate). A string input does not. The settings of generation_config.json apply, such as sampling with temperature, top_k and top_p. The options override them. Give :seed in the input or as an option to get the same text again.

The logits processors of Python transformers that read the tokens of a row also apply: no_repeat_ngram_size, min_length, min_new_tokens, forced_bos_token_id and forced_eos_token_id. They count the tokens as Python does. min_length and the n-grams include the prompt, and min_new_tokens counts only the new tokens. Transmuter.Generation gives the rule of each processor.

The serving works with any decoder export that has a KV cache. It reads the shape of the cache from the graph:

• Exports with GroupQueryAttention that counts the valid rows from the values of the attention mask, such as the onnx-community exports of Qwen3 and SmolLM2, write the cache in place.
• Exports that declare position_ids keep the cache in a buffer with masked free rows.
• Merged decoders without position_ids, such as Xenova/gpt2, grow the cache by one row for each token. Gemma 3 also grows the cache, because its GroupQueryAttention counts the valid rows from the length of the attention mask. Each cache length is a new shape, which compiles on its first step and stays cached. So the first request is slower. With Xenova/gpt2 and 64 new tokens on an Apple silicon GPU, the first request takes 2.9 s and later requests take 0.4 s.

Transmuter.Generation describes the three modes of the cache.

The prompt runs in parts with a power-of-two length, and each new token is a part of length 1. So a small number of compiled shapes covers all prompt lengths.

Streaming

Set stream: true to get the text in chunks as the tokens arrive:

serving = Transmuter.Text.generation({:hf, "HuggingFaceTB/SmolLM2-135M-Instruct"}, stream: true)

serving
|> Nx.Serving.run(%{messages: [%{role: "user", content: "Write a haiku about Elixir."}]})
|> Enum.each(&IO.write/1)

The serving returns a stream of strings. Consume the stream in the process that called the serving. The chunks join to the text of a serving without :stream. The serving decodes the text with the rule of Bumblebee and of TextStreamer in Python transformers:

• A chunk ends before the last space of the pending text, so a token that changes the space before it does not change a chunk that is already out.
• A newline or a CJK character ends a chunk at once.
• A character of several byte-level tokens, such as an emoji, comes out only when all of its tokens are there.

Set stream_done: true to get {:done, %{token_summary: summary}} after the last chunk. With :stream, the input must be one input, not a list. text_to_text/2, Transmuter.Vision.image_to_text/2 and Transmuter.Vision.image_text_to_text/2 take the same options. Transmuter.Audio.speech_to_text/2 does not stream, because it merges the windows of long audio only at the end.

Batches

Set compile: [batch_size: n] to generate for up to n inputs in one loop. The inputs come from a list, or from concurrent calls to Nx.Serving.batched_run/2. Each step of the loop runs the model once for all rows. On a GPU, a batch gives more tokens per second in total:

The table shows the total tokens per second of 16 concurrent requests to a started serving with batch_timeout: 10, with greedy selection and up to 128 new tokens for each request. The values are the median of 3 or more runs with EMLX on an Apple M2 Pro GPU. A batch runs until its longest row stops, so the gain is smaller when the lengths of the answers are very different.

The prompts of a batch can have different lengths. Each row has its own length in the KV cache, its own positions, its own sampling key and its own stop at the eos token. A row that stops gets padding tokens until the batch ends. When fewer inputs than n arrive, padding rows fill the batch. With greedy selection, and with sampling from a seed, each input gives the same text as when it runs alone.

The mode of the KV cache decides how the rows share the cache:

• When GroupQueryAttention writes the cache in place, the attention mask of each row tells the operator how many rows of the cache are valid. The operator writes the new keys and values of each row after its valid rows.
• With position_ids, the attention mask hides the free rows of each row, and the new rows go into the buffer after the valid rows of each row.
• A decoder that grows its cache gets its positions from the length of the cache. That length is the same for all rows, so only prompts of the same length can share a cache. The serving runs a group for each prompt length, one group after the other, and each step of a group runs all rows of the batch. So a batch is faster only for the prompts that have the same length. For 16 prompts with 7 different lengths, Xenova/gpt2 gave 133 tokens/s with batch_size: 1 and 148 tokens/s with batch_size: 8. The prompts of text-to-text and speech-to-text have one length, so all their rows run as one group. With streaming, the rows of a later group get their first token when the groups before it finish.

Prompts of up to 4096 tokens go into the same batch. A longer prompt goes into a batch with prompts of the same power-of-two width.

Beam search

Set :num_beams to keep the best n sequences at each step and return the best one at the end:

serving = Transmuter.Text.generation({:hf, "HuggingFaceTB/SmolLM2-135M-Instruct"}, num_beams: 3)

The default is num_beams of generation_config.json, else 1, as in Python transformers. :strategy does not change it, so a model with num_beams in its configuration, such as Marian, runs beam search by default. Give num_beams: 1 for greedy selection. :length_penalty and :early_stopping work as in Python transformers, and their defaults also come from the configuration. Beam search takes greedy selection, not sampling, and it does not stream, because a later step can change the earlier tokens of the best sequence.

The beams of an input are rows of the batch, so the decoder compiles for batch_size * num_beams rows. The encoder of an encoder-decoder model runs once for each input. The text is the same as the generate function of Python transformers 4.57 with the same num_beams. All tasks that generate text take these options.

Translation and other text-to-text tasks

serving = Transmuter.Text.text_to_text({:hf, "Xenova/opus-mt-en-de"})
Nx.Serving.run(serving, "The house is wonderful.")

The serving runs an encoder-decoder model, such as T5 or Marian. The repository must have onnx/encoder_model.onnx and onnx/decoder_model_merged.onnx. The serving does these steps:

1. It encodes the text and pads it to a length of :compile, as the other text tasks.
2. It runs the encoder once.
3. It starts the decoder from decoder_start_token_id. The first step of the decoder computes the cross-attention cache from the outputs of the encoder. The later steps use this cache again.

T5 takes the task in the text, for example "translate English to German: The house is wonderful.". A Marian model translates one language pair, and its tokenizer encodes the text with source.spm, as MarianTokenizer of Python transformers.

The settings of generation_config.json apply, as for generation/2. As in Python transformers, num_beams of the configuration gives beam search, so Marian runs 4 beams by default. The text is the same as the generate function of transformers with the same num_beams.

NLLB and M2M100

serving =
  Transmuter.Text.text_to_text({:hf, "Xenova/nllb-200-distilled-600M"},
    source_language_token: "eng_Latn",
    target_language_token: "fra_Latn"
  )

Nx.Serving.run(serving, "The house is wonderful.")

NLLB and M2M100 translate between many languages, and each language has a token in the tokenizer: "eng_Latn" and "fra_Latn" for NLLB, "__en__" and "__fr__" for M2M100. The serving does what the translation pipeline of transformers does with src_lang and tgt_lang:

1. The source language token goes at the start of the text, in the place of the language token in the template of the post-processor of tokenizer.json. The text then ends with </s>.
2. The id of the target language token is forced_bos_token_id, so the first new token of the decoder is the target language token.

Without :source_language_token, the text gets the language token of tokenizer.json, which is English. Without :target_language_token, the decoder does not get a forced token.

Speech recognition

audio = Nx.from_binary(File.read!("speech.f32"), :f32)
serving = Transmuter.Audio.speech_to_text({:hf, "onnx-community/whisper-base"}, language: "en")
Nx.Serving.run(serving, audio)
#=> %{chunks: [%{text: " Mr. Quilter is the apostle of the middle classes, ...", ...}]}

Transmuter.Audio.speech_to_text/2 runs Whisper on PCM samples at 16 kHz. For an audio file, give {:file, path}: ffmpeg decodes it and changes its sampling rate to 16 kHz. The ffmpeg executable must be in the PATH.

Nx.Serving.run(serving, {:file, "speech.mp3"})

The input can also be a stream of tensors of PCM samples, such as the parts of a long recording. The serving transcribes the windows of the stream when they are complete, and gives the result of the whole audio when the stream ends. See Transmuter.Audio for the inputs.

The audio can have any length. The serving splits audio longer than 30 s into windows that overlap, as the pipeline of transformers with chunk_length_s=30. It transcribes each window and merges the texts at the overlaps. The :chunk_num_seconds and :context_num_seconds options set the windows.

With compile: [batch_size: n], the decoder transcribes up to n windows together. The windows come from one long audio, from a list of audio, or from concurrent calls. For 62 s of audio, Xenova/whisper-tiny.en takes 732 ms with batch_size: 4 and 1040 ms with batch_size: 1.

With timestamps: :segments, the result has one chunk for each segment of speech, with its start and end in seconds:

serving = Transmuter.Audio.speech_to_text({:hf, "Xenova/whisper-tiny.en"}, timestamps: :segments)
Nx.Serving.run(serving, audio)
#=> %{chunks: [%{text: " Mr. Quilter is ...", start_timestamp_seconds: 0.0, end_timestamp_seconds: 50.44}, ...]}

With timestamps: :words, the result has one chunk for each word, as return_timestamps="word" in transformers. The decoder must give its cross attentions, as the decoder of Xenova/whisper-tiny.en and the _timestamped exports of onnx-community do:

serving = Transmuter.Audio.speech_to_text({:hf, "Xenova/whisper-tiny.en"}, timestamps: :words)
Nx.Serving.run(serving, audio)
#=> %{chunks: [%{text: " Mr.", start_timestamp_seconds: 0.0, end_timestamp_seconds: 0.96}, ...]}

See Speech and audio for the details.

Audio classification

audio = Nx.from_binary(File.read!("speech.f32"), :f32)
serving = Transmuter.Audio.audio_classification({:hf, "Xenova/ast-finetuned-audioset-10-10-0.4593"})
Nx.Serving.run(serving, audio)
#=> %{predictions: [%{label: "Speech", score: 0.84}, %{label: "Speech synthesizer", score: 0.025}, ...]}

Transmuter.Audio.audio_classification/2 runs AST and wav2vec2 classifiers on PCM samples at the sampling rate of the model. Transmuter.Audio.Featurizer prepares the audio as the feature extractor of Python transformers, which feature_extractor_type in preprocessor_config.json selects. The scores are a softmax of the logits, as in the audio-classification pipeline of Python transformers.

AST takes 1024 frames of log-mel features, about 10 s of audio. The serving cuts longer audio and pads shorter audio, so the audio goes into batches of compile: [batch_size: n]. wav2vec2 takes the mean over all frames of the audio, so padding would change the result. The serving runs each audio alone, and the model compiles each new audio length on its first call.

The tests compare the labels and the scores with the pipeline of Python transformers for AST, and with ONNX Runtime for onnx-community/wav2vec2-base-Speech_Emotion_Recognition-ONNX, whose PyTorch model is no longer on the Hub. The labels are the same, and the scores agree within 1.0e-4.

Zero-shot audio classification

audio = Nx.from_binary(File.read!("speech_48k.f32"), :f32)
labels = ["a man speaking", "music", "a dog barking", "rain"]
serving = Transmuter.Audio.zero_shot_audio_classification({:hf, "Xenova/clap-htsat-unfused"}, labels)
Nx.Serving.run(serving, audio)
#=> %{predictions: [%{label: "a man speaking", score: 0.95}, %{label: "music", score: 0.047}, ...]}

Transmuter.Audio.zero_shot_audio_classification/3 runs CLAP on PCM samples at 48 kHz, the sampling rate of the model. For an audio file at another sampling rate, give {:file, path}, and ffmpeg changes the sampling rate to 48 kHz. Each label goes into the hypothesis template "This is a sound of {}.", and the scores are a softmax of logits_per_audio over the labels, as in the zero-shot-audio-classification pipeline of Python transformers.

Transmuter.Audio.Featurizer gives the log-mel features of ClapFeatureExtractor. CLAP takes 10 s of audio. Shorter audio repeats and gets zeros at the end, so the audio goes into batches of compile: [batch_size: n]. For longer audio, Python transformers takes 10 s at a random position, and the serving takes the first 10 s.

The tests compare the features with ClapFeatureExtractor within 1.0e-3 dB. With the fp32 file, the labels are the same as those of the pipeline of Python transformers, and the scores agree within 1.0e-4. The q8 file quantizes the activations when the model runs, and its scores are up to 0.13 away from the fp32 model. Its scores are within 0.007 of ONNX Runtime on the same file on the EXLA CPU, and within 0.047 on CUDA. The quantization makes small differences large: when the features change by 1e-5 of their values, the scores of ONNX Runtime change by up to 0.014.

Speech synthesis

serving = Transmuter.Audio.text_to_speech({:hf, "Xenova/mms-tts-eng"})
Nx.Serving.run(serving, %{text: "Hello, my dog is cute.", seed: 42})
#=> %{audio: #Nx.Tensor<f32[31744]>, sampling_rate: 16000}

Transmuter.Audio.text_to_speech/2 turns text into PCM samples. Write them to a file with File.write!("speech.f32", Nx.to_binary(audio)), and convert the file with ffmpeg -f f32le -ar 16000 -ac 1 -i speech.f32 speech.wav. The serving runs two model types:

• VITS, such as the MMS-TTS models of Meta (Xenova/mms-tts-eng), makes the audio in one pass.
• SpeechT5 (Xenova/speecht5_tts) makes a spectrogram step by step and turns it into audio with a vocoder. It needs a speaker embedding of 512 values in the :speaker_embedding option or in the input.

speaker = Nx.from_binary(File.read!("speaker.f32"), :f32)
serving = Transmuter.Audio.text_to_speech({:hf, "Xenova/speecht5_tts"}, speaker_embedding: speaker)
Nx.Serving.run(serving, %{text: "Hello, my dog is cute.", seed: 42})
#=> %{audio: #Nx.Tensor<f32[25088]>, sampling_rate: 16000}

Both models are random, also in Python. The seed of an input decides the audio, and an input without a seed gets a random seed. With noise: false, the random operators give fixed values. VITS then gives the audio of Python transformers with noise_scale and noise_scale_duration at 0, and the decoder of SpeechT5 keeps all values in its dropout. The tests compare this audio with PyTorch: the length is the same, and the samples are within 1e-4.

The serving runs each text alone, because the length of the audio depends on the text. Kokoro reads phonemes, not text, so the serving does not run it. See Speech and audio for the details.

Images

The servings of Transmuter.Vision take a StbImage struct or a tensor of shape {height, width, channels} of type :u8. To read image files, add {:stb_image, "~> 1.0"} to your deps:

image = StbImage.read_file!("cats.jpg")

Each function also takes a list of images and then gives a list of results. The tests of Transmuter compare each task with the pipeline of Python transformers on the same image. The labels are the same, and the scores agree within 1.0e-4. For image_to_text/2, the text is the same as the text of the image-to-text pipeline in Python transformers. For Florence-2, the text is the same as the beam search of Python transformers on the image features of the export, and the coordinates are those of post_process_generation before it truncates them to integers. For image_text_to_text/2, the text is the same as the text of ONNX Runtime on the same ONNX files. For object_detection/2, the labels are the same as those of post_process_object_detection in Python transformers, the scores agree within 2.0e-5, and the boxes agree within 0.001 pixels. For zero_shot_object_detection/3, the labels are the same as those of the zero-shot-object-detection pipeline, and the boxes agree within 0.011 pixels. The scores of OWL-ViT and OWLv2 agree within 1.1e-4. The reference of Grounding DINO is ONNX Runtime with the post-processing of the pipeline, and its scores agree within 6.0e-4. For depth_estimation/2, the depth agrees within 1.0e-4 relative, and a value of the 8-bit depth image can be 1 step away. For image_segmentation/2, the labels and the masks are the same as those of the image-segmentation pipeline. For background_removal/2, a value of the mask can be 1 step away from ONNX Runtime with the post-processing of transformers.js and the bilinear resize of Pillow. For mask_generation/2, the masks are those of SamModel and Sam2Model of Python transformers with post_process_masks, except for at most a few pixels at the edges, where the logits are near 0, and the scores agree within 1.0e-4. The PyTorch weights of SAM 3 are gated, so the reference of the SAM 3 tracker is ONNX Runtime on the export, with the processor of Python transformers 5.

Image preprocessing

Transmuter.Featurizer reads preprocessor_config.json and prepares the images as the image processors of Python transformers do. It converts the image to RGB, resizes it, crops the center, rescales the values and normalizes them. The result is a tensor of shape {batch, 3, height, width} and type :f32.

The resize gives the same pixels as Pillow, for all resample filters of Pillow. With the hamming filter, a small number of pixels can differ by 1. A fast image processor, whose image_processor_type ends in Fast, resizes with torchvision in Python, so the featurizer uses the weights of torchvision for it. These weights have fewer bits, and a pixel can be one step away from Pillow when the image gets smaller. The scores of a model change when the pixels change, so this match is important. The serving resizes on the :backend of the task, because the resize is slow on Nx.BinaryBackend.

The serving compiles the model for one image size, which comes from size or crop_size in preprocessor_config.json. When the size depends on the image, for example with shortest_edge and no crop, the serving runs each image alone, and the model compiles each new image size on its first call. Then the batch size must be 1. image_to_text/2 raises an error for these sizes.

Image classification

serving = Transmuter.Vision.image_classification({:hf, "Xenova/vit-base-patch16-224"})
Nx.Serving.run(serving, image)
#=> %{predictions: [%{label: "Egyptian cat", score: 0.94}, ...]}

The labels come from id2label in config.json.

Image embeddings

serving = Transmuter.Vision.image_embedding({:hf, "Xenova/clip-vit-base-patch32"})
%{embedding: embedding} = Nx.Serving.run(serving, image)

For CLIP and SigLIP, the serving uses vision_model.onnx, and the embedding is the image_embeds output. For other models, the embedding is pooler_output, or else the first token of last_hidden_state. Set embedding_processor: :l2_norm to normalize the embedding.

Zero-shot image classification

labels = ["cats", "dogs", "a remote control"]
serving = Transmuter.Vision.zero_shot_image_classification({:hf, "Xenova/clip-vit-base-patch32"}, labels)
Nx.Serving.run(serving, image)
#=> %{predictions: [%{label: "cats", score: 0.98}, ...]}

The serving uses model.onnx, which has the text tower, the image tower, and the logit scale and bias of the model. vision_model.onnx and text_model.onnx do not have the logit scale and bias. Each label goes into the template "This is a photo of {}.", as in Python transformers. Use :hypothesis_template to change it.

The scores are a softmax over the labels for CLIP. For SigLIP, the scores are a sigmoid of each label, so they do not add up to 1. The SigLIP tokenizer also removes the punctuation of the text and makes it lowercase, as in Python transformers.

Image to text

serving = Transmuter.Vision.image_to_text({:hf, "Xenova/vit-gpt2-image-captioning"})
Nx.Serving.run(serving, image)
#=> %{results: [%{text: "a cat laying on a blanket next to a cat laying on a bed "}]}

The serving runs encoder-decoder exports of VisionEncoderDecoderModel, such as ViT-GPT2 for captions and TrOCR for handwritten text. The encoder reads the image once, and the decoder generates the text. The settings come from generation_config.json, so ViT-GPT2 and TrOCR use greedy selection. When the configuration has no max_length, the text has at most 256 new tokens, as in the image-to-text pipeline of Python transformers. Use :max_new_tokens to change this limit. The serving takes the options of generation, such as :compile, :stream and :strategy.

The serving also runs Florence-2, whose encoder reads the image features and the embeddings of a prompt. Give the task in the input:

serving = Transmuter.Vision.image_to_text({:hf, "onnx-community/Florence-2-base-ft"})
Nx.Serving.run(serving, %{image: image, task: "<CAPTION_TO_PHRASE_GROUNDING>", text: "two cats"})
#=> %{results: [%{text: "two cats<loc_11>...", boxes: [%{xmin: 3.68, ymin: 11.16, xmax: 319.52, ymax: 235.32}, ...], labels: ["two cats", ...]}]}

A task is a task token of preprocessor_config.json, such as "<CAPTION>", "<OD>" or "<OCR_WITH_REGION>". The default is "<CAPTION>", and the :task option changes it. The result has the keys of post_process_generation of Python transformers: :boxes (as in object_detection/2), :quad_boxes or :polygons, and :labels. <OPEN_VOCABULARY_DETECTION> gives :boxes, :box_labels, :polygons and :polygon_labels. The coordinates are floats in pixels of the input image. The generation config of Florence-2 does beam search with 3 beams. The vision encoder of onnx-community/Florence-2-base-ft gives other features than PyTorch, so the text can differ from the PyTorch model (see the README).

Object detection

serving = Transmuter.Vision.object_detection({:hf, "Xenova/detr-resnet-50"})
Nx.Serving.run(serving, image)
#=> %{detections: [%{label: "remote", score: 0.997, box: %{xmin: 19.9, ymin: 35.1, xmax: 87.3, ymax: 59.2}}, ...]}

The serving runs DETR, YOLOS, RT-DETR and RF-DETR exports, such as Xenova/detr-resnet-50, Xenova/yolos-tiny and onnx-community/rfdetr_nano-ONNX. The boxes are floats in pixels of the input image. The pipeline of Python transformers truncates them to integers. The serving keeps the detections with a score higher than threshold. The default threshold is 0.5, as in Python transformers.

model_type in config.json selects the scores. DETR and YOLOS take the softmax over the classes, without the last class, which is "no object". Each query gives one detection, in the order of the queries. RT-DETR and RF-DETR take a sigmoid of each pair of a query and a class, and keep the :top_k pairs with the highest scores. The default :top_k is the number of queries.

DETR and YOLOS resize the short side to shortest_edge, so each image has its own size. The serving runs each image alone, and the model compiles each new image size on its first call. Use batch_size: 1 with these models. RF-DETR resizes each image to 384x384, so its images go into batches.

Zero-shot object detection

labels = ["cat", "remote control", "couch"]
serving = Transmuter.Vision.zero_shot_object_detection({:hf, "Xenova/owlv2-base-patch16-ensemble"}, labels)
Nx.Serving.run(serving, image)
#=> %{detections: [%{label: "remote control", score: 0.747, box: %{xmin: 167.3, ymin: 38.5, xmax: 185.0, ymax: 93.7}}, ...]}

The serving runs OWL-ViT, OWLv2 and Grounding DINO exports, such as Xenova/owlvit-base-patch32, Xenova/owlv2-base-patch16-ensemble and onnx-community/grounding-dino-tiny-ONNX. You give the labels when you build the serving. As in the zero-shot-object-detection pipeline of Python transformers, the score of a box for a label is the sigmoid of the highest logit of the box. The serving keeps each box with a score higher than threshold for each label, so one box can be a detection of more than one label. The default threshold is 0.1. The boxes are floats in pixels of the input image.

OWL-ViT and OWLv2 score each text query alone, so the serving runs all labels as the text queries of one model run, and the images go into batches. The scores are the same as with one run for each label, as in Python. OWLv2 pads the image to a square with gray before it resizes it. Its resize is the one of scikit-image: a gaussian blur when the image gets smaller, then a linear interpolation.

Grounding DINO reads the text and the image together, so the model runs once for each label and image. The ONNX exports of Grounding DINO compute the phrases of the text for one phrase and a period, so the text of a label is the label in lowercase and a period, as GroundingDinoProcessor of Python transformers makes it for a list of labels. The pipeline of Python transformers with the PyTorch model gives the label alone, so its detections are different. A label cannot have "." or "?", because these characters end a phrase. The scores of Grounding DINO change by up to 5.0e-4 when the pixel values change by 1.0e-7, also in ONNX Runtime, because the model keeps 900 of 13294 boxes with TopK.

Depth estimation

serving = Transmuter.Vision.depth_estimation({:hf, "onnx-community/depth-anything-v2-small"})
Nx.Serving.run(serving, image)
#=> %{predicted_depth: #Nx.Tensor<f32[240][320]>, depth: #Nx.Tensor<u8[240][320]>}

The serving runs Depth Anything exports, such as onnx-community/depth-anything-v2-small and Xenova/depth-anything-small-hf. predicted_depth is the output of the model, resized to the size of the image with the bicubic interpolation of PyTorch, as in the depth-estimation pipeline of Python transformers. depth is the same map as an 8-bit gray image: the nearest pixel is 255 and the farthest pixel is 0.

The DPT image processor keeps the aspect ratio and makes each side a multiple of 14, so a 320x240 image gives pixel values of 518x392. The size depends on the image, so the serving runs each image alone, and the model compiles each new image size on its first call.

The ONNX file of Xenova/depth-anything-small-hf gives a depth up to 0.9% away from its PyTorch model, also in ONNX Runtime. Transmuter gives the depth of ONNX Runtime.

Image segmentation

serving = Transmuter.Vision.image_segmentation({:hf, "Xenova/segformer-b0-finetuned-ade-512-512"})
Nx.Serving.run(serving, image)
#=> %{segments: [%{label: "wall", score: nil, mask: #Nx.Tensor<u8[240][320]>}, ...]}

The serving runs semantic segmentation models, such as the Segformer exports of Xenova. As in the image-segmentation pipeline of Python transformers, the logits are resized to the size of the image with the bilinear interpolation of PyTorch, and each pixel gets the label with the highest logit. Each label that is present gives one segment, in the order of the label ids. A mask has 255 for the pixels of its label and 0 for the other pixels. The score is nil, because semantic segmentation gives no scores. Panoptic and instance segmentation models, such as DETR and Mask2Former, are not supported yet.

Background removal

serving = Transmuter.Vision.background_removal({:hf, "onnx-community/ormbg-ONNX"})
%{mask: mask} = Nx.Serving.run(serving, image)

Python transformers has no pipeline for this task, so the serving follows the background-removal pipeline of transformers.js. The mask is the first output of the model, with a sigmoid when a value is outside 0..1, as 8 bits. The bilinear resize of Pillow gives the mask the size of the image. transformers.js resizes with sharp in Node, which does not filter when the image gets smaller, so its mask can be different at the edges of the foreground. The ONNX file of onnx-community/ormbg-ONNX has a fixed batch size of 1. See Transmuter.Vision.background_removal/2 for an example that makes an image with an alpha channel.

Mask generation

serving = Transmuter.Vision.mask_generation({:hf, "onnx-community/sam2.1-hiera-tiny-ONNX"})

prompts = [
  %{points: [[80, 120]]},
  %{points: [[80, 120], [150, 200]], labels: [1, 0]},
  %{box: [40, 60, 200, 230]}
]

%{masks: masks, iou_scores: scores} = Nx.Serving.run(serving, %{image: image, prompts: prompts})

The serving runs the SAM, SAM 2 and SAM 3 tracker exports of transformers.js, such as Xenova/slimsam-77-uniform, onnx-community/sam2.1-hiera-tiny-ONNX and onnx-community/sam3-tracker-ONNX, with prompts of points and boxes. A point with the label 1 is on the object, and a point with the label 0 is not on the object. The default label is 1. A box is [x1, y1, x2, y2]. Only an export whose decoder has a box input, such as SAM 2.1 and the SAM 3 tracker, takes boxes. Each prompt gives 3 masks of different sizes and the score that the model predicts for each mask. A mask has 255 for the pixels of the object and 0 for the other pixels.

The vision encoder runs once for each image, and the decoder runs once for each prompt. Python transformers pads the points of the prompts of a batch to one length, and the padding changes the masks, so the serving runs each prompt alone, as one call of Python with one prompt. The decoder compiles once for each number of points, with or without a box. The mask-generation pipeline of Python transformers segments the whole image with a grid of points. The serving does not do that yet.

SAM 2.1 and SAM 3 have only a fast image processor, which resizes with torchvision. SAM 3 resizes to 1008x1008. torchvision gives the weights of an 8-bit resize fewer bits than Pillow, so the featurizer does the same for each image_processor_type that ends in Fast. When the image gets smaller, a pixel of Pillow can be one step away.

Image to image

serving = Transmuter.Vision.image_to_image({:hf, "Xenova/swin2SR-classical-sr-x2-64"})
%{image: image} = Nx.Serving.run(serving, image)
StbImage.write_file!(StbImage.from_nx(image), "cats_x2.png")

Transmuter.Vision.image_to_image/2 runs super-resolution models, such as Swin2SR. The result is a :u8 image of shape {height, width, 3}, upscale times larger than the input image. The featurizer pads the image to a multiple of pad_size as Swin2SRImageProcessor of Python transformers, and the serving rounds the output of the model to 8 bits as the image-to-image pipeline. The pipeline of Python transformers also gives the upscaled padding at the bottom and the right of the image. The serving removes it. The output size depends on the image, so each image runs alone, and the model compiles each new image size on its first call.

The tests compare the image with the image-to-image pipeline of Python transformers, without its padding, for Xenova/swin2SR-classical-sr-x2-64 and Xenova/swin2SR-realworld-sr-x4-64-bsrgan-psnr. A value can be 1 step away.

Image-text-to-text

serving = Transmuter.Vision.image_text_to_text({:hf, "HuggingFaceTB/SmolVLM-256M-Instruct"})

chat = %{
  messages: [
    %{role: "user", content: [%{type: "image", image: image}, %{type: "text", text: "Can you describe this image?"}]}
  ]
}

Nx.Serving.run(serving, chat)
#=> %{results: [%{text: " In the image we can see a cat sleeping on a pink bed. ...", token_summary: %{input: 876, output: 24, padding: 0}}]}

The serving runs decoder-only vision language models whose repository has vision_encoder, embed_tokens and decoder_model_merged, as the ImageTextToText exports of transformers.js. It supports Idefics3 models, such as SmolVLM, and Qwen2-VL. The messages use the chat format of Python transformers: the content of a message is a list of parts, %{type: "image", image: image} and %{type: "text", text: text}. A chat can have more than one image.

For SmolVLM, the image processor is the one of Idefics3. The long side of each image becomes 2048 pixels, each side becomes a multiple of 512, and the image splits into tiles of 512x512 pixels. The image resized to one tile comes after the tiles. Each tile gives 64 image tokens in the prompt, so an image of 4x3 tiles gives 832 image tokens. The vision encoder turns the tiles into features, and the features replace the embeddings of the image tokens. Then the decoder generates as Transmuter.Text.generation/2 does, with the same options.

The default files are the q4 vision encoder, the q8 embeddings and the q4 decoder: 179 MB for SmolVLM-256M. The fp32 files give the text of ONNX Runtime on the fp32 files too. They do not give the text of the PyTorch model, because the vision encoder of the export gives other features than transformers 4.57.

For Qwen2-VL, the image processor is the one of Qwen2-VL. Each side of the image becomes the nearest multiple of 28 pixels, within min_pixels and max_pixels of preprocessor_config.json, and the image splits into patches of 14x14 pixels. Each group of 2x2 patches gives one image token, so the 320x240 image of the tests gives 99 image tokens. The vision encoder of the export takes one image in each call. Its graph has shapes that depend on the size of the image, so it compiles once for each new image size. On a Mac GPU, this compile takes 20 to 40 seconds, and later calls with that size take less than 1 second. The decoder takes 3-D positions (M-RoPE): the image tokens have the positions of their rows and columns in the image, as get_rope_index of Python transformers.

serving =
  Transmuter.Vision.image_text_to_text({:hf, "onnx-community/Qwen2-VL-2B-Instruct"},
    variant: %{vision_encoder: :fp32}
  )

The configuration of the repository picks the 8-bit vision encoder, the fp16 embeddings and the q4 decoder. The 8-bit vision encoder gives other features in Transmuter than in ONNX Runtime, so the generated text can be different. With the fp32 vision encoder (2.7 GB), the text is the same as the text of ONNX Runtime. The q4 decoder is 1 GB and the fp16 embeddings are 0.5 GB.

With compile: [batch_size: n], chats go into one batch when the number of their tiles or patches rounds up to the same power of two. Rows of zeros fill each chat to that number. The vision encoder of SmolVLM removes the tiles of zeros, and the processor of Qwen2-VL gives the vision encoder only the patches of each image.

Options of all tasks

• :compile: :batch_size (the default is 1) and, for text, :sequence_length, an integer or a list of lengths (the default is [32, 128, 512], limited to the maximum length of the model). The serving compiles the model for each length and pads each input to the smallest length that fits. Inputs longer than the largest length are truncated. Generation and speech-to-text use only :batch_size, and they do not truncate the prompt.
• :backend and :compute_type: options of Transmuter.load!/2.
• :defn_options: options of Transmuter.compile/3, such as compiler: EMLX.
• :variant, :cache_dir, :revision and :offline: see above and Transmuter.Hub.
• :model, :tokenizer and :featurizer: a model, a tokenizer and a featurizer that you loaded yourself.

Run a serving in your application

Nx.Serving.run/2 compiles the model on each call. In an application, start the serving under your supervision tree. It then compiles once, when it starts:

children = [
  {Nx.Serving,
   serving:
     Transmuter.Text.text_embedding({:hf, "Xenova/all-MiniLM-L6-v2"},
       compile: [batch_size: 8, sequence_length: [64, 256]],
       backend: EMLX.Backend,
       defn_options: [compiler: EMLX]
     ),
   name: MyApp.Embedding,
   batch_timeout: 50}
]

Then call it from any process:

Nx.Serving.batched_run(MyApp.Embedding, "The cat sat on the mat.")

Concurrent calls go into batches of up to 8 inputs, and each batch runs one compiled shape.