Prepares the images and the prompt of Qwen2-VL, Qwen2.5-VL and Qwen3-VL, as Qwen2VLImageProcessor, Qwen2VLImageProcessorFast, Qwen2VLProcessor, Qwen2_5_VLProcessor and Qwen3VLProcessor of Python transformers do for images. Transmuter.Vision.image_text_to_text/2 uses it.
The steps for each image are:
- The image gets 3 channels, as in
Transmuter.Featurizer.rgb/1. - Each side gets the nearest multiple of
patch_size * merge_sizepixels: 28 for Qwen2-VL and 32 for Qwen3-VL. When the image then has more thanmax_pixelsor fewer thanmin_pixelspixels, the sides get smaller or larger with the same aspect ratio, assmart_resizeof Python transformers. The resize uses theresamplefilter of the configuration, bicubic by default, on the 8-bit image. The weights are those of Pillow, or of torchvision forQwen2VLImageProcessorFast, seeTransmuter.Featurizer.resize/4. - The image is rescaled and normalized, as in
Transmuter.Featurizer. - The image is repeated
temporal_patch_size(2) times, as a video of 2 frames, and splits into patches ofpatch_sizepixels (14x14 for Qwen2-VL). Each patch is one row of3 * 2 * patch_size * patch_sizevalues. Each group of 2x2 patches, which the vision encoder merges into one feature, comes in consecutive rows.
The grid of an image is [1, height / patch_size, width / patch_size], as image_grid_thw of Python transformers. In the prompt, each image is a placeholder <|image_pad|>, which becomes one <|image_pad|> for each feature: the number of patches divided by 4.
The image tokens of the prompt take 3-D positions for M-RoPE, as get_rope_index of Python transformers. The text tokens have the same position on the 3 axes. The image tokens of an image have the positions of their frame, row and column after the text before them, and the text after the image continues from the largest of these positions.
Summary
Functions
Runs the vision encoder encode, a function from Transmuter.compile/3, on the images of a batch, and gives the inputs of Transmuter.Generation.generate/5.
Loads the processor of a Transmuter.Repo from its preprocessor_config.json and config.json.
The text of an image in the prompt before process/3 expands it.
The 3-D positions of the tokens ids of a prompt with the images of grids grids, as get_rope_index of Python transformers. Returns a list of the positions on each of the 3 axes.
Turns an image into patches and gives the text of the image in the prompt.
The inputs of the vision encoder for an image of 2x2 patches, on Nx.BinaryBackend. The encoder compiles for them. The encoder takes one image in each call, so the template does not depend on batch_size.
Functions
Runs the vision encoder encode, a function from Transmuter.compile/3, on the images of a batch, and gives the inputs of Transmuter.Generation.generate/5.
tensors has the "pixel_values" of each row, of shape {batch, rows, values}, and the "image_grid_thw" of each row, of shape {batch, rows, 3}. Rows of zeros follow the patches and the grids of the images. prompts are the token ids of each row.
The export of the vision encoder of Qwen2-VL computes the rotary positions of the patches for one image, so each image runs alone. Returns %{image_features: features, position_ids: positions}: the features of the images of all rows, in order, and the 3-D positions of each prompt.
Loads the processor of a Transmuter.Repo from its preprocessor_config.json and config.json.
Options
:inputs- the input names of the vision encoder, such asmodel.inputsof thevision_encodercomponent. The input whose name ends ingrid_thwtakes the grid:"grid_thw"in the export of Qwen2-VL, and"image_grid_thw"in the export of Qwen3-VL. The default is["pixel_values", "grid_thw"].
The text of an image in the prompt before process/3 expands it.
The 3-D positions of the tokens ids of a prompt with the images of grids grids, as get_rope_index of Python transformers. Returns a list of the positions on each of the 3 axes.
Turns an image into patches and gives the text of the image in the prompt.
Returns {tensors, text}. tensors has "pixel_values" of shape {patches, 3 * 2 * patch_size * patch_size} (1176 for Qwen2-VL) and type :f32 on backend, and "image_grid_thw" of shape {1, 3} and type :s64.
The inputs of the vision encoder for an image of 2x2 patches, on Nx.BinaryBackend. The encoder compiles for them. The encoder takes one image in each call, so the template does not depend on batch_size.