# `Transmuter.Vision.Qwen2VLProcessor`
[🔗](https://github.com/georgeguimaraes/transmuter/blob/v2.0.0/lib/transmuter/vision/qwen2_vl_processor.ex#L1)

Prepares the images and the prompt of Qwen2-VL, Qwen2.5-VL and Qwen3-VL, as `Qwen2VLImageProcessor`, `Qwen2VLImageProcessorFast`, `Qwen2VLProcessor`, `Qwen2_5_VLProcessor` and `Qwen3VLProcessor` of Python transformers do for images. `Transmuter.Vision.image_text_to_text/2` uses it.

The steps for each image are:

  1. The image gets 3 channels, as in `Transmuter.Featurizer.rgb/1`.
  2. Each side gets the nearest multiple of `patch_size * merge_size` pixels: 28 for Qwen2-VL and 32 for Qwen3-VL. When the image then has more than `max_pixels` or fewer than `min_pixels` pixels, the sides get smaller or larger with the same aspect ratio, as `smart_resize` of Python transformers. The resize uses the `resample` filter of the configuration, bicubic by default, on the 8-bit image. The weights are those of Pillow, or of torchvision for `Qwen2VLImageProcessorFast`, see `Transmuter.Featurizer.resize/4`.
  3. The image is rescaled and normalized, as in `Transmuter.Featurizer`.
  4. The image is repeated `temporal_patch_size` (2) times, as a video of 2 frames, and splits into patches of `patch_size` pixels (14x14 for Qwen2-VL). Each patch is one row of `3 * 2 * patch_size * patch_size` values. Each group of 2x2 patches, which the vision encoder merges into one feature, comes in consecutive rows.

The grid of an image is `[1, height / patch_size, width / patch_size]`, as `image_grid_thw` of Python transformers. In the prompt, each image is a placeholder `<|image_pad|>`, which becomes one `<|image_pad|>` for each feature: the number of patches divided by 4.

The image tokens of the prompt take 3-D positions for M-RoPE, as `get_rope_index` of Python transformers. The text tokens have the same position on the 3 axes. The image tokens of an image have the positions of their frame, row and column after the text before them, and the text after the image continues from the largest of these positions.

# `encode`

Runs the vision encoder `encode`, a function from `Transmuter.compile/3`, on the images of a batch, and gives the inputs of `Transmuter.Generation.generate/5`.

`tensors` has the `"pixel_values"` of each row, of shape `{batch, rows, values}`, and the `"image_grid_thw"` of each row, of shape `{batch, rows, 3}`. Rows of zeros follow the patches and the grids of the images. `prompts` are the token ids of each row.

The export of the vision encoder of Qwen2-VL computes the rotary positions of the patches for one image, so each image runs alone. Returns `%{image_features: features, position_ids: positions}`: the features of the images of all rows, in order, and the 3-D positions of each prompt.

# `load!`

Loads the processor of a `Transmuter.Repo` from its `preprocessor_config.json` and `config.json`.

## Options

  * `:inputs` - the input names of the vision encoder, such as `model.inputs` of the `vision_encoder` component. The input whose name ends in `grid_thw` takes the grid: `"grid_thw"` in the export of Qwen2-VL, and `"image_grid_thw"` in the export of Qwen3-VL. The default is `["pixel_values", "grid_thw"]`.

# `placeholder`

The text of an image in the prompt before `process/3` expands it.

# `positions`

The 3-D positions of the tokens `ids` of a prompt with the images of grids `grids`, as `get_rope_index` of Python transformers. Returns a list of the positions on each of the 3 axes.

# `process`

Turns an image into patches and gives the text of the image in the prompt.

Returns `{tensors, text}`. `tensors` has `"pixel_values"` of shape `{patches, 3 * 2 * patch_size * patch_size}` (1176 for Qwen2-VL) and type `:f32` on `backend`, and `"image_grid_thw"` of shape `{1, 3}` and type `:s64`.

# `template`

The inputs of the vision encoder for an image of 2x2 patches, on `Nx.BinaryBackend`. The encoder compiles for them. The encoder takes one image in each call, so the template does not depend on `batch_size`.

---

*Consult [api-reference.md](api-reference.md) for complete listing*