# `Transmuter.Vision.PixtralProcessor`
[🔗](https://github.com/georgeguimaraes/transmuter/blob/v2.0.0/lib/transmuter/vision/pixtral_processor.ex#L1)

Prepares the images and the prompt of Pixtral and Mistral 3 models, such as Ministral 3, as `PixtralImageProcessor` and `PixtralProcessor` of Python transformers do. `Transmuter.Vision.image_text_to_text/2` uses it.

The steps for each image are:

  1. The image gets 3 channels, as in `Transmuter.Featurizer.rgb/1`. For `PixtralImageProcessor`, the slow image processor, an image with an alpha channel goes on a white background first, as `Image.paste` of Pillow does. `PixtralImageProcessorFast` removes the alpha channel.
  2. When a side is longer than `size["longest_edge"]` of `preprocessor_config.json`, both sides get smaller by the same ratio and are rounded down. Then each side gets the next multiple of `patch_size * spatial_merge_size` of `processor_config.json` (28 for Ministral 3). The resize uses the `resample` filter of the configuration on the 8-bit image, with the weights of torchvision for a fast image processor, as in `Transmuter.Featurizer`. Without `do_resize`, the image keeps its size, and the blocks of step 4 start at the top left corner. The pixels to the right of the last full block and below it are not used, as in Python transformers, where the processor gives `height // block` rows of tokens. A side with no full block raises an error.
  3. The image is rescaled and normalized, as in `Transmuter.Featurizer`.
  4. The image splits into blocks of `patch_size * spatial_merge_size` pixels on each side, from left to right and from top to bottom. Each block is one row of `3 * 28 * 28` values. The vision encoder merges the patches of a block into one feature, so the image has one row for each feature.

In the prompt, each image is a placeholder, the `image_token` of `processor_config.json` (`[IMG]` by default). It becomes one row of these tokens for each row of blocks, with one token for each block. `[IMG_BREAK]` comes after each row, and `[IMG_END]` replaces the last `[IMG_BREAK]`. The features replace the `[IMG]` tokens, and the prompt takes the default positions.

# `encode`

Runs the vision encoder `encode`, a function from `Transmuter.compile/3`, on the images of a batch, and gives the inputs of `Transmuter.Generation.generate/5`.

`tensors` has the `"pixel_values"` of each row, of shape `{batch, blocks, 3 * block * block}`, and the `"image_sizes"` of each row, of shape `{batch, rows, 2}`. Rows of zeros follow the blocks and the sizes of the images.

The export of the vision encoder computes the rotary positions of the patches from the size of the image, so each image runs alone. Returns `%{image_features: features}`: the features of the images of all rows, in order. The prompts take the default positions.

# `load!`

Loads the processor of a `Transmuter.Repo` from its `preprocessor_config.json` and `processor_config.json`.

# `new`

Builds the processor from the maps of a `preprocessor_config.json` and a `processor_config.json`.

# `placeholder`

The text of an image in the prompt before `process/3` expands it.

# `process`

Turns an image into blocks and gives the text of the image in the prompt.

Returns `{tensors, text}`. `tensors` has `"pixel_values"` of shape `{blocks, 3 * block * block}` and type `:f32` on `backend`, and `"image_sizes"` of shape `{1, 2}` and type `:s64`, the height and the width of the image after the resize.

# `template`

The inputs of the vision encoder for an image of one block, on `Nx.BinaryBackend`. The encoder compiles for them. The encoder takes one image in each call, so the template does not depend on `batch_size`.

---

*Consult [api-reference.md](api-reference.md) for complete listing*