# `Transmuter.Vision.Idefics3Processor`
[🔗](https://github.com/georgeguimaraes/transmuter/blob/v2.0.0/lib/transmuter/vision/idefics3_processor.ex#L1)

Prepares the images and the prompt of Idefics3 models, such as SmolVLM, as `Idefics3ImageProcessor` and `Idefics3Processor` of Python transformers do. `Transmuter.Vision.image_text_to_text/2` uses it.

The steps for each image are:

  1. The image gets 3 channels, as in `Transmuter.Featurizer.rgb/1`.
  2. The long side of the image gets `size["longest_edge"]` of `preprocessor_config.json`, and the short side keeps the aspect ratio. A small image also gets larger.
  3. Each side gets the next multiple of `max_image_size["longest_edge"]`, the size of a tile.
  4. The image splits into tiles, from left to right and from top to bottom. The image resized to one tile comes after them. Without `do_image_splitting`, the image resized to one tile is the only tile.
  5. Each tile is rescaled and normalized, as in `Transmuter.Featurizer`.

The resizes use the `resample` filter of the configuration on the 8-bit image, as Pillow does, or with the weights of torchvision for `Idefics3ImageProcessorFast`, which uses bicubic in place of lanczos. See `Transmuter.Featurizer.resize/4`.

In the prompt, each image is a placeholder `<image>`, and `process/3` gives the text that replaces it: `<fake_token_around_image><row_r_col_c>` and `image_seq_len` image tokens for each tile, a newline after each row, and then `\n<fake_token_around_image><global-img>`, the image tokens of the last tile and `<fake_token_around_image>`.

# `encode`

Runs the vision encoder `encode`, a function from `Transmuter.compile/3`, on the `"pixel_values"` of `tensors`, of shape `{batch, tiles, 3, tile, tile}`, and gives the inputs of `Transmuter.Generation.generate/5`.

The encoder removes the tiles of zeros, as `Idefics3Model.get_image_features` of Python transformers does. So the features are those of the real tiles of all rows, in order. Returns `%{image_features: features}`, a tensor of shape `{tiles * image_seq_len, hidden}`, with rows of zeros after the features for the removed tiles. The prompts take the default positions.

# `load!`

Loads the processor of a `Transmuter.Repo` from its `preprocessor_config.json` and `processor_config.json`.

# `placeholder`

The text of an image in the prompt before `process/3` expands it.

# `process`

Turns an image into tiles and gives the text of the image in the prompt.

Returns `{tensors, text}`. `tensors` has `"pixel_values"` of shape `{tiles, 3, tile, tile}` and type `:f32`, on `backend`.

# `template`

The inputs of the vision encoder for one tile in each row of a batch of `batch_size` rows, on `Nx.BinaryBackend`. The encoder compiles for them.

---

*Consult [api-reference.md](api-reference.md) for complete listing*