# `Transmuter.Vision.LlavaProcessor`
[🔗](https://github.com/georgeguimaraes/transmuter/blob/v2.0.0/lib/transmuter/vision/llava_processor.ex#L1)

Prepares the images and the prompt of LLaVA models, such as LLaVA-Interleave, as `LlavaProcessor` of Python transformers does. `Transmuter.Vision.image_text_to_text/2` uses it.

The image processor of `preprocessor_config.json` turns each image into pixel values of one size, as in `Transmuter.Featurizer`: `SiglipImageProcessor` resizes to `size`, and `CLIPImageProcessor` resizes the short side and crops the center.

In the prompt, each image is a placeholder, the `image_token` of the tokenizer configuration, else of `processor_config.json` (`<image>` by default). It becomes one image token for each feature: `(height // patch_size) * (width // patch_size) + num_additional_image_tokens` tokens of `processor_config.json`, and one token less when its `vision_feature_select_strategy` is `"default"`. The height and the width are those of the pixel values. The vision encoder gives one feature for each of these tokens.

# `encode`

Runs the vision encoder `encode`, a function from `Transmuter.compile/3`, on each image of a batch alone, and gives the inputs of `Transmuter.Generation.generate/5`.

`tensors` has the `"pixel_values"` of each row, of shape `{batch, rows, 3, height, width}`, and the `"image_sizes"` of each row, of shape `{batch, rows, 2}`. Rows of zeros follow the images of a row, and they do not run. Returns `%{image_features: features}`: the features of the images of all rows, in order. The prompts take the default positions.

# `load!`

Loads the processor of a `Transmuter.Repo` from its `preprocessor_config.json`, `processor_config.json` and tokenizer configuration.

The image token is that of the tokenizer, as `LlavaProcessor` of Python transformers reads it, else the `image_token` of `processor_config.json`, else the `:image_token` option, the text of the token of `image_token_index` in `config.json`.

# `placeholder`

The text of an image in the prompt before `process/3` expands it.

# `process`

Turns an image into pixel values and gives the text of the image in the prompt.

Returns `{tensors, text}`. `tensors` has `"pixel_values"` of shape `{1, 3, height, width}` and type `:f32` on `backend`, and `"image_sizes"` of shape `{1, 2}` and type `:s64`, the height and the width of the image.

# `template`

The inputs of the vision encoder for one image, on `Nx.BinaryBackend`. The encoder compiles for them. The encoder takes one image in each call, so the template does not depend on `batch_size`.

---

*Consult [api-reference.md](api-reference.md) for complete listing*