Prepares the images and the prompt of Pixtral and Mistral 3 models, such as Ministral 3, as PixtralImageProcessor and PixtralProcessor of Python transformers do. Transmuter.Vision.image_text_to_text/2 uses it.
The steps for each image are:
- The image gets 3 channels, as in
Transmuter.Featurizer.rgb/1. ForPixtralImageProcessor, the slow image processor, an image with an alpha channel goes on a white background first, asImage.pasteof Pillow does.PixtralImageProcessorFastremoves the alpha channel. - When a side is longer than
size["longest_edge"]ofpreprocessor_config.json, both sides get smaller by the same ratio and are rounded down. Then each side gets the next multiple ofpatch_size * spatial_merge_sizeofprocessor_config.json(28 for Ministral 3). The resize uses theresamplefilter of the configuration on the 8-bit image, with the weights of torchvision for a fast image processor, as inTransmuter.Featurizer. Withoutdo_resize, the image keeps its size, and the blocks of step 4 start at the top left corner. The pixels to the right of the last full block and below it are not used, as in Python transformers, where the processor givesheight // blockrows of tokens. A side with no full block raises an error. - The image is rescaled and normalized, as in
Transmuter.Featurizer. - The image splits into blocks of
patch_size * spatial_merge_sizepixels on each side, from left to right and from top to bottom. Each block is one row of3 * 28 * 28values. The vision encoder merges the patches of a block into one feature, so the image has one row for each feature.
In the prompt, each image is a placeholder, the image_token of processor_config.json ([IMG] by default). It becomes one row of these tokens for each row of blocks, with one token for each block. [IMG_BREAK] comes after each row, and [IMG_END] replaces the last [IMG_BREAK]. The features replace the [IMG] tokens, and the prompt takes the default positions.
Summary
Functions
Runs the vision encoder encode, a function from Transmuter.compile/3, on the images of a batch, and gives the inputs of Transmuter.Generation.generate/5.
Loads the processor of a Transmuter.Repo from its preprocessor_config.json and processor_config.json.
Builds the processor from the maps of a preprocessor_config.json and a processor_config.json.
The text of an image in the prompt before process/3 expands it.
Turns an image into blocks and gives the text of the image in the prompt.
The inputs of the vision encoder for an image of one block, on Nx.BinaryBackend. The encoder compiles for them. The encoder takes one image in each call, so the template does not depend on batch_size.
Functions
Runs the vision encoder encode, a function from Transmuter.compile/3, on the images of a batch, and gives the inputs of Transmuter.Generation.generate/5.
tensors has the "pixel_values" of each row, of shape {batch, blocks, 3 * block * block}, and the "image_sizes" of each row, of shape {batch, rows, 2}. Rows of zeros follow the blocks and the sizes of the images.
The export of the vision encoder computes the rotary positions of the patches from the size of the image, so each image runs alone. Returns %{image_features: features}: the features of the images of all rows, in order. The prompts take the default positions.
Loads the processor of a Transmuter.Repo from its preprocessor_config.json and processor_config.json.
Builds the processor from the maps of a preprocessor_config.json and a processor_config.json.
The text of an image in the prompt before process/3 expands it.
Turns an image into blocks and gives the text of the image in the prompt.
Returns {tensors, text}. tensors has "pixel_values" of shape {blocks, 3 * block * block} and type :f32 on backend, and "image_sizes" of shape {1, 2} and type :s64, the height and the width of the image after the resize.
The inputs of the vision encoder for an image of one block, on Nx.BinaryBackend. The encoder compiles for them. The encoder takes one image in each call, so the template does not depend on batch_size.