Prepares the images and the prompt of Idefics3 models, such as SmolVLM, as Idefics3ImageProcessor and Idefics3Processor of Python transformers do. Transmuter.Vision.image_text_to_text/2 uses it.
The steps for each image are:
- The image gets 3 channels, as in
Transmuter.Featurizer.rgb/1. - The long side of the image gets
size["longest_edge"]ofpreprocessor_config.json, and the short side keeps the aspect ratio. A small image also gets larger. - Each side gets the next multiple of
max_image_size["longest_edge"], the size of a tile. - The image splits into tiles, from left to right and from top to bottom. The image resized to one tile comes after them. Without
do_image_splitting, the image resized to one tile is the only tile. - Each tile is rescaled and normalized, as in
Transmuter.Featurizer.
The resizes use the resample filter of the configuration on the 8-bit image, as Pillow does, or with the weights of torchvision for Idefics3ImageProcessorFast, which uses bicubic in place of lanczos. See Transmuter.Featurizer.resize/4.
In the prompt, each image is a placeholder <image>, and process/3 gives the text that replaces it: <fake_token_around_image><row_r_col_c> and image_seq_len image tokens for each tile, a newline after each row, and then \n<fake_token_around_image><global-img>, the image tokens of the last tile and <fake_token_around_image>.
Summary
Functions
Runs the vision encoder encode, a function from Transmuter.compile/3, on the "pixel_values" of tensors, of shape {batch, tiles, 3, tile, tile}, and gives the inputs of Transmuter.Generation.generate/5.
Loads the processor of a Transmuter.Repo from its preprocessor_config.json and processor_config.json.
The text of an image in the prompt before process/3 expands it.
Turns an image into tiles and gives the text of the image in the prompt.
The inputs of the vision encoder for one tile in each row of a batch of batch_size rows, on Nx.BinaryBackend. The encoder compiles for them.
Functions
Runs the vision encoder encode, a function from Transmuter.compile/3, on the "pixel_values" of tensors, of shape {batch, tiles, 3, tile, tile}, and gives the inputs of Transmuter.Generation.generate/5.
The encoder removes the tiles of zeros, as Idefics3Model.get_image_features of Python transformers does. So the features are those of the real tiles of all rows, in order. Returns %{image_features: features}, a tensor of shape {tiles * image_seq_len, hidden}, with rows of zeros after the features for the removed tiles. The prompts take the default positions.
Loads the processor of a Transmuter.Repo from its preprocessor_config.json and processor_config.json.
The text of an image in the prompt before process/3 expands it.
Turns an image into tiles and gives the text of the image in the prompt.
Returns {tensors, text}. tensors has "pixel_values" of shape {tiles, 3, tile, tile} and type :f32, on backend.
The inputs of the vision encoder for one tile in each row of a batch of batch_size rows, on Nx.BinaryBackend. The encoder compiles for them.