Prepares the images and the prompt of Gemma 3, as Gemma3ImageProcessor and Gemma3Processor of Python transformers do. Transmuter.Vision.image_text_to_text/2 uses it.
Each image gets 3 channels, the size of preprocessor_config.json (896x896 for Gemma 3) without its aspect ratio, and the rescale and the normalization of the configuration, as in Transmuter.Featurizer. Python transformers does pan and scan only when the caller asks for it, so the processor does not do it.
The image tokens come from the tokenizer configuration, as in Gemma3Processor: boi_token (<start_of_image>), image_token (<image_soft_token>) and eoi_token (<end_of_image>). In the prompt, each image is a placeholder boi_token, and process/3 gives the text that replaces it: \n\n, boi_token, image_seq_length image tokens, eoi_token and \n\n. image_seq_length comes from processor_config.json, else from mm_tokens_per_image of config.json, else it is 256. The vision encoder gives one feature for each image token.
Summary
Functions
Runs the vision encoder encode, a function from Transmuter.compile/3, on the images of a batch, and gives the inputs of Transmuter.Generation.generate/5.
Loads the processor of a Transmuter.Repo from its preprocessor_config.json, processor_config.json, config.json and tokenizer configuration.
The text of an image in the prompt before process/3 expands it.
Turns an image into pixel values and gives the text of the image in the prompt.
The inputs of the vision encoder for one image, on Nx.BinaryBackend. The encoder compiles for them. The encoder takes one image in each call, so the template does not depend on batch_size.
Functions
Runs the vision encoder encode, a function from Transmuter.compile/3, on the images of a batch, and gives the inputs of Transmuter.Generation.generate/5.
tensors has the "pixel_values" of each row, of shape {batch, rows, 3, height, width}, and the "image_sizes" of each row, of shape {batch, rows, 2}. Rows of zeros follow the images of a row. Each image runs alone, so the encoder has one compiled shape and the memory of one image, and the rows of zeros do not run. Returns %{image_features: features}: the features of the images of all rows, in order, of shape {images * image_seq_length, hidden}. The prompts take the default positions.
Loads the processor of a Transmuter.Repo from its preprocessor_config.json, processor_config.json, config.json and tokenizer configuration.
The text of an image in the prompt before process/3 expands it.
Turns an image into pixel values and gives the text of the image in the prompt.
Returns {tensors, text}. tensors has "pixel_values" of shape {1, 3, height, width} and type :f32 on backend, and "image_sizes" of shape {1, 2} and type :s64, the height and the width of the image. Each pixel value row of a batch has 3 height width values (9.6 MB for 896x896), also a row of zeros that fills a chat to its batch key.
The inputs of the vision encoder for one image, on Nx.BinaryBackend. The encoder compiles for them. The encoder takes one image in each call, so the template does not depend on batch_size.