Prepares the images and the prompt of LLaVA models, such as LLaVA-Interleave, as LlavaProcessor of Python transformers does. Transmuter.Vision.image_text_to_text/2 uses it.
The image processor of preprocessor_config.json turns each image into pixel values of one size, as in Transmuter.Featurizer: SiglipImageProcessor resizes to size, and CLIPImageProcessor resizes the short side and crops the center.
In the prompt, each image is a placeholder, the image_token of the tokenizer configuration, else of processor_config.json (<image> by default). It becomes one image token for each feature: (height // patch_size) * (width // patch_size) + num_additional_image_tokens tokens of processor_config.json, and one token less when its vision_feature_select_strategy is "default". The height and the width are those of the pixel values. The vision encoder gives one feature for each of these tokens.
Summary
Functions
Runs the vision encoder encode, a function from Transmuter.compile/3, on each image of a batch alone, and gives the inputs of Transmuter.Generation.generate/5.
Loads the processor of a Transmuter.Repo from its preprocessor_config.json, processor_config.json and tokenizer configuration.
The text of an image in the prompt before process/3 expands it.
Turns an image into pixel values and gives the text of the image in the prompt.
The inputs of the vision encoder for one image, on Nx.BinaryBackend. The encoder compiles for them. The encoder takes one image in each call, so the template does not depend on batch_size.
Functions
Runs the vision encoder encode, a function from Transmuter.compile/3, on each image of a batch alone, and gives the inputs of Transmuter.Generation.generate/5.
tensors has the "pixel_values" of each row, of shape {batch, rows, 3, height, width}, and the "image_sizes" of each row, of shape {batch, rows, 2}. Rows of zeros follow the images of a row, and they do not run. Returns %{image_features: features}: the features of the images of all rows, in order. The prompts take the default positions.
Loads the processor of a Transmuter.Repo from its preprocessor_config.json, processor_config.json and tokenizer configuration.
The image token is that of the tokenizer, as LlavaProcessor of Python transformers reads it, else the image_token of processor_config.json, else the :image_token option, the text of the token of image_token_index in config.json.
The text of an image in the prompt before process/3 expands it.
Turns an image into pixel values and gives the text of the image in the prompt.
Returns {tensors, text}. tensors has "pixel_values" of shape {1, 3, height, width} and type :f32 on backend, and "image_sizes" of shape {1, 2} and type :s64, the height and the width of the image.
The inputs of the vision encoder for one image, on Nx.BinaryBackend. The encoder compiles for them. The encoder takes one image in each call, so the template does not depend on batch_size.