Transmuter.Featurizer (transmuter v2.0.0)

Copy Markdown View Source

Turns images into the pixel_values input of a vision model, with the preprocessor_config.json of a model repository.

featurizer = Transmuter.Featurizer.load!({:hf, "Xenova/vit-base-patch16-224"})
image = StbImage.read_file!("cat.jpg")
Transmuter.Featurizer.process(featurizer, [image], backend: EMLX.Backend)
#=> #Nx.Tensor<f32[1][3][224][224]>

An image is a StbImage struct or a tensor of shape {height, width, channels}, usually of type :u8. To read image files, add {:stb_image, "~> 1.0"} to your deps.

The steps and their order are those of the image processors of Python transformers:

  1. do_convert_rgb - the image gets 3 channels. A gray image repeats its channel, and the alpha channel is removed.
  2. do_resize - the image gets the size of the configuration with the resample filter of Pillow. Without resample, the filter is the default of the class of image_processor_type in Python transformers, such as bicubic for CLIPImageProcessor and lanczos for Idefics3ImageProcessor, else bilinear. A fast image processor resizes with bicubic in place of lanczos, because torchvision has no lanczos for tensors. See resize/4. For an image_processor_type that ends in Fast, the weights are those of torchvision, so a pixel can be one step away from Pillow. A size with height and width gives that size. A size with shortest_edge keeps the aspect ratio, and longest_edge limits the long side. With ensure_multiple_of, each side becomes the nearest multiple of it, as in the DPT image processor. With keep_aspect_ratio, both sides take the scale of size that is nearer to 1, so the size depends on the image. The DETR, YOLOS and Grounding DINO image processors compute this size with their own rounding, and YOLOS rounds each side down to a multiple of 16. The SAM image processor gives the long side longest_edge, also when the image is smaller. The featurizer reads image_processor_type and does the same.
  3. do_center_crop - the center region of crop_size stays. A configuration with crop_pct, such as ConvNeXt, resizes the short side to shortest_edge / crop_pct and crops a square of shortest_edge. When shortest_edge is 384 or more, it resizes to a square of shortest_edge and does not crop.
  4. do_rescale - the values are multiplied by rescale_factor.
  5. do_normalize - the values are normalized with image_mean and image_std.
  6. do_pad of the SAM image processor - zeros on the bottom and the right give the image the pad_size, 1024x1024 by default.

The OWLv2 image processor (Owlv2ImageProcessor) does the steps in another order, and the featurizer reads image_processor_type and does the same. It rescales the image, pads it to a square on the bottom and the right with the value 0.5 (do_pad), resizes it with the :gaussian_linear method of resize/4, and normalizes it.

The Swin2SR image processor (Swin2SRImageProcessor) only rescales and pads. With do_pad, the bottom and the right get the "symmetric" padding of numpy (the edge pixel repeats: a b c | c b a) to the next multiple of size_divisor (pad_size in older configurations). As in Python transformers, a side that is already a multiple gets size_divisor more pixels.

The result has the shape {batch, channels, height, width} and the type :f32.

Summary

Functions

image_size(featurizer, size)

Returns the {height, width} of the image in the result of process/2 for an image of size {height, width}. A pad_size (as in SAM) adds padding on the bottom and the right after this size, and size/1 gives the size with the padding.

load!(repo_or_source, opts \\ [])

Loads the featurizer of a Transmuter.Repo or of a source ({:hf, repo} or a local directory) from its preprocessor_config.json.

new(config)

Builds a featurizer from the map of a preprocessor_config.json.

process(featurizer, images, opts \\ [])

Turns a list of images into one tensor of shape {batch, channels, height, width} and type :f32. The images must give the same size.

resize(image, size, method, weights \\ :pillow)

Resizes an image of shape {height, width, channels} to {height, width} as Image.resize/2 of Pillow.

rgb(image)

Converts an image of shape {height, width, channels} to 3 channels, as Image.convert("RGB") of Pillow. A gray image repeats its channel, and the alpha channel is removed.

size(featurizer)

Returns the {height, width} of the images that process/2 gives, or nil when the size depends on the image, for example with shortest_edge and no crop.

tensor(image)

Returns {:ok, tensor} for an image, a StbImage struct or a tensor of shape {height, width, channels}. Returns {:error, message} for other values.

Functions

image_size(featurizer, size)

Returns the {height, width} of the image in the result of process/2 for an image of size {height, width}. A pad_size (as in SAM) adds padding on the bottom and the right after this size, and size/1 gives the size with the padding.

load!(repo_or_source, opts \\ [])

Loads the featurizer of a Transmuter.Repo or of a source ({:hf, repo} or a local directory) from its preprocessor_config.json.

Options go to Transmuter.Repo.load!/2 when the argument is a source.

new(config)

Builds a featurizer from the map of a preprocessor_config.json.

An older configuration names a feature_extractor_type, such as ViTFeatureExtractor, in place of an image_processor_type. As AutoImageProcessor of Python transformers, the featurizer reads it as the image processor of the same name, such as ViTImageProcessor.

process(featurizer, images, opts \\ [])

Turns a list of images into one tensor of shape {batch, channels, height, width} and type :f32. The images must give the same size.

Options

  • :backend - the backend that does the steps, such as EMLX.Backend. The default is the backend of each image. The resize is slow on Nx.BinaryBackend.

resize(image, size, method, weights \\ :pillow)

Resizes an image of shape {height, width, channels} to {height, width} as Image.resize/2 of Pillow.

method is :nearest, :box, :bilinear, :hamming, :bicubic or :lanczos. The filters other than :nearest widen when the image gets smaller, so they also remove aliasing. The width changes first and the height second.

A :u8 image gives a :u8 image. As in Pillow, the weights are fixed-point numbers with 22 fraction bits, and each pass rounds to :u8. An image of another type is resized in :f32.

weights is :pillow, the default, or :torchvision, which the featurizer uses for the fast image processors of Python transformers (an image_processor_type that ends in Fast). On Apple silicon and on x86 with AVX2, torchvision resizes a :u8 image with the filters and the taps of Pillow, but its weights are 16-bit integers. So for :bilinear, :bicubic and :lanczos, the weights of an axis get the most fraction bits, up to 22, that keep the largest weight as an integer below 2^15. With fewer bits, a pixel can be one step away from Pillow when the image gets smaller. The other methods and the images of other types resize as with :pillow.

:gaussian_linear is the resize of the OWLv2 image processor of Python transformers, which is skimage.transform.resize with anti-aliasing. On each axis, a gaussian blur with sigma max(0, (scale - 1) / 2) comes first, where scale is the input size divided by the output size. Then a linear interpolation between the pixel centers gives the output. Both steps reflect the image at its edges, as the "mirror" mode of scipy.ndimage. The result is clipped to the minimum and the maximum of the image.

rgb(image)

Converts an image of shape {height, width, channels} to 3 channels, as Image.convert("RGB") of Pillow. A gray image repeats its channel, and the alpha channel is removed.

size(featurizer)

Returns the {height, width} of the images that process/2 gives, or nil when the size depends on the image, for example with shortest_edge and no crop.

tensor(image)

Returns {:ok, tensor} for an image, a StbImage struct or a tensor of shape {height, width, channels}. Returns {:error, message} for other values.