# `Transmuter.Featurizer`
[🔗](https://github.com/georgeguimaraes/transmuter/blob/v2.0.0/lib/transmuter/featurizer.ex#L1)

Turns images into the `pixel_values` input of a vision model, with the `preprocessor_config.json` of a model repository.

    featurizer = Transmuter.Featurizer.load!({:hf, "Xenova/vit-base-patch16-224"})
    image = StbImage.read_file!("cat.jpg")
    Transmuter.Featurizer.process(featurizer, [image], backend: EMLX.Backend)
    #=> #Nx.Tensor<f32[1][3][224][224]>

An image is a `StbImage` struct or a tensor of shape `{height, width, channels}`, usually of type `:u8`. To read image files, add `{:stb_image, "~> 1.0"}` to your deps.

The steps and their order are those of the image processors of Python transformers:

  1. `do_convert_rgb` - the image gets 3 channels. A gray image repeats its channel, and the alpha channel is removed.
  2. `do_resize` - the image gets the `size` of the configuration with the `resample` filter of Pillow. Without `resample`, the filter is the default of the class of `image_processor_type` in Python transformers, such as bicubic for `CLIPImageProcessor` and lanczos for `Idefics3ImageProcessor`, else bilinear. A fast image processor resizes with bicubic in place of lanczos, because torchvision has no lanczos for tensors. See `resize/4`. For an `image_processor_type` that ends in `Fast`, the weights are those of torchvision, so a pixel can be one step away from Pillow. A `size` with `height` and `width` gives that size. A `size` with `shortest_edge` keeps the aspect ratio, and `longest_edge` limits the long side. With `ensure_multiple_of`, each side becomes the nearest multiple of it, as in the DPT image processor. With `keep_aspect_ratio`, both sides take the scale of `size` that is nearer to 1, so the size depends on the image. The DETR, YOLOS and Grounding DINO image processors compute this size with their own rounding, and YOLOS rounds each side down to a multiple of 16. The SAM image processor gives the long side `longest_edge`, also when the image is smaller. The featurizer reads `image_processor_type` and does the same.
  3. `do_center_crop` - the center region of `crop_size` stays. A configuration with `crop_pct`, such as ConvNeXt, resizes the short side to `shortest_edge / crop_pct` and crops a square of `shortest_edge`. When `shortest_edge` is 384 or more, it resizes to a square of `shortest_edge` and does not crop.
  4. `do_rescale` - the values are multiplied by `rescale_factor`.
  5. `do_normalize` - the values are normalized with `image_mean` and `image_std`.
  6. `do_pad` of the SAM image processor - zeros on the bottom and the right give the image the `pad_size`, 1024x1024 by default.

The OWLv2 image processor (`Owlv2ImageProcessor`) does the steps in another order, and the featurizer reads `image_processor_type` and does the same. It rescales the image, pads it to a square on the bottom and the right with the value 0.5 (`do_pad`), resizes it with the `:gaussian_linear` method of `resize/4`, and normalizes it.

The Swin2SR image processor (`Swin2SRImageProcessor`) only rescales and pads. With `do_pad`, the bottom and the right get the `"symmetric"` padding of numpy (the edge pixel repeats: `a b c | c b a`) to the next multiple of `size_divisor` (`pad_size` in older configurations). As in Python transformers, a side that is already a multiple gets `size_divisor` more pixels.

The result has the shape `{batch, channels, height, width}` and the type `:f32`.

# `image_size`

Returns the `{height, width}` of the image in the result of `process/2` for an image of size `{height, width}`. A `pad_size` (as in SAM) adds padding on the bottom and the right after this size, and `size/1` gives the size with the padding.

# `load!`

Loads the featurizer of a `Transmuter.Repo` or of a source (`{:hf, repo}` or a local directory) from its `preprocessor_config.json`.

Options go to `Transmuter.Repo.load!/2` when the argument is a source.

# `new`

Builds a featurizer from the map of a `preprocessor_config.json`.

An older configuration names a `feature_extractor_type`, such as `ViTFeatureExtractor`, in place of an `image_processor_type`. As `AutoImageProcessor` of Python transformers, the featurizer reads it as the image processor of the same name, such as `ViTImageProcessor`.

# `process`

Turns a list of images into one tensor of shape `{batch, channels, height, width}` and type `:f32`. The images must give the same size.

## Options

  * `:backend` - the backend that does the steps, such as `EMLX.Backend`. The default is the backend of each image. The resize is slow on `Nx.BinaryBackend`.

# `resize`

Resizes an image of shape `{height, width, channels}` to `{height, width}` as `Image.resize/2` of Pillow.

`method` is `:nearest`, `:box`, `:bilinear`, `:hamming`, `:bicubic` or `:lanczos`. The filters other than `:nearest` widen when the image gets smaller, so they also remove aliasing. The width changes first and the height second.

A `:u8` image gives a `:u8` image. As in Pillow, the weights are fixed-point numbers with 22 fraction bits, and each pass rounds to `:u8`. An image of another type is resized in `:f32`.

`weights` is `:pillow`, the default, or `:torchvision`, which the featurizer uses for the fast image processors of Python transformers (an `image_processor_type` that ends in `Fast`). On Apple silicon and on x86 with AVX2, torchvision resizes a `:u8` image with the filters and the taps of Pillow, but its weights are 16-bit integers. So for `:bilinear`, `:bicubic` and `:lanczos`, the weights of an axis get the most fraction bits, up to 22, that keep the largest weight as an integer below 2^15. With fewer bits, a pixel can be one step away from Pillow when the image gets smaller. The other methods and the images of other types resize as with `:pillow`.

`:gaussian_linear` is the resize of the OWLv2 image processor of Python transformers, which is `skimage.transform.resize` with anti-aliasing. On each axis, a gaussian blur with sigma `max(0, (scale - 1) / 2)` comes first, where `scale` is the input size divided by the output size. Then a linear interpolation between the pixel centers gives the output. Both steps reflect the image at its edges, as the `"mirror"` mode of `scipy.ndimage`. The result is clipped to the minimum and the maximum of the image.

# `rgb`

Converts an image of shape `{height, width, channels}` to 3 channels, as `Image.convert("RGB")` of Pillow. A gray image repeats its channel, and the alpha channel is removed.

# `size`

Returns the `{height, width}` of the images that `process/2` gives, or `nil` when the size depends on the image, for example with `shortest_edge` and no crop.

# `tensor`

Returns `{:ok, tensor}` for an image, a `StbImage` struct or a tensor of shape `{height, width, channels}`. Returns `{:error, message}` for other values.

---

*Consult [api-reference.md](api-reference.md) for complete listing*