# `Transmuter.Audio.Featurizer` [🔗](https://github.com/georgeguimaraes/transmuter/blob/v2.0.0/lib/transmuter/audio/featurizer.ex#L1) Turns audio into the `input_values` input of an audio model, with the `preprocessor_config.json` of a model repository. featurizer = Transmuter.Audio.Featurizer.load!({:hf, "Xenova/ast-finetuned-audioset-10-10-0.4593"}) Transmuter.Audio.Featurizer.process(featurizer, [audio], backend: EMLX.Backend) #=> #Nx.Tensor The audio is a tensor of PCM samples `{samples}` at the sampling rate of the model, of any float type. `feature_extractor_type` in the configuration selects the steps of the feature extractor of Python transformers: * `"Wav2Vec2FeatureExtractor"` - the samples. When `do_normalize` is `true`, each audio gets a mean of 0 and a variance of 1: `(x - mean) / sqrt(variance + 1.0e-7)`. The result has the shape `{batch, samples}`, so the shape depends on the audio. * `"ASTFeatureExtractor"` - the log-mel filter bank of Kaldi. The result has the shape `{batch, max_length, num_mel_bins}` for all audio. The steps are those of the numpy path of Python transformers: 1. Cut the audio into frames of 400 samples every 160 samples. The last frame ends in the audio. 2. In each frame, subtract the mean, apply a preemphasis of 0.97 and multiply by a symmetric Hann window. 3. Take the power spectrum of 512 points, with zeros after the 400 samples. 4. Multiply by `num_mel_bins` mel filters from 20 Hz to half of the sampling rate. The filters are triangles on the Kaldi mel scale. 5. Take the natural log of the values, at least 1.1920929e-7. 6. Keep the first `max_length` frames. Shorter audio gets frames of zeros after its end. 7. When `do_normalize` is `true`, give `(x - mean) / (2 * std)` with `mean` and `std` of the configuration. * `"ClapFeatureExtractor"` - the log-mel spectrogram in dB of CLAP, as `input_features`. The result has the shape `{batch, 1, frames, feature_size}` for all audio. The steps are those of Python transformers with `truncation: "rand_trunc"`: 1. Fit the audio to `max_length_s` seconds. Shorter audio gets the `padding` of the configuration: `"repeatpad"` repeats the audio as many whole times as fit and adds zeros, `"repeat"` repeats the audio and cuts it, and `"pad"` adds zeros. Python transformers cuts longer audio at a random position. The featurizer keeps the first `max_length_s` seconds, which is one of the cuts that Python can give. 2. Add `fft_window_size / 2` samples at each end, which reflect the audio. 3. Cut the audio into frames of `fft_window_size` samples every `hop_length` samples, and multiply each frame by a periodic Hann window. 4. Take the power spectrum, and multiply it by `feature_size` mel filters from `frequency_min` to `frequency_max`, with the Slaney mel scale and the Slaney norm. 5. Give `10 * log10(x)`, with `x` at least 1.0e-10. When torchaudio is installed, Python transformers computes the AST features with `torchaudio.compliance.kaldi.fbank` in place of the numpy path. # `audio` Accepts one audio at `sampling_rate` in Hz, and returns `{:ok, chunks}` or `{:error, message}`. The chunks are an enumerable of f32 tensors of PCM samples `{samples}` on `Nx.BinaryBackend`, in the order of time. Audio without samples gives an error. The audio can be: * A tensor of PCM samples `{samples}` at `sampling_rate`, of any float type. It is one chunk. * `{:file, path}` - an audio or video file, which `ffmpeg` decodes into one chunk. The `ffmpeg` executable must be in the `PATH`. As `ffmpeg_read` of Python transformers, ffmpeg takes the first audio stream, mixes its channels into one, and changes its sampling rate to `sampling_rate`. The whole file decodes into memory: about 230 MB for one hour at 16 kHz. A path that does not exist, a directory and a file that ffmpeg cannot decode give an error. A named pipe or `/dev/stdin` goes to ffmpeg. * A stream of tensors `{samples}`: a `Stream`, a `File.Stream`, an `IO.Stream` or a stream function. The tensors are the chunks. This function reads the first chunk and checks it, so an empty stream or a first chunk that is not a tensor `{samples}` gives an error. The other chunks come when the enumerable gives them, and a later chunk that is not a tensor `{samples}` raises when it comes. A list is not a stream, because a list of inputs is a batch. # `hann_window` Returns the Hann window of `size` samples as an f32 tensor. A periodic window, as `torch.hann_window`, divides by `size`. A symmetric window, as `numpy.hanning`, divides by `size - 1`. # `input_name` Returns the name of the model input of the features: `"input_features"` for CLAP, and `"input_values"` for the others, as `model_input_names` of the feature extractors of Python transformers. # `load!` Loads the featurizer of a `Transmuter.Repo` or of a source (`{:hf, repo}` or a local directory) from its `preprocessor_config.json`. Options go to `Transmuter.Repo.load!/2` when the argument is a source. # `mel_filters` Returns the mel filters `{bins, mels}` of `mel_filter_bank` in Python transformers, for `bins` frequency bins from 0 Hz to half of `sampling_rate`. Filter `m` is a triangle that rises from point `m` to point `m + 1` and falls to point `m + 2`. The points are evenly spaced on the mel scale from `:min_frequency` to `:max_frequency`. ## Options * `:min_frequency` and `:max_frequency` - in Hz. * `:mel_scale` - `:slaney` or `:kaldi`. * `:norm` - `:slaney` multiplies each filter by 2 divided by its width, so each filter has about the same energy. The default is `nil`. * `:triangularize_in_mel_space` - `true` makes the triangles on the mel scale, as Kaldi does. The default is `false`, which makes the triangles in Hz. # `new` Builds a featurizer from the map of a `preprocessor_config.json`, with the defaults of Python transformers. # `power_spectrogram` Returns the power spectrogram `{batch, frames, bins}` of audio `{batch, samples}`, as `spectrogram` of Python transformers with `center=True` and `power=2.0`. The audio gets `size / 2` samples at each end, which reflect the audio without the edge sample. Each frame of `size` samples every `hop_length` samples is multiplied by `window` `{size}`, and gives the power of its `size / 2 + 1` frequency bins. # `process` Turns a list of audio into the features `{batch, ...}` on the `:backend` of `opts`. Without `:backend`, the featurizer uses `Nx.default_backend/0`. For a featurizer whose `shape/1` is `nil`, the audio of a list must have one length. # `shape` Returns the shape of the features of one audio that `process/3` gives, or `nil` when the shape depends on the audio. --- *Consult [api-reference.md](api-reference.md) for complete listing*