Turns audio into the input_values input of an audio model, with the preprocessor_config.json of a model repository.
featurizer = Transmuter.Audio.Featurizer.load!({:hf, "Xenova/ast-finetuned-audioset-10-10-0.4593"})
Transmuter.Audio.Featurizer.process(featurizer, [audio], backend: EMLX.Backend)
#=> #Nx.Tensor<f32[1][1024][128]>The audio is a tensor of PCM samples {samples} at the sampling rate of the model, of any float type. feature_extractor_type in the configuration selects the steps of the feature extractor of Python transformers:
"Wav2Vec2FeatureExtractor"- the samples. Whendo_normalizeistrue, each audio gets a mean of 0 and a variance of 1:(x - mean) / sqrt(variance + 1.0e-7). The result has the shape{batch, samples}, so the shape depends on the audio."ASTFeatureExtractor"- the log-mel filter bank of Kaldi. The result has the shape{batch, max_length, num_mel_bins}for all audio. The steps are those of the numpy path of Python transformers:- Cut the audio into frames of 400 samples every 160 samples. The last frame ends in the audio.
- In each frame, subtract the mean, apply a preemphasis of 0.97 and multiply by a symmetric Hann window.
- Take the power spectrum of 512 points, with zeros after the 400 samples.
- Multiply by
num_mel_binsmel filters from 20 Hz to half of the sampling rate. The filters are triangles on the Kaldi mel scale. - Take the natural log of the values, at least 1.1920929e-7.
- Keep the first
max_lengthframes. Shorter audio gets frames of zeros after its end. - When
do_normalizeistrue, give(x - mean) / (2 * std)withmeanandstdof the configuration.
"ClapFeatureExtractor"- the log-mel spectrogram in dB of CLAP, asinput_features. The result has the shape{batch, 1, frames, feature_size}for all audio. The steps are those of Python transformers withtruncation: "rand_trunc":- Fit the audio to
max_length_sseconds. Shorter audio gets thepaddingof the configuration:"repeatpad"repeats the audio as many whole times as fit and adds zeros,"repeat"repeats the audio and cuts it, and"pad"adds zeros. Python transformers cuts longer audio at a random position. The featurizer keeps the firstmax_length_sseconds, which is one of the cuts that Python can give. - Add
fft_window_size / 2samples at each end, which reflect the audio. - Cut the audio into frames of
fft_window_sizesamples everyhop_lengthsamples, and multiply each frame by a periodic Hann window. - Take the power spectrum, and multiply it by
feature_sizemel filters fromfrequency_mintofrequency_max, with the Slaney mel scale and the Slaney norm. - Give
10 * log10(x), withxat least 1.0e-10.
- Fit the audio to
When torchaudio is installed, Python transformers computes the AST features with torchaudio.compliance.kaldi.fbank in place of the numpy path.
Summary
Functions
Accepts one audio at sampling_rate in Hz, and returns {:ok, chunks} or {:error, message}. The chunks are an enumerable of f32 tensors of PCM samples {samples} on Nx.BinaryBackend, in the order of time. Audio without samples gives an error. The audio can be
Returns the Hann window of size samples as an f32 tensor. A periodic window, as torch.hann_window, divides by size. A symmetric window, as numpy.hanning, divides by size - 1.
Returns the name of the model input of the features: "input_features" for CLAP, and "input_values" for the others, as model_input_names of the feature extractors of Python transformers.
Loads the featurizer of a Transmuter.Repo or of a source ({:hf, repo} or a local directory) from its preprocessor_config.json.
Returns the mel filters {bins, mels} of mel_filter_bank in Python transformers, for bins frequency bins from 0 Hz to half of sampling_rate. Filter m is a triangle that rises from point m to point m + 1 and falls to point m + 2. The points are evenly spaced on the mel scale from :min_frequency to :max_frequency.
Builds a featurizer from the map of a preprocessor_config.json, with the defaults of Python transformers.
Returns the power spectrogram {batch, frames, bins} of audio {batch, samples}, as spectrogram of Python transformers with center=True and power=2.0. The audio gets size / 2 samples at each end, which reflect the audio without the edge sample. Each frame of size samples every hop_length samples is multiplied by window {size}, and gives the power of its size / 2 + 1 frequency bins.
Turns a list of audio into the features {batch, ...} on the :backend of opts. Without :backend, the featurizer uses Nx.default_backend/0. For a featurizer whose shape/1 is nil, the audio of a list must have one length.
Returns the shape of the features of one audio that process/3 gives, or nil when the shape depends on the audio.
Functions
Accepts one audio at sampling_rate in Hz, and returns {:ok, chunks} or {:error, message}. The chunks are an enumerable of f32 tensors of PCM samples {samples} on Nx.BinaryBackend, in the order of time. Audio without samples gives an error. The audio can be:
A tensor of PCM samples
{samples}atsampling_rate, of any float type. It is one chunk.{:file, path}- an audio or video file, whichffmpegdecodes into one chunk. Theffmpegexecutable must be in thePATH. Asffmpeg_readof Python transformers, ffmpeg takes the first audio stream, mixes its channels into one, and changes its sampling rate tosampling_rate. The whole file decodes into memory: about 230 MB for one hour at 16 kHz. A path that does not exist, a directory and a file that ffmpeg cannot decode give an error. A named pipe or/dev/stdingoes to ffmpeg.A stream of tensors
{samples}: aStream, aFile.Stream, anIO.Streamor a stream function. The tensors are the chunks. This function reads the first chunk and checks it, so an empty stream or a first chunk that is not a tensor{samples}gives an error. The other chunks come when the enumerable gives them, and a later chunk that is not a tensor{samples}raises when it comes. A list is not a stream, because a list of inputs is a batch.
Returns the Hann window of size samples as an f32 tensor. A periodic window, as torch.hann_window, divides by size. A symmetric window, as numpy.hanning, divides by size - 1.
Returns the name of the model input of the features: "input_features" for CLAP, and "input_values" for the others, as model_input_names of the feature extractors of Python transformers.
Loads the featurizer of a Transmuter.Repo or of a source ({:hf, repo} or a local directory) from its preprocessor_config.json.
Options go to Transmuter.Repo.load!/2 when the argument is a source.
Returns the mel filters {bins, mels} of mel_filter_bank in Python transformers, for bins frequency bins from 0 Hz to half of sampling_rate. Filter m is a triangle that rises from point m to point m + 1 and falls to point m + 2. The points are evenly spaced on the mel scale from :min_frequency to :max_frequency.
Options
:min_frequencyand:max_frequency- in Hz.:mel_scale-:slaneyor:kaldi.:norm-:slaneymultiplies each filter by 2 divided by its width, so each filter has about the same energy. The default isnil.:triangularize_in_mel_space-truemakes the triangles on the mel scale, as Kaldi does. The default isfalse, which makes the triangles in Hz.
Builds a featurizer from the map of a preprocessor_config.json, with the defaults of Python transformers.
Returns the power spectrogram {batch, frames, bins} of audio {batch, samples}, as spectrogram of Python transformers with center=True and power=2.0. The audio gets size / 2 samples at each end, which reflect the audio without the edge sample. Each frame of size samples every hop_length samples is multiplied by window {size}, and gives the power of its size / 2 + 1 frequency bins.
Turns a list of audio into the features {batch, ...} on the :backend of opts. Without :backend, the featurizer uses Nx.default_backend/0. For a featurizer whose shape/1 is nil, the audio of a list must have one length.
Returns the shape of the features of one audio that process/3 gives, or nil when the shape depends on the audio.