# `Transmuter.Audio.WordTimestamps`
[🔗](https://github.com/georgeguimaraes/transmuter/blob/v2.0.0/lib/transmuter/audio/word_timestamps.ex#L1)

The word timestamps of Whisper, as `return_timestamps="word"` in the `automatic-speech-recognition` pipeline of Python transformers 4. See `Transmuter.Audio.speech_to_text/2`.

The time of each token comes from the cross attentions of the decoder, as `_extract_token_timestamps` of `WhisperForConditionalGeneration` computes it:

  1. After each pass of generation, the decoder runs again over the prompt and the new tokens, without the cache. This run gives the cross attentions that Python collects in the steps of generation. Only the heads of `alignment_heads` in `generation_config.json` stay.
  2. The rows of the matrix are the new tokens that were an input of a step: all new tokens when the pass ends with eos, else all tokens but the last. The columns are the encoder positions of the audio after the seek of the pass. A position is two feature frames.
  3. Each head is normalized over the rows to a mean of 0 and a standard deviation of 1, and filtered along the columns with the median of `median_filter_width` values (7). The matrix is the mean of the heads.
  4. Dynamic time warping finds the path of the least cost through the negated matrix. The time of a row is the column where the path enters the row, at 0.02 s for each column. A token without a row gets the time of the last row.

The words come from `_combine_tokens_into_words` of the Whisper tokenizer. The tokens split where their text is complete characters. A piece that starts with a space or that is punctuation starts a new word, and other pieces join the word before them. In Chinese, Japanese, Thai, Lao, Burmese and Cantonese, each piece is a word, as OpenAI whisper and transformers.js do. The pipeline of Python transformers does not know the language there, so it splits all languages at spaces. Then opening punctuation joins the next word, and closing punctuation joins the word before it.

# `aligner`

Returns a function that takes the defn options, compiles the alignment of `decoder`, and returns a function that gives the time of each new token. Raises an `ArgumentError` when `generation_config.json` has no `alignment_heads` or the decoder does not give their cross attentions.

The compiled function takes the outputs of the encoder, the prompts, the new tokens of each row (without eos), the seek of each row and the number of feature frames of the audio of each row. It returns the end time of each token in feature frames from the start of the window. A row without a prompt is a padding row.

`settings` has `:prompt_length`, `:max_new_tokens`, `:frames_per_timestamp` and `:positions`, the largest number of tokens of the decoder.

# `outputs`

Returns the names of the outputs of a decoder that `aligner/3` can read: the cross attentions of all layers. Give these names with those of `Transmuter.Generation.outputs/1` to the `:outputs` option of `Transmuter.load!/2`.

# `words`

Returns the words of `elements`, a list of tuples whose first value is a token, as `{text, elements}` with the elements of each word. `language` is the language code of the text, which is `"en"` for a translation.

---

*Consult [api-reference.md](api-reference.md) for complete listing*