# `Transmuter.Fusion`
[🔗](https://github.com/georgeguimaraes/transmuter/blob/v2.0.0/lib/transmuter/fusion.ex#L1)

Replaces normalizations and attention cores that an export writes as separate operators with one operator.

Many exports, for example from Optimum, write RMSNorm, LayerNorm and the attention core with primitive operators. `Transmuter.load!/2` replaces these subgraphs:

  * `Pow(x, 2)`, `ReduceMean`, `Add(eps)`, `Sqrt`, then `x / sqrt`, `x * (1 / sqrt)` or `x * Reciprocal(sqrt)`, then `Mul(weight)` becomes `SimplifiedLayerNormalization`.
  * `ReduceMean`, `Sub`, the RMSNorm of the difference, then `Mul(gamma)` and `Add(beta)` becomes `LayerNormalization`.
  * `MatMul(q, k)`, an optional `Mul` or `Div` by a scale, an optional `Cast` to a float type, an optional `Where(keep, scores, fill)`, an optional `Add(mask)`, `Softmax`, an optional `Cast` to a float type, an optional pair of `Reshape` and `MatMul(probabilities, v)` becomes `AttentionCore` (see `Transmuter.Ops.Transformer`). GPT-2 exports write the attention core in this way.

The operators then use the fused kernels of `EMLX.Fast` on EMLX (see `Transmuter.Ops.emlx?/1`), and every other backend runs the same function as before.

Each replacement node has a `metadata_props` entry with the key `transmuter.fused`. Its value lists the op type and the name of each node that the replacement replaces, the anchor first, separated by ", ". `Transmuter.Graph` shows these lists.

A subgraph matches only when the replacement computes the same function:

  * The exponent is 2, the numerator of `1 / sqrt` is 1, and eps and a `Mul` scale are constants with one value. A `Div` by a value that is not a constant becomes the divisor input of `AttentionCore`, which reads it when the graph runs.
  * The fill of the `Where` is a constant of at most -1.0e4, so the hidden keys get no weight.
  * Eps is an f32, f16 or bf16 value. It has the type of x, so x is not f64.
  * `ReduceMean` reduces the axis -1 and keeps it.
  * The weight, gamma and beta are constants of at most one axis, so they broadcast only along the last axis.
  * No value inside the subgraph has a consumer outside it, also not a graph output or a subgraph.

Without shapes at load time, `AttentionCore` checks the shapes when it runs and uses the fused kernel only for 4-D heads, or 3-D heads in the batch as in Whisper and TrOCR exports, with a softmax over the last axis. A `Reshape` pair of the probabilities, as in the Whisper exports of Xenova, must give back the shape of the probabilities. Else the core runs as the operators of the graph.

# `graph`

Rewrites the matching subgraphs of `graph` and of its subgraphs.

`opset` is the ONNX opset of the model. `outer` holds the constant tensors of the enclosing graphs, by name.

---

*Consult [api-reference.md](api-reference.md) for complete listing*