Transmuter.EMLX (transmuter v0.1.0)

Copy Markdown View Source

Moves a model's params to EMLX and keeps 4-bit and 8-bit weights packed.

MatMulNBits weights become MLX quantized tensors. Nx.dot then runs mx::quantized_matmul on the packed weights, so the float weights never exist in memory. ONNX packs the lowest bits first, which is also the MLX order, and an ONNX zero point becomes the MLX bias -zero_point * scale.

model = Transmuter.load!({:hf, "onnx-community/Qwen3-0.6B-ONNX", "onnx/model_q4.onnx"}, backend: EMLX.Backend)

Transmuter.load!/2 with backend: EMLX.Backend calls params/2 for you.

Weights that MLX cannot hold (other bit widths, group sizes other than 32, 64 or 128, or a K that is not a whole number of groups) stay packed bytes and are dequantized in the graph.

Summary

Functions

Returns model.params on EMLX, with compatible MatMulNBits weights quantized.

Functions

params(model, opts \\ [])

Returns model.params on EMLX, with compatible MatMulNBits weights quantized.

Options

  • :device - the EMLX device, :gpu (default) or :cpu