# `Transmuter.Hub` [🔗](https://github.com/georgeguimaraes/transmuter/blob/v2.0.0/lib/transmuter/hub.ex#L1) Reads the files of a model repository on the Hugging Face Hub or in a local directory. A source is `{:hf, repo}` or the path of a local directory. Hub files download into the cache directory and come from there later. When the metadata of the repository gives a new ETag for a file, the file downloads again. Each download must match its ETag. The cache directory is `~/.cache/transmuter`, or the `:cache_dir` option. An error status of the Hub, such as 401 for a gated repository without a token, raises `Transmuter.HubError`. ## Environment This module reads these environment variables, as Python `huggingface_hub` does: * `HF_TOKEN` (or the older `HUGGING_FACE_HUB_TOKEN`) - the token for private and gated repositories. Without it, the token comes from the file that `hf auth login` writes: `HF_TOKEN_PATH`, else `token` in `HF_HOME`, else `huggingface/token` in `XDG_CACHE_HOME`, else `~/.cache/huggingface/token`. A token from the browser login of `hf auth login` expires. Python `huggingface_hub` refreshes it in the token file when less than one day remains, on each command that reads the token, such as `hf auth whoami`. Transmuter only reads the file, so when the Hub gives status 401 for such a token, run `hf auth whoami`. * `HTTPS_PROXY` (and `HTTP_PROXY` for an `http` URL), else `ALL_PROXY` - the HTTP or HTTPS proxy, such as `http://user:password@proxy:3128`. A lowercase name wins over the uppercase name. * `NO_PROXY` - hosts that connect without the proxy, separated by commas. A host that ends with an entry connects directly, and `*` matches all hosts. The Hub redirects downloads to other hosts, such as `cas-bridge.xethub.hf.co`, and each host gets its own rule. * `HF_HUB_OFFLINE` and `TRANSFORMERS_OFFLINE` - see the `:offline` option. ## Options * `:cache_dir` - the directory for Hub downloads. The default is `~/.cache/transmuter`. * `:revision` - a branch, a tag or a commit hash of the repository. The default is `"main"`. The files of another revision go in a directory of their own in the cache, `@`. The files of a commit hash do not change, so its metadata comes from the Hub only once. * `:offline` - when `true`, no request goes to the Hub. Files and metadata come only from the cache, and a file that is not in the cache raises. A file in the cache is used also when its ETag differs from the saved metadata. The default is `true` when the `HF_HUB_OFFLINE` or `TRANSFORMERS_OFFLINE` environment variable is `1`, `ON`, `YES` or `TRUE`, as in Python, and `false` otherwise. The tasks of `Transmuter.Text`, `Transmuter.Vision` and `Transmuter.Audio`, `Transmuter.Repo.load!/2` and `Transmuter.load!/2` take these options too. A large download shows a progress bar on the standard output. To disable it, set `config :transmuter, progress_bar_enabled: false`. # `fetch` Returns `{:ok, path}` for `file` in `source`, or `:error` when the repository does not have the file. # `fetch!` Returns the local path of `file` in `source`. Raises if the repository does not have the file. A Hub file downloads when it is not in the cache or when its ETag in the metadata (see `info/2`) changed. An ONNX file also gets its external data files. # `files` Returns the names of all files in `source`, with paths relative to the repository root. # `info` Returns the metadata of a Hub repository from the Hub API, such as `"sha"` (the commit), `"siblings"` (the files with their ETags), `"pipeline_tag"` and `"cardData"`, or `nil` for a local directory. The metadata is saved in the cache directory and comes from there later. The `refresh: true` option reads it from the Hub again, and uses the saved metadata when the Hub cannot be reached, so a repository that was downloaded once also works without a network. With `offline: true`, the metadata comes only from the cache. # `json` Reads and decodes the JSON `file` in `source`, or returns `nil` when the repository does not have the file. # `options` Returns the names of the options of this module, which the tasks take and give to it. --- *Consult [api-reference.md](api-reference.md) for complete listing*