Reads the files of a model repository on the Hugging Face Hub or in a local directory.

A source is {:hf, repo} or the path of a local directory. Hub files download into the cache directory and come from there later. When the metadata of the repository gives a new ETag for a file, the file downloads again. Each download must match its ETag. The cache directory is ~/.cache/transmuter, or the :cache_dir option. An error status of the Hub, such as 401 for a gated repository without a token, raises Transmuter.HubError.

Environment

This module reads these environment variables, as Python huggingface_hub does:

• HF_TOKEN (or the older HUGGING_FACE_HUB_TOKEN) - the token for private and gated repositories. Without it, the token comes from the file that hf auth login writes: HF_TOKEN_PATH, else token in HF_HOME, else huggingface/token in XDG_CACHE_HOME, else ~/.cache/huggingface/token. A token from the browser login of hf auth login expires. Python huggingface_hub refreshes it in the token file when less than one day remains, on each command that reads the token, such as hf auth whoami. Transmuter only reads the file, so when the Hub gives status 401 for such a token, run hf auth whoami.

• HTTPS_PROXY (and HTTP_PROXY for an http URL), else ALL_PROXY - the HTTP or HTTPS proxy, such as http://user:password@proxy:3128. A lowercase name wins over the uppercase name.

• NO_PROXY - hosts that connect without the proxy, separated by commas. A host that ends with an entry connects directly, and * matches all hosts. The Hub redirects downloads to other hosts, such as cas-bridge.xethub.hf.co, and each host gets its own rule.

• HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE - see the :offline option.

Options

• :cache_dir - the directory for Hub downloads. The default is ~/.cache/transmuter.

• :revision - a branch, a tag or a commit hash of the repository. The default is "main". The files of another revision go in a directory of their own in the cache, <repo>@<revision>. The files of a commit hash do not change, so its metadata comes from the Hub only once.

• :offline - when true, no request goes to the Hub. Files and metadata come only from the cache, and a file that is not in the cache raises. A file in the cache is used also when its ETag differs from the saved metadata. The default is true when the HF_HUB_OFFLINE or TRANSFORMERS_OFFLINE environment variable is 1, ON, YES or TRUE, as in Python, and false otherwise.

The tasks of Transmuter.Text, Transmuter.Vision and Transmuter.Audio, Transmuter.Repo.load!/2 and Transmuter.load!/2 take these options too.

A large download shows a progress bar on the standard output. To disable it, set config :transmuter, progress_bar_enabled: false.