Interop

PyTorch and HuggingFace plumbing: applying a PyTorch state_dict to a Lux (ps, st) pair, resolving and caching weights through the HuggingFace Hub, and loading .safetensors blobs.

State-dict application

Luximm.Interop.apply_state_dictFunction
apply_state_dict(ps, state_dict, mapping) -> ps

Rebuild a Lux parameter NamedTuple by replacing leaves according to mapping, an iterable of (pytorch_key, lux_path, transform) triples where:

  • pytorch_key: dotted name as written by Python's model.state_dict().keys().
  • lux_path: tuple of Symbols naming the leaf in ps, e.g. (:stage1, :layer_1, :norm1, :gn, :scale).
  • transform: function Array{Float32} -> Array{Float32} applied to the raw HDF5-read array. Common transforms are identity (HDF5-natural matches Lux) and axis_reverse (full axis reversal, used when the Julia tensor layout matches PyTorch's logical axis order rather than its reversed storage order).

The original ps is not mutated; the caller must bind the return value.

source

HuggingFace Hub

Luximm.Interop.hf_hub_downloadFunction
hf_hub_download(repo_id, filename; revision="main",
                 cache_dir=hf_hub_cache_dir(),
                 repo_type="model") -> String

Resolve <repo_id>/<filename> against revision (a branch, tag, or commit) and return the local snapshot path, downloading only what is not already cached. The on-disk layout matches huggingface_hub:

<cache_dir>/models--<org>--<name>/blobs/<etag>
<cache_dir>/models--<org>--<name>/snapshots/<commit>/<filename>
    -> ../../blobs/<etag>
<cache_dir>/models--<org>--<name>/refs/<revision>     # text: <commit>

This means a timm.create_model(..., pretrained=True) call and a hf_hub_download call against the same repo see each other's cached blob.

The function always performs a no-redirect HEAD against https://huggingface.co/<repo_id>/resolve/<revision>/<filename> to look up the current commit (X-Repo-Commit) and the blob's etag (X-Linked-ETag for LFS-backed files, ETag otherwise). If the HEAD fails (e.g. offline), the function falls back to the most recently recorded commit in refs/<revision> and returns the existing snapshot path if present; otherwise the original error is rethrown.

Set repo_type="dataset" for dataset repos; default "model" matches timm's usage.

source
Luximm.Interop.hf_downloadFunction
hf_download(url, dest) -> String

Download url to dest unless dest already exists. Returns dest.

If HUGGING_FACE_HUB_TOKEN is set, it is sent as a Bearer token in the Authorization header. The download streams into a sibling temp file and is renamed into place only on success, so an interrupted call (network drop, ^C) never leaves a partial file at dest that a later call would mistake for a cache hit.

source
Luximm.Interop.hf_hub_cache_dirFunction
hf_hub_cache_dir() -> String

Cache root that matches huggingface_hub (Python). Honors the same env-var precedence:

  1. HF_HUB_CACHE if set,
  2. otherwise $HF_HOME/hub if HF_HOME is set,
  3. otherwise ~/.cache/huggingface/hub.

Files downloaded into this directory by Luximm are visible to timm / huggingface_hub, and vice versa.

source

SafeTensors

Luximm.Interop.load_safetensors_state_dictFunction
load_safetensors_state_dict(path; reverse_axes=true) -> Dict{String, Array{Float32}}

Read a .safetensors file from disk into a dict of Float32 arrays.

When reverse_axes = true (default), every tensor's axes are reversed so the resulting layout matches read_parity's HDF5-natural Julia layout (PyTorch axes reversed). Set reverse_axes = false to keep PyTorch's logical axis order if a caller wants that layout explicitly.

source