Flaky connections like Starlink crashes hf xet large downloads

[twnskai05]

huggingface_hub sometimes crashes on big files when on networks like Starlink … is there a work-around ? … GPT says to disable Xet, concurrency, etc … but this prevents big file downloads >32GB … see crash error message below … also, mid-download CACHE of many many GBs gets WIPED-OUT and cannot resume … please fix ASAP …

(uvpvenv) MINGW64
$ hf download ggml-org/Qwen3-Coder-Next-GGUF --repo-type=model --local-dir ./models–ggml-org–Qwen3-Coder-Next-GGUF/
Fetching 3 files: 67%|████████████████████████████████████████████████████████████████████████████████ | 2/3 [2:05:18<1:02:39, 3759.30s/it]
Traceback (most recent call last):.): 8%|████████▏ | 6.53GB / 84.8GB, 32.1MB/s
File “”, line 203, in _run_module_as_main | 0/3 [00:00<?, ?it/s]
File “”, line 88, in run_code
File “C:\msys64\home\utils\uvpvenv\Scripts\hf.exe_main.py”, line 10, in
sys.exit(main())

File "C:\msys64\home\utils\uvpvenv\Lib\site-packages\huggingface_hub\cli\hf.py", line 133, in main
app()
~~~^^
File "C:\msys64\home\utils\uvpvenv\Lib\site-packages\click\core.py", line 1569, in call
return self.main(*args, **kwargs)
~~~~~~~~~^^^^^^^^^^^^^^^^^
File "C:\msys64\home\utils\uvpvenv\Lib\site-packages\click\core.py", line 1490, in main
rv = self.invoke(ctx)
File "C:\msys64\home\utils\uvpvenv\Lib\site-packages\huggingface_hub\cli_cli_utils.py", line 131, in invoke
return super().invoke(ctx)
~~~~~~~~~~~~~~^^^^^
File "C:\msys64\home\utils\uvpvenv\Lib\site-packages\click\core.py", line 1970, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "C:\msys64\home\utils\uvpvenv\Lib\site-packages\click\core.py", line 1353, in invoke
return ctx.invoke(self.callback, **ctx.params)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\msys64\home\utils\uvpvenv\Lib\site-packages\click\core.py", line 907, in invoke
return callback(*args, **kwargs)
File "C:\msys64\home\utils\uvpvenv\Lib\site-packages\huggingface_hub\cli_framework.py", line 365, in handler
return func(**call_kwargs)
File "C:\msys64\home\utils\uvpvenv\Lib\site-packages\huggingface_hub\cli\download.py", line 225, in download
_print_result(run_download())
~~~~~~~~~~~~^^
File "C:\msys64\home\utils\uvpvenv\Lib\site-packages\huggingface_hub\cli\download.py", line 187, in run_download
return snapshot_download(
repo_id=repo_id,
...<10 lines>...
dry_run=dry_run,
)
File "C:\msys64\home\utils\uvpvenv\Lib\site-packages\huggingface_hub\utils_validators.py", line 88, in _inner_fn
return fn(*args, **kwargs)
File "C:\msys64\home\utils\uvpvenv\Lib\site-packages\huggingface_hub_snapshot_download.py", line 522, in snapshot_download
hf_thread_map(
~~~~~~~~~~~~~^
_inner_hf_hub_download,
^^^^^^^^^^^^^^^^^^^^^^^
...<3 lines>...
tqdm_class=tqdm_class,
^^^^^^^^^^^^^^^^^^^^^^
)
^
File "C:\msys64\home\utils\uvpvenv\Lib\site-packages\huggingface_hub\utils\tqdm.py", line 441, in hf_thread_map
results[future_to_index[future]] = future.result()
~~~~~~~~~~~~~^^
File "C:\Users\AppData\Roaming\uv\python\cpython-3.14-windows-x86_64-none\Lib\concurrent\futures_base.py", line 447, in result
return self.__get_result()
~~~~~~~~~~~~~~~~~^^
File "C:\Users\AppData\Roaming\uv\python\cpython-3.14-windows-x86_64-none\Lib\concurrent\futures_base.py", line 396, in __get_result
raise self._exception
File "C:\Users\AppData\Roaming\uv\python\cpython-3.14-windows-x86_64-none\Lib\concurrent\futures\thread.py", line 86, in run
result = ctx.run(self.task)
File "C:\Users\AppData\Roaming\uv\python\cpython-3.14-windows-x86_64-none\Lib\concurrent\futures\thread.py", line 73, in run
return fn(*args, **kwargs)
File "C:\msys64\home\utils\uvpvenv\Lib\site-packages\huggingface_hub_snapshot_download.py", line 502, in _inner_hf_hub_download
hf_hub_download( # type: ignore
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
repo_id,
^^^^^^^^
...<14 lines>...
dry_run=dry_run,
^^^^^^^^^^^^^^^^
)
^
File "C:\msys64\home\utils\uvpvenv\Lib\site-packages\huggingface_hub\utils_validators.py", line 88, in _inner_fn
return fn(*args, **kwargs)
File "C:\msys64\home\utils\uvpvenv\Lib\site-packages\huggingface_hub\file_download.py", line 998, in hf_hub_download
return _hf_hub_download_to_local_dir(
# Destination
...<16 lines>...
dry_run=dry_run,
)
File "C:\msys64\home\utils\uvpvenv\Lib\site-packages\huggingface_hub\file_download.py", line 1457, in _hf_hub_download_to_local_dir
_download_to_tmp_and_move(
~~~~~~~~~~~~~~~~~~~~~~~~~^
incomplete_path=paths.incomplete_path(etag),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<8 lines>...
tqdm_class=tqdm_class,
^^^^^^^^^^^^^^^^^^^^^^
)
^
File "C:\msys64\home\utils\uvpvenv\Lib\site-packages\huggingface_hub\file_download.py", line 1960, in _download_to_tmp_and_move
xet_get(
~~~~~~~^
incomplete_path=tmp_path,
^^^^^^^^^^^^^^^^^^^^^^^^^
...<4 lines>...
tqdm_class=tqdm_class,
^^^^^^^^^^^^^^^^^^^^^^
)
^
File "C:\msys64\home\utils\uvpvenv\Lib\site-packages\huggingface_hub\file_download.py", line 565, in xet_get
with session.new_file_download_group(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
token_refresh_url=xet_file_data.refresh_route,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<2 lines>...
progress_callback=progress.update_progress,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
) as group:
^
RuntimeError: Task error: File reconstruction error: CAS Client Error: Format error: I/O error: error decoding response body


-----------------------------


$ uv pip list
Using Python 3.14.6 environment at: C:\msys64\home\utils\uvpvenv
Package           Version
----------------- ---------
anyio             4.14.2
certifi           2026.7.22
click             8.4.2
colorama          0.4.6
filelock          3.32.3
fsspec            2026.7.0
h11               0.16.0
hf-xet            1.6.0
httpcore          1.0.9
httpx             0.28.1
huggingface-hub   1.27.0
idna              3.18
packaging         26.3
pyyaml            6.0.3
tqdm              4.70.0
typing-extensions 4.16.0
(uvpvenv)


-----------------------------

[John6666]

I may have reproduced a similar failure mode in Colab:

---

I would not treat this as Starlink-specific yet.

I made a small CPU-only fault-injection control with the same client versions you reported — huggingface_hub 1.27.0 and hf-xet 1.6.0 — using a public Xet-backed file of about 350 MiB.

The clean control completed normally. Then I repeated the download from a fresh cache while deliberately truncating the encrypted HTTPS tunnels carrying Xet storage/CDN traffic. I also pinned Xet’s internal download concurrency to 1, so high/adaptive concurrency was not required for the failure.

That reached the same terminal error class:

File reconstruction error:
CAS Client Error: Format error: I/O error:
error decoding response body

The Xet log was particularly useful: the range request retried through the configured retry budget and eventually ended with the equivalent of:

retry 1
retry 2
retry 3
retry 4
retry 5
No more retries; aborting:
Format error: I/O error: error decoding response body

I put the executed control here so the exact setup is inspectable/re-runnable:

HF Xet mid-response interruption reproducer — CPU Colab

Important caveat: this is not a Starlink reproduction, and matching the terminal error does not establish that your failure had the same root cause. It only shows that a generic mid-response connection interruption is sufficient to drive hf-xet 1.6.0 into this failure class under at least one controlled setup.

There is also an open xet-core issue #920 for the same error decoding response body terminal signature. That issue does not currently establish a root cause either, so I would treat it as a useful signature match rather than an explanation by itself.

The cheapest next check

Before re-downloading another ~85 GB, I think the highest-information/lowest-cost thing would be to look at the existing Xet log around the failure.

In particular, I would look for something like:

s3::get_range
retry ...
...
No more retries; aborting
error decoding response body

and also terms such as:

UnexpectedEof
IncompleteMessage
ConnectionReset
timeout

The current Xet storage documentation documents a default maximum of 5 retry attempts, along with the retry-duration and HTTP timeout controls.

So the first useful split is roughly:

body/range/network-looking failure
        |
        +--> retries 1 ... 5
        |        |
        |        +--> "No more retries; aborting"
        |                 |
        |                 +--> error decoding response body
        |
        +--> looks quite close to the controlled failure above

versus

error decoding response body
without the same retry sequence
        |
        +--> probably worth treating as a different branch
             rather than assuming the same mechanism

If your existing log already contains that sequence, it would be much more informative than another blind 84.8 GB retry.

Why I think the restart cost is a separate part of the problem

There seem to be two distinct questions here:

1. Why did the Xet reconstruction become fatal?
2. Why is a fatal reconstruction so expensive to recover from?

The Colab control says something useful about (1): transient mid-body interruption can be enough to reach this terminal error.

But (2) is a separate design issue.

In June 2026, huggingface_hub PR #4306 deliberately changed the download/cache path so that each download writes into a process-unique temporary file rather than resuming a shared .incomplete file from a previous call.

That change was made for a real correctness reason: on Lustre, GPFS and some NFS configurations, apparently successful file locking could allow two processes to append to the same incomplete file and even produce silent cache corruption.

The trade-off is stated quite directly in that PR: a failed download is no longer resumed by a later call, while in-process request retry/resume is unaffected.

So there are two recovery levels that are easy to conflate:

one HTTP/Xet request fails
        |
        +--> retry inside the same download operation

whole reconstruction becomes fatal
        |
        +--> later hf_hub_download invocation
             is a different recovery problem

That distinction seems especially important on a flaky connection: request-level retry may work perfectly well most of the time, but if enough interruptions exhaust that retry path, the cost of crossing the fatal boundary is enormous for an 84.8 GB object.

Why disabling Xet is not really an escape hatch here

One small numeric correction: the current regular-HTTP client limit is 50 GB, rather than 32 GB.

But that does not change the practical situation for an 84.8 GB file — it is still over the limit, so:

export HF_HUB_DISABLE_XET=1

is not a general fallback for this case.

There is already an open huggingface_hub issue #3868 describing almost exactly this constraint structure:

• file is over 50 GB;
• ordinary HTTP path is refused;
• Xet therefore becomes the supported path;
• a network interruption can make restarting the huge transfer very expensive;
• resumability in Xet mode would therefore be especially valuable.

I would not assume #3868 is the same underlying bug as this thread; it is useful because the large-file / no-HTTP-fallback / restart-cost combination is essentially the same problem shape.

A small decision tree for further testing

If another attempt is practical, I would change only one thing at a time.

A. The log shows retry exhaustion like the Colab control

For example:

range/body failure
retry 1
...
retry 5
No more retries; aborting
error decoding response body

Then transient transport failure becomes a much stronger working hypothesis.

At that point there are two relatively inexpensive controls.

1. Pin Xet’s internal download concurrency

export HF_XET_FIXED_DOWNLOAD_CONCURRENCY=1

The Xet docs say this bypasses the adaptive controller and fixes the internal Xet concurrency to the specified value.

This is not the same control as:

hf download --max-workers=1 ...

--max-workers is about concurrent file-download workers at the Hub API level; HF_XET_FIXED_DOWNLOAD_CONCURRENCY controls the parallelism inside Xet’s transfer path.

My fault-injection control still failed with Xet concurrency pinned to 1, so I would not expect this setting to magically fix arbitrary connection loss. Its value is diagnostic:

concurrency=1 becomes stable
        -> concurrency may be amplifying the problem

concurrency=1 still fails
        -> high/adaptive concurrency is not required
           and the basic transport/retry path becomes more interesting

2. If the log really shows retry exhaustion, increase retry tolerance modestly

The same documentation currently exposes:

HF_XET_CLIENT_RETRY_MAX_ATTEMPTS
HF_XET_CLIENT_RETRY_MAX_DURATION
HF_XET_CLIENT_CONNECT_TIMEOUT
HF_XET_CLIENT_READ_TIMEOUT

with documented defaults including:

retry attempts: 5
total retry duration: 360s
connect timeout: 60s
read timeout: 120s

I would consider those a mitigation experiment, not a diagnosis.

If increasing the retry budget lets the same flaky path finish, the safe conclusion would be something like:

transient failures were recoverable when Xet was allowed to tolerate them for longer

—not:

the root cause was definitely Starlink.

B. There is no retry sequence before the terminal error

Then I would stop trying to map it onto the Colab reproducer.

The same final message can apparently be reached through more than one internal path, and xet-core #920 is still open.

That branch would probably benefit more from the exact Xet log than from changing network settings.

C. It repeatedly fails at the same object/range even on another network

That would make a content/range/Xet-specific path more interesting than generic flaky connectivity.

D. The same download is stable on another network

That would increase the likelihood of some network-path interaction — transient connectivity, routing/CDN path, etc. — but still would not uniquely identify Starlink as the cause.

A related older case that may be useful for comparison

There is also an older huggingface_hub issue #3036 from a normal slow residential connection, not Starlink.

That report contains Xet-side errors including roughly:

Decode
UnexpectedEof
IncompleteBody

and on a later attempt:

Connection reset by peer

It used an older Xet stack, so I would not treat it as the same bug.

But it is useful evidence that body-truncation / connection-reset-shaped failures are not unique to satellite links.

About the Xet chunk cache

I also tried the documented Xet chunk-cache knob as a separate recovery experiment.

The current Hub environment-variable documentation says the download chunk cache is disabled by default and recommends a large value (10 GB or greater) when enabling it.

I tested:

HF_XET_CHUNK_CACHE_SIZE_BYTES=12000000000

with hf-xet 1.6.0.

Xet’s own log recognized the 12 GB setting, but in my small control I could not get useful persistent chunk-cache data to materialize, and deleting only the Hub file cache caused essentially the entire object to be transferred again.

After an injected terminal failure, retrying with the same cache root also transferred essentially the full clean-baseline amount again.

That resembles xet-core #715, where a user configured a 10 GB chunk cache, Xet logged the configured value, but no useful chunk cache appeared and interrupted downloads restarted from zero.

However, #715 was reported against hf-xet 1.4.2, so I do not know whether my observation on 1.6.0 has the same cause.

I also would not equate the Xet chunk cache with file-level resumability: the documented chunk cache is an optimization for reusing content-addressed chunks/ranges, whereas the .incomplete / later-call resume behavior is a separate layer.

So I would keep this as a secondary observation rather than making it the main diagnosis.

If the ordinary Xet log is not enough

xet-core now has official diagnostic wrappers for Linux, macOS, and Windows Git-Bash.

Since your example is already running under MINGW/Git-Bash, the Windows path is directly relevant:

hf-xet diagnostic scripts

The documented Windows wrapper is:

./scripts/diag/hf-xet-diag-windows.sh -- python your-download-script.py

It collects console/environment information and Windows mini-dumps with matching debug symbols.

I would only go this far if the normal Xet log is inconclusive, though. Looking at the existing failure window is much cheaper.

Also, before posting diagnostic output publicly, I would check it for tokens, authorization data, or signed-URL query parameters.

So my current best guess is not really “Starlink breaks Xet.”

It is narrower:

A transient interruption of Xet storage traffic can reproduce this terminal failure class, and the painful part for very large files is what happens once request-level recovery is exhausted and the download crosses into a fatal reconstruction failure.

For this particular case, the existing Xet log around the failure seems like the highest-information next artifact. If it contains the same retry-exhaustion sequence as the small control, that would connect the two observations much more strongly without requiring another 84.8 GB experiment.

[twnskai05]

Thanks, John. Appreciate it. For the moment, I’ve found a combination of “hf download …” to get the .cache json file and a manual jq/wget script to read the json and get the files to work ok so far. Until it’s fixed, thanks.