# FP8 Storage FP8 Storage cuts a model’s VRAM footprint roughly in half by keeping weights on the GPU in 8-bit floating-point format (`float8_e4m3fn`). During inference, each layer’s weights are cast on-the-fly back up to the compute precision (FP16/BF16), then cast back to FP8 after the forward pass — so quality is largely preserved. It pairs well with [Low-VRAM mode](/content/configuration/low-vram-mode/index.html): low-VRAM mode streams layers between RAM and VRAM, while FP8 Storage shrinks the layers themselves. ## Requirements - **Nvidia GPU on Windows or Linux.** FP8 Storage uses CUDA tensor types and is silently disabled on CPU and MPS. - **CUDA 12.x and recent PyTorch.** The `float8_e4m3fn` dtype was added in PyTorch 2.1 — InvokeAI’s bundled versions satisfy this. There is no hardware requirement for FP8 _compute_ — InvokeAI casts back to FP16/BF16 for math. This means FP8 Storage works on GPUs that do not natively support FP8 matmul (e.g. RTX 30-series), at a small per-step throughput cost. ## Hardware support tiers InvokeAI’s FP8 path stores weights in FP8 and casts them back to BF16/FP16 on each forward pass via its own `register_forward_pre_hook` / `register_forward_hook` wrappers. The practical benefit of toggling FP8 Storage depends on what your GPU can do natively. There are three tiers: ### RTX 30-series and older Ampere workstation cards — VRAM win only The toggle works as advertised: the UNet / transformer drops by roughly 50% on the GPU. Per-step latency is the same or marginally slower because every forward pass adds an FP8 → BF16 cast on entry and a BF16 → FP8 cast on exit. This is the **largest target group**: 3090 owners squeezing FLUX into 24 GB benefit the most. ### RTX 40-series, RTX 50-series, and Hopper — VRAM win today, compute win possible later These GPUs have native FP8 tensor cores. The toggle still buys you the same ~50% VRAM reduction today, because the forward pass still runs in BF16. If InvokeAI later wires up a true FP8 matmul path, the same toggle will _also_ unlock compute speedups on this hardware. Until then, treat the benefit as “VRAM only, same as Ampere”. ### Older CUDA cards — still a VRAM win `float8_e4m3fn` is a pure storage dtype in PyTorch and works on any CUDA device, so pre-Ampere cards (GTX 16-series, RTX 20-series, etc.) get the same ~50% VRAM reduction as Ampere. There are no native FP8 tensor cores on these GPUs, so the throughput trade-off is the same as on the 30-series: cast in, compute in BF16/FP16, cast back out. ### MPS and CPU — no-op FP8 Storage is silently disabled on anything that is not CUDA. On CPU PyTorch _technically_ supports FP8 dtypes, but the cast operations are software-emulated and end up costing more than the memory savings buy back, so InvokeAI gates the entire path on `device.type == "cuda"`. If you toggle it on CPU or MPS, the loader skips the cast and returns the model unchanged with no log line. ## Enabling FP8 Storage FP8 Storage is a **per-model setting**, configured from the Model Manager: 1. Open the **Model Manager**. 2. Select a model (Main, ControlNet, or T2I-Adapter). 3. Under **Default Settings**, toggle **FP8 Storage (Save VRAM)**. 4. Click **Save**. The setting takes effect on the next load. If the model is already in the cache, InvokeAI evicts the cached copy automatically. ## What FP8 Storage applies to FP8 Storage is **only** applied to layers where the precision trade-off is acceptable: | Model type | FP8 applied? | |----------------------------------------|---------------| | Main models (SD1, SD2, SDXL) | Yes | | FLUX.1 / FLUX.2 Klein | Yes | | ControlNet, T2I-Adapter | Yes | | VAE | No | | Text encoders, tokenizers | No | | Z-Image (any variant) | No | | LoRA, ControlLoRA | No | Within a supported model, **norm layers, position/patch embeddings, and `proj_in`/`proj_out` are skipped** so precision-sensitive tiny learned scalars aren’t crushed to FP8. ## Quality trade-offs FP8 Storage is **near-lossless** for most workloads because: - Norms and embeddings (the precision-sensitive layers) are skipped. - The actual matmul still happens in FP16/BF16 — FP8 is only the on-GPU storage format. That said, some artifacts have been reported on: - **VAEs** — never cast (the toggle has no effect on VAE submodels). - **Heavy LoRA stacks** — very precision-sensitive LoRAs may show slight drift. If you see unexpected quality regressions, disable FP8 Storage on the affected model and re-run. ## Combining with Low-VRAM mode **FP8 + partial loading**: fully supported. FP8 Storage shrinks the layers; partial loading streams them between RAM and VRAM as needed. Use both on tight VRAM budgets. ## Troubleshooting ### ”I toggled FP8 Storage but VRAM usage didn’t change” The cache eviction is immediate for idle models, but **deferred until the next unlock** if the model is mid-generation. Wait for the current generation to finish, then start a new one. If VRAM still hasn’t dropped: - Check the InvokeAI log for `FP8 layerwise casting enabled for `. If the line isn’t there, the model is on the exclusion list. - Confirm you are on CUDA. FP8 Storage is silently disabled on CPU and MPS. ### Quality regression on a specific model Disable FP8 Storage for that model in Model Manager and reload. If quality is restored, the model has FP8-sensitive layers that fall outside the default skip list. ### ”RuntimeError: … float8_e4m3fn …” You’re on a PyTorch version that predates FP8 support. Reinstall InvokeAI using the official launcher. ### Reporting an FP8 issue If FP8 Storage misbehaves, please open a GitHub issue and include: - **What you did**: the workflow / generation step that triggered the problem. - **Model**: exact name and variant. - **LoRAs**: whether any LoRAs (or ControlLoRAs) are stacked on the model. - **Other toggles**: any configured VRAM limit. - **GPU**: model and VRAM size. - **OS**: Windows or Linux, plus driver / CUDA version if you have it. - **Logs**: the InvokeAI log around the failure.