Low-VRAM mode | InvokeAI Documentation

Low-VRAM mode

As of v5.6.0, Invoke has a low-VRAM mode. It works on systems with dedicated GPUs (Nvidia GPUs on Windows/Linux and AMD GPUs on Linux).

This allows you to generate even if your GPU doesn’t have enough VRAM to hold full models. Most users should be able to run even the beefiest models - like the ~24GB unquantised FLUX dev model.

Enabling Low-VRAM mode

Low-VRAM mode is enabled by default via the enable_partial_loading: true setting in invokeai.yaml. No action is required to turn it on.

Windows users should also disable the Nvidia sysmem fallback.

It is possible to fine-tune the settings for best performance or if you still get out-of-memory errors (OOMs).

If you want to disable partial loading (e.g. on systems with plenty of VRAM where full loading is faster), add this line to your invokeai.yaml and restart Invoke:

enable_partial_loading: false

Details and fine-tuning

Low-VRAM mode involves 4 features, each of which can be configured or fine-tuned:

Read on to learn about these features and understand how to fine-tune them for your system and use-cases.

Partial model loading

Invoke’s partial model loading works by streaming model “layers” between RAM and VRAM as they are needed.

When an operation needs layers that are not in VRAM, but there isn’t enough room to load them, inactive layers are offloaded to RAM to make room.

Enabling partial model loading

Partial model loading is enabled by default. The corresponding setting in invokeai.yaml is:

enable_partial_loading: true

Set it to false to disable partial loading.

PyTorch CUDA allocator config

The PyTorch CUDA allocator’s behavior can be configured using the pytorch_cuda_alloc_conf config. Tuning the allocator configuration can help to reduce the peak reserved VRAM. The optimal configuration is dependent on many factors (e.g. device type, VRAM, CUDA driver version, etc.), but switching from PyTorch’s native allocator to using CUDA’s built-in allocator works well on many systems. To try this, add the following line to your invokeai.yaml file:

pytorch_cuda_alloc_conf: "backend:cudaMallocAsync"

A more complete explanation of the available configuration options is here.

Dynamic RAM and VRAM cache sizes

Loading models from disk is slow and can be a major bottleneck for performance. Invoke uses two model caches - RAM and VRAM - to reduce loading from disk to a minimum.

By default, Invoke manages these caches’ sizes dynamically for best performance.

Fine-tuning cache sizes

Prior to v5.6.0, the cache sizes were static, and for best performance, many users needed to manually fine-tune the ram and vram settings in invokeai.yaml.

As of v5.6.0, the caches are dynamically sized. The ram and vram settings are no longer used, and new settings are added to configure the cache.

Most users will not need to fine-tune the cache sizes.

But, if your GPU has enough VRAM to hold models fully, you might get a perf boost by manually setting the cache sizes in invokeai.yaml:

# The default max cache RAM size is logged on InvokeAI startup. It is determined based on your system RAM / VRAM.

# You can override the default value by setting `max_cache_ram_gb`.

# Increasing `max_cache_ram_gb` will increase the amount of RAM used to cache inactive models, resulting in faster model

# reloads for the cached models.

# As an example, if your system has 32GB of RAM and no other heavy processes, setting the `max_cache_ram_gb` to 28GB

# might be a good value to achieve aggressive model caching.
max_cache_ram_gb: 28

# The default max cache VRAM size is adjusted dynamically based on the amount of available VRAM (taking into

# consideration the VRAM used by other processes).

# You can override the default value by setting `max_cache_vram_gb`.

# CAUTION: Most users should not manually set this value. See warning below.
max_cache_vram_gb: 16

Working memory

Invoke cannot use all of your VRAM for model caching and loading. It requires some VRAM to use as working memory for various operations.

Invoke reserves 3GB VRAM as working memory by default, which is enough for most use-cases. However, it is possible to fine-tune this setting if you still get OOMs.

Fine-tuning working memory

You can increase the working memory size in invokeai.yaml to prevent OOMs:

# The default is 3GB - bump it up to 4GB to prevent OOMs.
device_working_mem_gb: 4

Keeping a RAM weight copy

Invoke has the option of keeping a RAM copy of all model weights, even when they are loaded onto the GPU. This optimization is on by default, and enables faster model switching and LoRA patching. Disabling this feature will reduce the average RAM load while running Invoke (peak RAM likely won’t change), at the cost of slower model switching and LoRA patching. If you have limited RAM, you can disable this optimization:

# Set to false to reduce the average RAM usage at the cost of slower model switching and LoRA patching.
keep_ram_copy_of_weights: false

Disabling Nvidia sysmem fallback (Windows only)

On Windows, Nvidia GPUs are able to use system RAM when their VRAM fills up via sysmem fallback. While it sounds like a good idea on the surface, in practice, it causes massive slowdowns during generation.

It is strongly suggested to disable this feature:

Troubleshooting

Windows page file

Invoke has high virtual memory (a.k.a. ‘committed memory’) requirements. This can cause issues on Windows if the page file size limits are hit. If you run out of page file space, InvokeAI may crash. Often, these crashes will happen with one of the following errors:

If you are running out of page file space, try the following solutions: