DeepSeek V4.1-Flash 763B Runs Locally on 12GB GPU with Quantization

Now DeepSeek-V4.1-Flash with 763 billion parameters can be run locally even on a GPU with 12 GB of video memory.

Researchers from the Austrian institute IST Austria presented a quantized version of the model using the GSQ‑RCO method, and the developers of the inference engine Strata provided the ability to run it on consumer GPUs.

According to the test results:

> RTX 5070 with 12 GB of video memory and 64 GB of RAM provides about 5 tokens per second when generating a response and up to 231 tokens per second when processing the input context.
> RTX 3060 with 12 GB of video memory and 46 GB of RAM reaches 8–10 tokens per second when generating a response and about 149 tokens per second when processing the input context.

However, running it will require a significant amount of RAM and at least 239 GB of free storage space, because most of the model weights are stored in RAM and on an SSD, from where they are loaded as needed.

Of course, such speed is hard to call high, and performing complex agent tasks still requires substantial computational resources.

Nevertheless, the very possibility of running a model with 763 billion parameters on a regular computer indicates significant progress in quantization and inference optimization.