Skip to content

Free Server + Free Model: How Fast Can It Run?

中文版

Ternary vs 4-bit Quantised LLMs on ARM CPUs: A Practical Performance Study

About the title

Tested in November 2026. Models, inference engines and free cloud offers change every few months, so these numbers are a snapshot of that time. My server was set up some time ago, when the free tier allowed 4 cores and 24 GB of memory. Today's free tier is half that, but you can still sign up and use it.

The title is a bit of clickbait, but the content is serious. The "free server" is an ARM server from Oracle Cloud's Always Free tier. The "free model" is an open model that you download and run yourself, so you do not pay per token for a frontier model's API.

In this post I measure how fast ternary-weight models and 4-bit quantised models run on an ordinary CPU server, a free-tier ARM server in my case. If you are a student with a free server or an ordinary laptop, you can repeat the test yourself with the commands below.

1. Can a free ARM server run a large language model?

Many people have an Oracle Cloud Always Free account. The most popular part of it is the ARM server, the VM.Standard.A1.Flex shape with Ampere processors. In the past the free tier allowed up to 4 cores and 24 GB of memory. Oracle's documentation now gives 1,500 OCPU hours and 9,000 GB hours per month, which is 2 cores and 12 GB of memory1. There is no GPU. People use these servers for websites, forums and small experiments.

So the obvious question is: can a server with no GPU run a large language model? I once tried Google's Gemma on a similar server, and even simple questions were slow. I no longer remember which model size or quantisation I used, so that is only an impression.

Things have changed a lot in the last two years. Two trends are moving in opposite directions. Training the best frontier models keeps getting more expensive, and only a few large companies can afford it. Running a model that is "good enough" keeps getting cheaper: for the same capability, inference cost fell by more than a hundred times in two years2. Small models, distillation, quantisation and better inference engines have together brought large language models from the data centre to ordinary computers. This post tests how far that second trend has come.

One interesting path in this direction is ternary weights. Each weight can only be -1, 0 or +1. Many people expect ternary models to be more efficient.

Ternary models are a busy area right now, in both software and hardware. The Bonsai models I tested come from PrismML. They have since released a new Bonsai 2 family; I tested an earlier version. Some of my friends also work in this area. T3ISC builds ternary chips, where the hardware itself computes with three states. ternii puts tiny ternary models on your phone, so they run fully offline and your data stays on the device.

So I decided to test it myself: on an ARM cloud server with no GPU, how fast do ternary models and common 4-bit models actually run? Ternary model files are smaller. Does that also make them faster?

The short answer

On this 4-core ARM server, I compared a common 4-bit model (Qwen3 in Q4_K_M format) with the ternary model Bonsai in two formats. Both are small models of about 1.7 billion parameters.

  • Qwen3, 4-bit: about 30.5 tokens per second. The fastest of the three.
  • Bonsai, generic ternary format Q2_0_g64: about 14.9 tokens per second.
  • Bonsai, PrismML's own format PQ2_0: about 23.2 tokens per second. More than 50% faster than the generic format, but still slower than 4-bit.

The ternary files are about a third of the size of the 4-bit file, but they are not faster. Fewer bits save storage, but speed also depends on how well the compute code is written.

I also tried Microsoft's BitNet. It crashed on this ARM server, so I have no numbers for it. More on that below.

2. Results

First, speed with four threads. The 4-bit Qwen3 generates text fastest. Bonsai in PrismML's PQ2_0 format is about 55% faster than in the generic Q2_0_g64 format, but it is still slower than Q4.

Bar chart of speed with 4 threads Bar chart of speed with 4 threads

Next, the number of threads. All three get steadily faster with more threads. If you have today's free tier (2 cores), look at the middle column: Q4 generates about 17 tokens per second. That is worth trying for light text tasks for one user.

Line chart of generation speed by thread count Line chart of generation speed by thread count

The last chart shows the main point of this post. The ternary files are much smaller, but they are not faster.

Scatter chart of file size against generation speed Scatter chart of file size against generation speed

Full results (tokens per second):

Model and format File size Generation
1 / 2 / 4 threads		 Prompt processing
4 threads
Qwen3 · Q4_K_M 1.19 GiB 9.3 / 17.5 / 30.5 94.1
Bonsai · Q2_0_g64 462 MiB 4.1 / 8.0 / 14.9 20.6
Bonsai · PQ2_0 436 MiB 6.7 / 12.6 / 23.2 33.6

I ran Q2_0_g64 on both engines, the original llama.cpp and PrismML's version. The results were almost the same, so the table uses PrismML's numbers.

3. A few terms, briefly

4-bit quantisation (Q4). Model weights are normally stored as 16-bit floating-point numbers. Quantisation stores them with fewer bits, for example 4, so the file and the memory use get smaller. Q4_K_M is a widely used 4-bit format in llama.cpp.

Ternary weights. Each weight has only three possible values: -1, 0 or +1, multiplied by a scale factor. Three states need only log₂3 ≈ 1.58 bits in theory, so these are often called "1.58-bit models". In practice the weights are packed in 2-bit units with some extra data, so the files are not quite that small.

GGUF, llama.cpp and kernels. GGUF is a model file format. llama.cpp is the program that loads it and runs the model on your own machine. What really decides the speed is the low-level code that does the matrix maths, usually called the kernel. Two ternary models can run at very different speeds if their weights are packed differently or their kernels are written differently.

Bonsai's two formats. Q2_0_g64 is a generic format that the original llama.cpp can read. PQ2_0 is PrismML's own packing format, and it needs PrismML's modified llama.cpp. Both files come from the same model; only the packing and the compute code differ.

A side story: llama.cpp was written in an evening

llama.cpp was written by Georgi Gerganov. In 2022 OpenAI released Whisper, its speech recognition model. Gerganov rewrote it in plain C/C++ as whisper.cpp, with no deep learning framework, so it ran on an ordinary computer. Along the way he wrote a small tensor library called ggml.

In March 2023, soon after Meta released its LLaMA model, he used ggml to write llama.cpp. The first README (10 March 2023) was very direct. The goal was "to run the model using 4-bit quantization on a MacBook", and Apple silicon was a "first-class citizen". It also said, honestly:

This was hacked in an evening - I have no idea if it works correctly.

The ARM NEON code he wrote for Apple chips later helped every ARM processor, including the Oracle server in this post. Today llama.cpp supports GPUs from NVIDIA, AMD, Intel, Qualcomm Snapdragon and Moore Threads, and even IBM mainframes. GGUF is now one of the most common file formats for local models. Two of the three engines in this post are built on llama.cpp, and Microsoft's BitNet includes it as a submodule. A project that started in one evening now supports much of the local LLM world. That deserves credit.

4. How I tested

The machine. Oracle VM.Standard.A1.Flex, an Ampere Altra processor with ARM Neoverse-N1 cores, 4 cores, 24 GB of memory, Ubuntu 22.04 and no GPU. The processor supports ARM's NEON and dotprod vector instructions, but not newer ones such as SVE or i8mm.

The software. The original llama.cpp, PrismML's modified llama.cpp, and Microsoft's BitNet. These projects change quickly. These are the versions I used, so you can download exactly the same code:

Project Version (Git commit)
llama.cpp 9f0e5098a
PrismML's llama.cpp 6447906bf
Microsoft BitNet 0b341e5

A first hello. Before testing speed, I asked each model one question: "Explain what DNS does in one sentence." Qwen3 surprised me at first. I gave it a limit of 50 tokens, and it used all of them on "thinking" before it gave any answer. With a limit of 512 tokens it answered correctly. Bonsai gave a correct one-sentence answer straight away.

Then the speed test. I used llama-bench, the benchmark tool that comes with llama.cpp. I ran each model with 1, 2 and 4 threads, three times each, and took the average:

./build/bin/llama-bench -m <model file> -t 4 -p 128 -n 128 -r 3

It reports two numbers:

  • Prompt processing: how fast the model reads your input.
  • Token generation: how fast the model writes its answer. This is what you mostly feel in a chat.

5. Why isn't a smaller file faster?

You might expect a smaller file to be faster, because less data has to move from memory. On a CPU, speed also depends on other things:

  • Unpacking. Ternary weights are tightly packed. Before the CPU can compute with them, they have to be unpacked into a form it can use. This step takes time, and it can cancel out the memory bandwidth you saved.
  • The quality of the compute code. 4-bit formats have been optimised in llama.cpp for a long time. Ternary formats are newer. With PrismML's own format and code, Bonsai became 55% faster, so there is clearly a lot of room left.
  • CPU instructions. This processor has no new instructions designed for low-bit maths, so ternary models may not show their full advantage.

So it depends on what you need. For the smallest download and the least memory, ternary models already win. For the fastest single response on this machine, 4-bit is still ahead.

6. Why BitNet didn't run

Microsoft's BitNet b1.58 is one of the best-known ternary models. It compiled on my server, but with both 1 and 4 threads it crashed before producing its first word. The error was a failed numerical check (Assertion scale > 0.0f failed). This usually means an earlier step had already produced an infinite or invalid number.

I looked at the source code and found something suspicious. For the same ternary format, the code decodes weights in groups of 128 on x86 processors, but in groups of 64 on ARM. Several developers on GitHub have reported similar problems on ARM, and someone has submitted a fix. Most of them see garbled output, while mine crashed, so I cannot yet say it is the same cause.

So BitNet is not in the comparison. This does not mean BitNet itself is bad, only that its ARM code still has problems. I plan to dig into the code later, try to fix it, and write a separate post about it.

If you want to dig in yourself, start with these GitHub discussions (all still open as of 8 November 2026):

  • Issue #585: the ARM I2_S decoder uses groups of 64 weights, but the model file is packed in groups of 128, which gives garbled output. This is the clearest report, with an operator-by-operator comparison.
  • Issue #598: wrong I2_S output on AArch64, pointing to the group size, the unpacking and parts of the matrix multiplication code.
  • Issue #600: the same version I used (0b341e5) gives garbled output on ARM/NEON. Fixing one unpacking problem may not be enough.
  • PR #551: a community fix for ARM, not yet merged.

7. Things to keep in mind

This was a small test. Qwen3 and Bonsai are different models, and the engine versions are not exactly the same, so this is not a strict "4-bit versus ternary" experiment. I only measured speed. I did not measure answer quality, time to first token, peak memory or energy use, and I only used short inputs and outputs of 128 tokens. The numbers are good for a rough idea of what is usable. They are not enough to say that ternary models "don't work".

8. Final thoughts

Back to the question in the title: free server + free model, how fast can it run? On a 4-core ARM server, a small 4-bit model of about 1.7 billion parameters generates about 30 tokens per second. On today's 2-core free tier, it is about 17. That is already faster than most people read, so light text tasks are worth considering.

Ternary models did not win this time, but I am not pessimistic. A dedicated format made Bonsai more than 50% faster at once, so there is still a lot to gain in the code. With chips designed for ternary maths, the result could be very different.

Next, I want to fix BitNet and test it again. I also want to try a real task, such as checking spelling and grammar in an article, to see whether these small models are good enough. And I want to find out which part of the code makes ternary models slower.

If you have a free server or an ordinary computer, try the commands above and see how fast your machine is.

References

  1. Oracle Cloud Infrastructure documentation: Always Free Resources (checked 8 November 2026). ↩

  2. Stanford HAI, AI Index Report 2025: for a model at GPT-3.5 level (64.8% on MMLU), the inference price fell from about $20 per million tokens in November 2022 to about $0.07 in October 2024, roughly 280 times cheaper. ↩