As a participant representing TeamEPCC in the ISC Student Cluster Competition 2026, I’ve recently had the chance to work on a custom-built HPC cluster, to be shipped to Hamburg for the in-person competition. Until then, I have to keep cluster specs and optimization efforts secret, but I wanted to give my thoughts on a specific topic: can you optimize an application without fully understanding it?

The competition itself has never included Llama.cpp as a micro-benchmark, usually focusing on HPC-specific benchmarks like HPL and HPCG. This year introduced Llama.cpp, which allows LLM inference with minimal setup on a wide range of hardware. It is a plain C/C++ implementation with no dependencies. It runs surprisingly well on Apple silicon as well! Understanding Llama.cpp is fairly simple. Maximizing performance is whole other thesis. While the source code is very well written, and it is easy to find and read specific implementations of custom kernels, the majority of performance comes from tuning the runtime parameters, understanding the workload and how these parameters alter the characteristics of your problem on your given hardware. While I have to stay a bit vague, I believe this problem is somewhat unique from the other application benchmarks, like HPL.

HPL involves around 4 parameters, each can be tuned to achieve maximum performance on your hardware. Interestingly, the competition allows us to choose a matrix size for achieving this performance. Llama.cpp will force you to understand a different question. How does your chosen model interact with Llama.cpp’s implementation of Flash Attention? How does your batch size, parallel sequence count, impact your workload? How do different model types (MoE, Dense) interact with these parameters? Unfortunately, its not easy to predict.

With Llama.cpp, its as if the optimization is a moving target. Prefill is compute-bound, decode is memory-bandwidth-bound. The KV cache grows with context length and eats HBM that your weights also want. A flag that doubles prefill throughput can halve decode throughput on the same run. The “right” answer is hardly obvious, but the point of this post is that your success likely depends on your knowledge of AI in general. Specifically for the SCC, I find this interesting because I have found that your hardware would not necessarily determine your success on competition day. I’ve noticed this through nearly doubling throughput from understanding my workload better (secret!). While I don’t have too much prior experience in AI, I’ve spent the past month really diving deep into inference, understanding exactly what matters in production. I had a coursework related to scaling up a ViT model, improving efficiency with PyTorch, and we often treated optimizations provided by PyTorch, like Flash Attention, as guaranteed wins. Surprisingly, I’ve found cases where FA quarters throughput. I refrained from experimenting with parameters like batch size because I believed “efficiency” to be purely algorithmic, but I now realize I just did not understand the nature of ML serving back then.

To summarize my thoughts, many people refrain from claiming AI is a part of HPC. Throughout my MSc in HPC, we rarely talked about the theory of ML, leaning more into hardware and custom accelerators. In hindsight, I think this is justified. While my teammates in the SCC work on fixing endless bugs in OpenMPI and Mellanox RDMA, I am learning ML algorithms and theory. This isn’t a complaint, just an observation! I’m actually hoping to continue my studies in Efficient AI, with my dissertation heavily leaning into MLSys. And so, I think AI is an interesting field where it is hard to apply HPC optimization strategy without understanding workloads. Parameter sweeps cannot get you through the door over here.

Note: Throughout my experimentation, I’ve been ever-curious about how Llama.cpp would perform with a Grace-Hopper Superchip on these workloads. I would love the opportunity to talk about real numbers, if anyone bringing a GH200 to the SCC is reading this!