Recently I’ve been writing my MSc thesis at the EPCC on the energy cost of memory stalls during LLM inference. It was a self-proposed thesis, and I did a bunch of experiments and research to try and find a real literature gap. Although my time at the EPCC has focused on HPC and programming performance and efficiency at scale, I have enjoyed modern MlSys research. I wanted to help contribute to that field and begin research in this direction. So, the premise is fairly straightforward: when a GPU runs out of local HBM, and has to fetch the KV cache over PCIe, the SMs sit idle for microseconds at a time, fully energised, drawing power for no useful work. I wanted to measure that wasted energy directly. That’s not what this post is about, though.

I have spent a significant amount of time in the previous semester doing this thesis, producing an 18-page “feasibility report” as an interim submission. As far as I know, I received the highest grade in my class. My MSc dissertation has always been important to me, even in concept. In ways, it is my first serious research attempt. In the past 4 months, I have written code, conducted experiments, failed, re-evaluated, and kept on pivoting until I reached this thesis topic. Initially, I planned to implement a similar system as introduced by the ML.Energy project, Zeus and Perseus, but for LLM inference. Interconnect stalls, DVFS, Green HPC, etc. Where I am now, I keep on finding flaws in my methodology, my experimentation process. Doing any research alone is tough, but it has also been rewarding. At times I imagined myself a few years from now, presenting a mature version of this thesis to SK Hynix, and it helped me get through, but I am still nowhere near the “finish line”.

I have been wondering recently about the timeline of research. I’ve never been exposed to research, I went to a fairly small undergrad, and have not had the chance to work with any professors during this MSc due to it being extremely coursework focused (I’ve taken 6 MSc courses each semester!). However, my thesis has fairly hard deadlines due to hardware access. Dealing with office politics for this hardware access was … tricky, but it has forced me into this state of neverending urgency to put all my energy into producing results by certain dates (May 27th, I am unsurprisingly procrastinating right now writing this post). This, unsurprisingly, leads to burnout. Hardware access is tricky though, it is infeasible to request more time on such expensive hardware, and so I must either force myself to complete my code and experiments by a certain date, or I must forfeit my research. I have a few personal reasons for wanting to be successful, and I promised myself that I would have a truly impressive thesis. This leads me to Goodhart’s Law:

“When a measure becomes a target, it ceases to be a good measure.”

I think there is something profound in a thesis. Or research, in general. I’m not sure what that is, because there is no one definition of research. There is the western perspective, the eastern perspective, shaped on thousands of years of history, from the individualism of Socrates and Ibn-Al-Haytham to the systematic approach of Xunzi. There are so many philosophical musings to be made, but the system is too profoundly broken for this to be a philosophical problem rather than a political one. A reader who has faced a similar dilemma does not need further explanation. And I’m not sure if this is the right platform for further explanation. In essence, the thesis has certainly become a target, where it was supposed to be a measure, of mastery, of current understanding in the field, and an investigation of what happens next. My goal still remains to publish my work, unfortunately so. However, I hope this is a stepping stone to a more stable point in my life, where I need not consider every aspect of work in general as a target to be achieved.