A Systems View of Efficient Machine Learning, from Silicon to Agents (Part 0: Introduction | Part 1: Rooflines)
Most of what is written about machine learning assumes the model is the interesting part and the machine is a detail. In practice the machine decides what you are allowed to build. This book is about the boundary where a model meets real hardware under a real budget for latency, memory, power and money: how to predict what that boundary will do to you, and what to change first when it does.
There is an enormous amount written about making models bigger. There is much less written about the other direction, which is where almost all of the deployed value in machine learning actually lives: taking a model that works and making it run inside a budget that somebody real has to pay for.
That budget has a shape. A robot has 33 milliseconds to see and react. A phone has a few watts and a thermal ceiling it will hit in ninety seconds. A satellite imagery pipeline has a cost per scene that decides whether the product has a business. A language model on a laptop has a memory bandwidth number that caps its token rate no matter how good your kernels are. In every one of those cases the constraint comes from the machine, and you cannot negotiate with it. You can only understand it early or discover it late.
I have spent my career on the unglamorous side of that boundary. I optimise machine learning infrastructure for Arm architectures: compilers, libraries, and the kind of kernel-level analysis that turns “the model is slow” into a specific instruction stream stalling on a specific cache. Before that I spent two years at Dyson putting convolutional networks onto robot hardware, where quantisation, pruning and distillation were not research topics but the difference between shipping and not shipping. Before that I built automated pipelines that processed high resolution satellite imagery in near real time. In between, I led the design of LLM agent systems, where the currency changes from FLOPs to tokens but the discipline is identical.
The thing that surprised me, moving across those domains, is how little the reasoning changes. The satellite pipeline, the robot vacuum, the on-device language model and the agent all fail in the same few ways, and they all yield to the same few questions:
Almost nobody asks those questions in that order. The default behaviour, which I have also been guilty of, is to reach for the most interesting optimisation rather than the one the measurement demands. You can spend a fortnight writing a beautiful fused kernel for an operator that accounts for four percent of your runtime, and the profile will look exactly the same when you are done.
Takeaway: efficiency work is not a bag of tricks. It is a measurement discipline with a bag of tricks attached. This book tries to teach the discipline first and keep the tricks in their proper place.
By the end of this book you should be able to pick up an unfamiliar model and an unfamiliar device and, in about fifteen minutes with a datasheet and a calculator, produce a defensible estimate of:
You will also, I hope, develop a healthy suspicion of FLOP counts. A large fraction of the disappointment in this field comes from people reducing FLOPs by 4x, measuring no speedup at all, and concluding that performance engineering is witchcraft. It is not witchcraft. It is arithmetic intensity, and Part 1 explains it.
The book runs from the silicon upward. Each part stands alone reasonably well, but the arguments compound.
Foundations. Part 1 builds the roofline model and extends it with the two constraints that datacentre treatments usually leave out, energy per operation and sustained versus peak clocks. It is the analytical backbone for everything else. Part 2 opens up the hardware itself: what CPU vector units, integrated GPUs and NPUs actually are, what the memory hierarchy costs you, and why the first benchmark run is always the fastest one you will ever see.
The stack between a model and a machine. Part 3 goes inside a single operator, from the naive triple loop to tiling, packing, vectorisation and fusion, and explains the single most useful counterintuitive fact in edge inference: why depthwise separable convolutions cut FLOPs enormously and latency far less. Part 4 covers the layer above, graph capture, intermediate representations, the optimisation passes that matter, memory planning, and the partitioning and fallback behaviour that decides whether your accelerator is used at all.
Making the model smaller. Part 5 is the most immediately useful chapter for most readers: the arithmetic of affine quantisation, calibration, post training versus quantisation aware training, what breaks in transformers, and how to run a sensitivity analysis instead of guessing. Part 6 covers pruning, structured and unstructured sparsity, distillation and architecture search, and argues for a specific order in which to apply the levers.
Applications. Part 7 is about vision systems in production, where preprocessing and non maximum suppression routinely cost as much as the network, and where resolution is the most powerful and most abused knob available. Part 8 applies the framework to language models on small machines, deriving why decode is bandwidth bound and what that implies for quantisation and speculative decoding. Part 9 is about perception inside a control loop, where predictable latency beats low average latency, and about where vision language models genuinely pay for themselves today.
Operations. Part 10 is the methodology chapter: how to measure honestly, which hardware counters answer which question, and a decision tree that turns a profile into a next action. Part 11 covers serving economics, batching, the utilisation and tail latency trade, drift, and safe rollout. Part 12 treats an agent as what it is, a distributed system whose latency is dominated by the depth of its call chain, and applies the same budget discipline with tokens in place of FLOPs.
Part 13 collects the through line and points at what to read next.
Read Part 1 first, properly, with a pen. Everything after it assumes you can place a workload on a roofline and say which side of the ridge point it sits on. After that, the applied chapters can be read in any order according to what you are stuck on.
Every chapter ends with a few problems to work. They are calculation problems rather than trivia, and the answers are written out in full behind a collapsible block. I would encourage you to actually attempt them. The skill this book is trying to transfer is a numerical instinct, and you do not get a numerical instinct by reading about one.
Expected background: you should be comfortable with neural networks at the level of knowing what a convolution and an attention head do, and comfortable enough with arithmetic to be unbothered by a page with numbers on it. You do not need to know any assembly, any compiler theory, or any particular framework. Where I use PyTorch or C, the code is short and commented.
A note on numbers. Where I use hardware figures, they are either publicly documented or clearly labelled as illustrative assumptions for a worked example. I have deliberately avoided quoting internal measurements from anywhere I have worked. The method matters more than the specific chip anyway, and the specific chip will be obsolete before the method is.
Takeaway: the goal is not to memorise the current generation of hardware. It is to be able to reason about the next one from its datasheet.
I am Usamah Zaheer. I am a Machine Learning Software Engineer at Arm in Cambridge, where I work on ML compilers, libraries and inference performance for Arm architectures. I was previously a Robotics Software Engineer at Dyson, building and deploying vision models on robot hardware, and before that a machine learning research assistant at the University of Leicester working on satellite imagery pipelines. I am currently reading for an MS in Artificial Intelligence at the University of Texas at Austin, on top of an MS in Embedded Systems and Control Engineering from Leicester and a BTech in Electronics and Communication Engineering.
I write more regularly at usamah.me, and my code is on GitHub. If you find an error in this book, and there will be errors, I would genuinely like to hear about it: usamahzaheer155 [at] gmail [dot] com.