There's a narrative in AI that goes something like this: progress equals scale. Bigger models, more data, more compute. And for the past several years, this narrative has been remarkably productive. But we're approaching a wall — not of ideas, but of physics and economics.
Training frontier models now costs hundreds of millions of dollars. Inference at scale consumes entire power plants' worth of electricity. The carbon footprint of AI is becoming a serious concern. And yet, the models keep getting bigger.
Something has to give. And I believe the answer is hardware-software co-design.
What Co-Design Actually Means
Co-design isn't just "optimize your code for the GPU." It's a fundamentally different way of thinking about the AI stack. It means designing algorithms, software frameworks, and hardware architectures together, so that each layer is aware of and optimized for the others.
At IBM Research, this is what my team does every day. We don't just build accelerators and hope software catches up. We don't just write algorithms and hope hardware can run them efficiently. We design the full stack as an integrated system.
Three Levels of Co-Design
Algorithm-Hardware Co-Design: When we develop new neural architectures, we simultaneously evaluate them on target hardware. A model that's theoretically efficient but can't exploit the memory hierarchy of actual accelerators isn't really efficient. Our work on neural architecture search for analog in-memory computing is a prime example — we search for architectures that are specifically optimized for the physics of analog devices.
Framework-System Co-Design: The software stack between the model and the hardware matters enormously. Techniques like FlashAttention, PagedAttention, and speculative decoding aren't just clever algorithms — they're co-designed with specific hardware capabilities in mind (memory bandwidth, cache sizes, parallel execution units).
Chip-Workload Co-Design: The most radical form of co-design is building hardware specifically for AI workloads. IBM's NorthPole chip, published in Science, takes inspiration from neuroscience to create a digital architecture that achieves remarkable energy efficiency for inference. It's not a general-purpose processor optimized for AI — it's an AI processor, period.
Why This Matters for the Industry
The companies that will win the next phase of AI aren't necessarily the ones with the biggest models. They're the ones that can deploy AI most efficiently — at the lowest cost, lowest latency, and lowest energy consumption. That's a co-design problem.
This is why I teach both High-Performance Machine Learning and Scaling LLMs at Columbia. Students need to understand the full stack. An AI engineer who can only write PyTorch is like a civil engineer who can only draw blueprints but doesn't understand materials science. You need both.
The Path Forward
The next AI revolution won't come from a single breakthrough model. It will come from a thousand optimizations across the stack — better quantization, smarter memory management, more efficient attention mechanisms, novel hardware paradigms, and the integration of all of these into coherent systems.
This is hard, unglamorous work. It doesn't make headlines the way a new chatbot does. But it's the work that will determine whether AI becomes a sustainable, accessible technology or remains an expensive luxury. And it starts at the silicon level.
Dr. Kaoutar El Maghraoui
Principal Research Scientist at IBM Research · Adjunct Professor at Columbia University