From Class Projects to Research Innovation: Industrial Mentorship in Columbia's HPML Class

In Spring 2026, Dr. Dhaval Patel of IBM Research visited my High Performance Machine Learning class at Columbia. I bring guest researchers into the course most terms, usually to give a talk and answer questions, and to show students what the field looks like from inside a lab. Dhaval came with something different. He brought a working benchmark that his team runs at IBM, a set of open research problems he did not yet have answers to, and a clear expectation that a strong graduate student could make real progress on one of them in a single semester. Several teams took him up on it. By the end of the term, three of those projects had become papers, and a fourth, on how to evaluate agents like these, was written across the whole class. All four are now on arXiv as preprints.

What made this different from the usual final project was where the problems came from. The students were not handed a clean exercise with a known answer to reproduce. They worked on a system IBM actually runs, on questions that did not yet have solutions, and they had to produce results that were measured and that others could check. I will describe what the teams built, in enough technical detail to be useful to anyone teaching a similar course, and then what the experience changed in how I think about where the problems in a systems class should come from.

A course built on measurement

HPML is a systems course. The habit I work hardest to build is to measure first, change one thing, and measure again. I ask students to open every project by profiling a training or inference workload with the PyTorch Profiler and NVIDIA Nsight, so they can see where the time actually goes across data loading, preprocessing, host-to-device transfer, and compute. It is rarely where they guessed. Only then do they apply a technique they can defend against that profile: quantization, low-rank adaptation, batched serving with vLLM, dataloader changes, key-value cache reuse, asynchronous execution, or distributed training. Each project reports before-and-after numbers on GPU hardware, logged in Weights & Biases so the rest of us can check them.

By the time Dr. Patel arrived, the class had spent weeks doing exactly this, and that preparation turned out to matter. Three of the four projects here amount to a measured claim that one change made a system faster or cheaper without breaking what it produced, the same claim every HPML assignment asks a student to make. The students already knew how to make it carefully, and how to show their work.

How the teams earned their projects

The teams did not receive their topics by assignment. They competed for them.

Dr. Patel wrote several research proposals grounded in AssetOpsBench, IBM's open benchmark for AI agents that operate industrial equipment [5][6]. Teams read the proposals and pitched for the ones they wanted. A pitch had to show that the team understood the problem, had an approach that could plausibly work, and knew how it would measure whether the approach did. The strongest pitches won the projects.

This step matters more than it looks. A team that has argued for its own problem, against other teams, begins the semester already knowing why the problem is hard and what a convincing result would have to show. That kind of ownership is hard to hand out with an assignment, and easy to recognize once a team has it.

The shared benchmark

All four projects build on the same benchmark, AssetOpsBench.

A data-center chiller raises an alarm. Diagnosing it means pulling the sensor history, matching the symptoms to known failure modes, forecasting how the fault will progress, and, if needed, opening a work order for a technician. AssetOpsBench models this work as a set of coordinating agents, each responsible for one part: an IoT agent that retrieves sensor readings, a Time Series Foundation Model (TSFM) agent that forecasts from them, a failure-analysis agent (FMSR) that maps symptoms to failure modes, a work-order agent that writes tickets, and a vibration agent for spectral analysis. It ships with 141 expert-authored scenarios over a simulated, CouchDB-backed sensor environment, and it scores whether the agents reach the correct answer without fabricating one.

Dr. Patel introduces AssetOpsBench to the HPML class.

The proposals shared a common starting point. Each began by exposing the benchmark's tools through the Model Context Protocol (MCP), an open standard that defines a common way to describe and call tools, resources, and prompts, so that a model and a tool built separately can work together [7]. MCP standardizes that interface. It does not remove the integration work. Each tool still has to be wrapped, described, and tested against the agents that call it. With that in place, the teams could measure what the change cost and what it improved, and from there they worked on separate questions.

The four projects build on the same benchmark but investigate different questions. They are independent rather than sequential.

These evaluations measure how well an agent plans and answers. They are not certifications that an agent is safe to operate a physical asset on its own, and the students were careful about that distinction. The benchmark measures decision quality, not control of real equipment.

Question 1: do leaderboard rankings predict performance on new tasks?

When an agent finishes first on a benchmark leaderboard, the ranking is usually read as a prediction that it will also do better on the next task. The broadest of the projects asks whether that prediction holds when the next task comes from a different distribution than the benchmark.

This is the subject of the largest paper from the collaboration, a position paper titled Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents [1]. It is not one team's project. Led by Dr. Patel and co-authored across the class, it draws on fourteen parallel implementation studies of the MCP-based benchmark and seven earlier benchmarks. Its argument is that a single aggregate score combines many separate deployment concerns into one number, and that rankings based on that number do not transfer reliably when the data distribution changes.

Instead of ranking by in-sample average, the paper proposes ranking by predictive validity: the correlation between how a configuration ranks on the data you tested and how it ranks on data you did not. It specifies a twelve-tier measurement framework, three falsifiable out-of-distribution criteria with explicit thresholds, and a pre-registered pilot to test the claim.

A configuration that leads in testing can slip once the distribution changes. The only real-world evidence in the paper comes from an external agent competition. Public-to-hidden rank correlation there was about 0.69 on the planning track and about -0.13 on the execution track. Both sit at or below the transfer threshold the paper proposes.

The paper proposes this framework and argues for it. It does not yet prove it, and it says so. The predictive-validity claim has not been tested at scale by the authors, and empirical validation is left to future work. The available evidence comes from the fourteen studies and the one external competition, not from a controlled trial designed to test the claim directly.

Question 2: can we cut latency without serving stale results?

Alimurtaza Merchant, Krish Veera, Sajal Kumar Goyla, and Shambhawi Bhure worked on latency, and they found a problem that ordinary caching does not handle.

The usual way to speed up a slow pipeline is to cache results and reuse them. Caching by text similarity is unsafe here. Two queries can read almost identically and still require different answers, because the answer depends on time, on which asset, and on which sensor parameters. "Chiller 12 in the last hour" and "chiller 12 last Tuesday" look similar to an embedding model but are not interchangeable. A semantic cache that ignores this will return a confident but stale answer [2].

A temporal semantic cache classifies each query by how its answer depends on time, into volatile, static, relative, and anchored categories, and reuses a stored result only when it is still valid for the new request. Volatile, live-state queries skip the cache entirely. Separately, a set of MCP workflow optimizations, namely a disk-backed tool-discovery cache and dependency-aware parallel execution of tool calls, speeds up the pipeline without changing its output.

The two speedups measure different things. The workflow layer returns identical results about 1.67 times faster. The temporal cache reached a median 30.6 times speedup on the subset of queries that hit.

The numbers should be kept separate. The workflow optimizations alone gave a 1.67 times end-to-end speedup and roughly 40 percent lower median latency, with identical results. The 30.6 times figure from the temporal cache is a median over cache hits, not over all traffic. Across the combined pipeline, the end-to-end median speedup was about 3.48 times. The team is clear about the limit. Semantic similarity alone is not a reliable proxy for validity. Their hit-decision F1 reached about 0.64, and cross-parameter false positives remain the hard case.

Question 3: can we generate realistic scenarios at scale?

Sagar Chethan Kumar and Rohith Kanathur worked on how to expand the benchmark. AssetOpsBench depends on its scenarios, and its 141 scenarios were written by hand by subject-matter experts (SMEs). That manual authoring is the bottleneck. It does not scale, and it limits how quickly new asset classes can be added.

They built a three-stage generator. It profiles a target asset and its tools, decides how many scenarios to allocate across cases for coverage, then generates candidates through a validate-and-repair loop. A candidate that fails a check (schema validity, tool-domain consistency, grounding in the right standard, deduplication, or budget compliance) is repaired and re-checked rather than discarded. To test it, they added a new asset class, the Smart Grid Transformer, with four diagnostic tools: three grounded in international standards (interpret_dga in IEC 60599, and two assessments in IEC 60076-7) and one learned random-forest health index.

Warm-cache generation reached roughly 8 times the throughput of the naive baseline at 50 scenarios. The composite quality score was 74.2 plus or minus 1.9 against the baseline's 73.8 plus or minus 3.0.

The speedup did not come at a measurable cost to quality. With caching, parallel generation, and batched calls, generating 50 scenarios ran about 8 times faster than the naive baseline. On the composite quality metric the generated scenarios scored 74.2 plus or minus 1.9, against 73.8 plus or minus 3.0 for the hand-tuned baseline. Those intervals overlap, so any difference in quality is smaller than the metric can resolve. That is not the same as showing the two are equivalent, and the authors claim only that quality was preserved. They are also careful that the generator reduces the expert authoring burden rather than removing it. They still recommend expert review before generated scenarios are trusted, and the study uses a single model backend, so how well it generalizes across models is untested.

Question 4: can a small model internalize a fixed tool catalog?

Yuval Shemla, Ayal Yakobe, and Tanmay Agarwal asked whether an agent needs its full tool schema in every prompt, or whether a small model can learn its tools instead.

In the standard setup, each request includes the full catalog of tool descriptions, roughly 2,400 tokens, on top of the query itself. Their idea was that for a fixed, known set of tools, the catalog could be stored in the model's weights instead of the prompt. Using 8-bit QLoRA (quantized low-rank adaptation, which fine-tunes a small adapter on top of a quantized base model), they trained Gemma 4 E4B and Qwen3-4B on roughly 1,700 examples so the model learned the tool knowledge directly [4].

Description-free inference cut the tool-description tokens from about 2,400 to about 128. The best fine-tuned Gemma configuration planned better than an informed baseline (agent-trajectory F1 of 0.65 against 0.47).

The main result is the reduction in prompt size. Moving the catalog into the weights drops the tool descriptions from about 2,400 tokens to about 128. Because each request still carries the query itself, the paper reports the overall input-length reduction as 82.6 percent. The best fine-tuned Gemma configuration planned better than the informed baseline that still receives full tool descriptions, with an agent-trajectory F1 (AT-F1, how well the planned sequence of tool calls matches a reference) of 0.65 and a judge score of 3.88, against 0.47 and 2.88. In a model-level comparison, Qwen3-4B ran about 2.5 times faster on 62 percent less memory than Gemma.

The efficiency numbers are narrower than they look. They are measured at the model level on a single A100, not end-to-end on the full agent, where the team ran only a small preliminary test. The gains also trade against general knowledge. Raising the adapter rank improved planning but eroded pretraining question-answering accuracy, with Gemma holding roughly 80 percent of it and Qwen closer to 61 percent, and rank 32 struck the best balance. The approach also assumes a fixed tool catalog, since the tools are now stored in the weights. The students present it as a proof of concept, which is what it is.

What made the mentorship work

I have supervised many final projects. The ones that turn into papers usually have one thing in common, and it is not raw talent. It is that someone outside the classroom actually needed the answer.

Dr. Patel provided that. The problems were open, not solved in advance. The benchmark was production code with real users at IBM. When a team reported a speedup, it was measured against a system that people actually run. Finished work had to be more than a demo that runs once. It had to be a result that others could check. Shuxin Lin and Dr. Patel worked as collaborators, reading drafts and questioning claims, and the students responded to that.

The course structure did the rest. Profiling made the teams find the real cost before changing anything. The optimization requirement made them choose a technique they could defend. Logging in Weights & Biases produced the before-and-after numbers that separate a real finding from a lucky run. The industrial problem gave the students a reason the answer mattered outside the room, and the course habits gave them a way to show the answer was real. Both were part of how these projects turned out.

What I want to carry into the next version of HPML has less to do with finding an industry partner than with how the problem is chosen. The projects that worked were open enough that the answer was not known in advance, and narrow enough that a two- or three-person team could reach a defensible result in a semester. That balance is hard to get right. Too open, and students spend the term deciding what to measure. Too closed, and they are reproducing an answer someone already has. Dr. Patel's proposals sat in that range because they came from work his team was actually doing, and a problem someone is genuinely working on tends to be both real and bounded. I do not think an industrial partner is the only way to find problems like that, but it has been the most reliable way I have found, and it is the part I will try to reproduce.

A follow-up and thanks

The four projects here are not everything the collaboration produced over the semester. I plan a follow-up post to cover more of it.

My thanks to Dr. Dhaval Patel and Shuxin Lin for giving my students real research problems and working with them as collaborators. And to Alimurtaza, Krish, Sajal, Shambhawi, Sagar, Rohith, Yuval, Ayal, and Tanmay, who competed for these projects and then did the work: thank you.

The four papers

First pages of the four preprints. Click any page to open it on arXiv.

---

References

[1] D. Patel, K. El Maghraoui, S. Lin, et al., "Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents," arXiv:2606.19704, 2026.

[2] A. Merchant, K. Veera, S. K. Goyla, S. Bhure, D. Patel, K. El Maghraoui, "Evaluating Temporal Semantic Caching and Workflow Optimization in Agentic Plan-Execute Pipelines," arXiv:2605.20630, 2026.

[3] S. Chethan Kumar, R. Kanathur, D. Patel, K. El Maghraoui, "Synthetic Scenario Generation for Evaluation of Industry 4.0 Agents," arXiv:2607.22563, 2026.

[4] Y. Shemla, A. Yakobe, T. Agarwal, D. Patel, K. El Maghraoui, "Internalizing Tool Knowledge in Small Language Models via QLoRA Fine-Tuning," arXiv:2605.17774, 2026.

[5] D. Patel, et al., "AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance," arXiv:2506.03828, 2025.

[6] IBM Research, "Meet AssetOpsBench, IBM's first Industry 4.0 benchmark," July 2025.

[7] Anthropic, "Model Context Protocol," 2024.