Gigaflow cache streamlines cloud traffic, with 51% higher hit rate and 90% lower misses for programmable SmartNICs

A new way to temporarily store memory, Gigaflow, helps direct heavy traffic in cloud data centers caused by AI and machine learning workloads, according to a study led by University of Michigan researchers.

The results were presented at the International Conference on Architectural Support for Programming Languages and Operating Systems in Rotterdam, The Netherlands.

Growing computational demands have led companies to rely upon offsite cloud computing rather than their own infrastructure, causing cloud data centers to scale up processing capacity with higher core densities and faster hardware technologies.

“While workloads are changing, cloud computing also has more resources available to take on more clients at once on a single server. Our research focuses on managing the increased data traffic,” said Muhammad Shahbaz, an assistant professor of computer science and engineering at U-M and corresponding author of the study.

The native server-based architecture typically used for everyday tasks like web browsing, email or file sharing cannot work at this scale because it dedicates physical servers to specific tasks, leaving resources underused.

Cloud environments instead leverage virtual machines—hypervisors that deploy programs and applications, mimicking a physical computer. Several virtual machines can operate at once on a single server to maximize efficiency.

Behind the scenes, software programs called virtual switches direct traffic, deciding which virtual machine a task will go to. Virtual switches could once run on a CPU with a link rate—the maximum speed data can be transmitted—of 10 gigabits per second. With link rates now at 100 to 400 gigabits per second and 800 gigabits per second on the horizon, CPUs can no longer handle the traffic.

To scale to this link rate, data centers are incorporating specialized hardware called SmartNICs (Network Interface Cards) that help accelerate network tasks. Importantly, SmartNICs are programmable, which allows adjustment to specific tasks or network needs.

The Gigaflow software program aims to improve SmartNIC performance by improving caching—a computing process that stores copies of data in a temporary, easy-to-access location.

Typically, virtual switches only cache new data packets, called flows, as they arrive. Known as temporal locality of traffic, this technique provides repeated access to the same data. However, large-scale workloads require more processing for cache generation, slowing down traffic.

Gigaflow instead caches shared rule segments—processing steps multiple flows have in common—instead of processing full flows, which the authors call pipeline-aware locality. Essentially, the system identifies the order of rules in the pipeline, finds the most frequently used rules and makes those easy to reach.

“We believe Gigaflow offers a fresh perspective on how we can rethink caching to capture novel localities, which previously wasn’t possible due to the limitations of available hardware,” said Annus Zulfiqar, a doctoral student of computer science and engineering at U-M and lead author of the study.

The program significantly increases cache efficiency, delivering up to a 51% higher cache hit rate and up to 90% lower cache misses. Gigaflow also captures a 450 times larger rule space, meaning a larger set of rules for directing traffic, while using 18% fewer cache entries compared to existing solutions.

“Lots of groups, both academic and industrial, had accelerated OpenFlow and Open vSwitch in hardware over the years, so I was impressed that Gigaflow got a 51% higher cache hit rate without using bigger caches. It’s a result that I didn’t expect going into the project,” said Ben Pfaff, chief engineer and co-founder of Feldera Inc. and contributing author of the study.

The huge jump in rule space was made possible by leveraging the pipeline-aware locality to divide the cache into smaller rule-processing steps, allowing combinations of the smaller steps in many different ways.

“Most people take established systems concepts, such as caching, for granted and temporal or spatial localities as their only options. Being able to break these assumptions based on emerging trends in programmable architectures and work from a non-traditional and sometimes even contrarian vantage point opens up unique opportunities,” said Shahbaz.

Looking forward, the research team plans to explore new opportunities to capture non-traditional localities in key-value (KV) caching for LLMs as their inference is the dominant workload in data centers today.

Purdue University, Feldera Inc., and Politecnico di Milano also contributed to this research.