How Roblox Scaled Its Kafka Platform to Over 18 Trillion Messages a Day

End-to-End Optimization for Efficient Systems

In the past two years, we scaled our internal message queue platform over 2.5x each year to over 18 trillion messages per day. We currently process over 300 million messages per second while maintaining 99.995% availability. This message queue platform, built on Apache Kafka, has become a critical part of how we move data across the company. It supports workloads such as analytics events, moderation pipelines, matchmaking systems, and many more. These numbers are exciting, but the more important story is how we got there. 

The Impact of End-to-End Optimization 

A common misconception about large-scale systems is that capacity is mostly a hardware problem. In our experience, the most impactful improvements are made by treating scalability as an end-to-end engineering problem.

Teams across Roblox have built various platforms on top of the messaging queue, including AI training pipelines, stream processing abstractions, and telemetry systems. Partnering with the owners of these platforms allows us to help ensure adherence to best practices such as enabling batching and compression. This results in efficient systems without involving end users in detailed configuration setup.

Aligning with teams that use the platform on business priorities, measurement, and success criteria is critical and helps drive cross-team collaboration. Key metrics vary between layers of the system: While we may optimize the Kafka servers to efficiently handle large data volumes with fewer machines, end users may want to optimize for end-to-end latency down to the millisecond level, or for total time for their batch job to complete writing data. 

Not All Traffic Is Equal

One key lesson we’ve learned is that large-scale platforms are more reliable when they don’t treat all traffic equally. Some workloads require very low latency and strong protection during spikes, such as our matchmaking systems, which operate with under 10 milliseconds of end-to-end latency. Others are extremely high volume yet more tolerant of delay, like our analytics event ingestion pipelines, which can run smoothly even with multiple seconds of end-to-end latency. If both are handled identically, the highest-volume traffic can crowd out more business-critical traffic, especially during peak load.

To solve this problem, we group traffic by use case requirements to ensure that each one is well supported. The trade-off between end-to-end latency and throughput in Kafka is one of the primary considerations when setting up new use cases. This holds true through the data path from producer to server to consumer: Each of these has configurations that can be tuned to optimize for either throughput or latency. Client-side configurations can be tuned for individual use cases, but server-side configurations are less flexible, as it’s untenable to maintain a dedicated cluster for each distinct use case. 

We apply this approach of optimizing each cluster for the use cases it serves down the stack to the hardware layer. 

Hardware Is Still Critical

We started by running Kafka on RAID10 HDD machines. As traffic volume grew, our machines’ disk I/O capacity became the primary bottleneck limiting per-server Kafka throughput. To address this, we made two optimizations. We converted brokers to use SSDs as much as possible, shifting the bottleneck from disk I/O capacity to network throughput or disk space (as the SSDs we acquired had less disk space). This increased disk I/O throughput by 10x. We also began converting our remaining HDDs from RAID10 to RAID0. This roughly doubled disk I/O capacity, as each write with RAID0 needs only a single disk write rather than two. 

The trade-off is that since each message replica is written to one disk per broker instead of two, moving from RAID10 to RAID0 can come with reliability risk. To maintain reliability, we worked with important use cases to increase acks and data replication, including replicating to multiple regions.

Leveraging each machine’s storage and throughput capabilities fully required additional work. We optimized our memory allocations and page cache management configs so that Kafka can buffer messages and serve reads or recent data from memory, only hitting disk and consuming valuable IOPS when necessary, such as when reading historical data. 

We also run performance benchmarks to understand hardware limits and determine what software-level limits we need to set. In practice, this means setting Kafka broker read and write throttles, which vary based on the hardware the clusters are running on. Our SSD machines can handle higher throughput per broker, while our HDD machines are better suited for use cases requiring long-term retention of historical data. 

Building for Reliability

The queue platform is a critical component of Roblox’s infrastructure, and maintaining reliability is crucial. Prelaunch testing as well as ongoing failure scenario testing (sometimes called game day exercises) have been instrumental in validating that our infrastructure is resilient to failure modes such as single host failure, rack failure, networking issues, and many others. Our reliability team’s internal chaos testing platform has enabled and streamlined the process of simulating various failure scenarios. These tests help identify system improvements or regressions and build operational excellence by regularly validating alerts and ensuring that operators are familiar with the steps to handle various issues—even rare ones.

We’ve also designed and implemented failover processes that can quickly restore queue functionality for dependent use cases. Even in the case of total failure of an individual cluster, we are able to switch clients to a healthy cluster with sufficient capacity. In order to ensure that we have enough unused capacity available, we maintain a standby Kafka cluster. This helps eliminate any risk of noisy neighbor issues when shifting traffic between live clusters.

The below diagram shows the high-level steps for executing a failover. First, an operator identifies the scope of the failover. Then, topics and access control lists (ACLs) are created on the standby cluster with an automated bulk operation. Finally, the operator updates our control plane service configs, which clients poll to determine whether any reconnection is needed. The illustration below is for generic real-time use cases. We have additional strategies and tooling for other use cases and scenarios, including topic mirroring using Mirror Maker 2, data backfill tooling, and more.

Roblox has strict requirements for security, which is also a critical part of reliability. We’ve implemented standard security features like encryption in transit (with mTLS), encryption at rest (with encrypted disk), data retention age requirements, and authentication/authorization with Kafka principals and ACLs. We also automatically rotate our users’ credentials to further improve security.

Besides this, we built custom handling in our control plane to facilitate fast OS patching and rack drains. During OS patching, we retain data on disk so patched machines can rapidly regain healthiness after rejoining the Kafka cluster. This way we can patch our entire fleet within 30 days, even as it continues to grow. With tools like Cruise Control, we replicate our data in a rack-aware way, so that entire racks can be taken offline for maintenance and network patching without affecting data availability.

The Road Ahead

The next chapter for our queue platform is about making the platform even simpler for developers while continuing to improve elasticity and isolation. We are investing in new abstractions, including producer and consumer proxies as well as fully managed connectors. These systems will further decouple application teams from infrastructure complexity and make our platforms easier to adopt and manage at scale. For example, a consumer proxy will enable the competing consumer pattern, allowing the platform to support a wider range of use cases and decouple scaling of processors from scaling of Kafka partitions. This is valuable for use cases like processing mass bursts of notifications, which don’t require ordering and can benefit from temporarily scaling up processor count without bound-to-achieve end-to-end delivery targets.

Scaling Kafka is not just about scaling the number and size of clusters. It’s about building the engineering discipline, abstractions, and operational model that allow Roblox to continue on our journey to connect a billion people every day with optimism and civility. 

Acknowledgments: Thank you to Danny Yuan and George Li for their significant contributions to the Roblox queue platform.