This post is my own summary of Netflix’s tech blog post Seeing through hardware counters: a journey to threefold performance increase, written the way I understood it, with some background I added along the way. The original post is the authoritative source, so check it directly for exact figures and details. The Problem A Netflix service was running short on CPU, so the team scaled its nodes up 3x. Given the CPU-intensive workload, they expected throughput to scale roughly in step. Instead, they got about a 25% improvement, and tail latency actually got worse. ...

Locks split into two broad families: spinlocks, which run entirely in userspace on raw CPU instructions, and mutexes, which lean on the kernel. Both exist to guarantee mutual exclusion, but they get there differently, and that difference decides which workload each one fits. Spinlock A spinlock lives entirely in userspace. The mechanism is simple: loop forever until a CAS (Compare-And-Swap) succeeds. while (!CAS(lock, 0, 1)) Most CPUs ship a dedicated instruction for this. On x86 it’s LOCK CMPXCHG. The catch is that a CAS only succeeds once the core holds exclusive write access to that cache line. Getting there pulls the cache coherence protocol into play, and passing ownership of a cache line back and forth triggers frequent invalidation, what people call cache line bouncing. That round trip usually costs somewhere between 4 and 80 nanoseconds. ...

Swiss Tables:A Modern, High-Performance Hash Table Swiss Table is a high-performance hash table design introduced by Google engineers in 2017. It has since inspired many standard-library implementations across languages, including: Go 1.24 ships its map with this design (up to 60% faster) Rust’s standard HashMap has also moved from Robin Hood hashing to a Swiss Table-inspired layout. Datadog reported as much as 70% memory savings after migrating to Swiss Table Open Addressing, Briefly An open addressing hash table is one of the implementation methods for hash tables. Unlike separate chaining — which uses external data structures such as linked lists or trees — open addressing implements the entire hash table as a single contiguous array. ...

This post is my own write-up of Rust hash iteration+reinsertion and the related Rust issue/PR, written the way I understood them. The original is the authoritative source, so check it directly for the exact details. I want to walk through an interesting bug that showed up in Rust’s HashMap. The code looks completely ordinary, yet under the right conditions, an operation that should be O(n) blows up to O(n²). Reproducing the Bug Look at the code below. It inserts values 1 through 5,000,000 into a first hash map (one, that’s T1), then iterates over one and reinserts every value into a second hash map (two, that’s T2). Nothing fancy. ...

SIMA (Scalable Instructable Multiworld Agent), which DeepMind published in 2024, is a generalist agent built for 3D virtual environments. Give it a screen and a simple natural-language instruction, and it plays a 3D game almost the way a human would. Not One Game, But Many What makes SIMA interesting is that it isn’t a bot tuned for one specific game. Working with eight game studios, DeepMind trained it across nine titles, including No Man’s Sky (exploring alien planets), Satisfactory (building automated factories on an alien world), and Valheim (a Norse-mythology survival crafting game). It can follow human instructions and play games it has never seen before. ...

EKS’s default service routing spreads traffic across a cluster’s pods randomly, or round-robin. That means requests frequently land on a pod in a different availability zone (AZ), and since AWS VPC charges for cross-AZ data transfer, that traffic pattern translates directly into higher cost and added latency. Topology Aware Routing (TAR), introduced in Kubernetes 1.24, targets exactly this problem: when pods talk to each other, it adjusts network policy to prefer a pod in the same AZ whenever one is available. ...

A Kafka partition has always been limited to one consumer at a time. That coupled partition count and consumer count tightly together, and scaling throughput often meant splitting a topic into far more partitions than the data itself justified. Plenty of workloads genuinely fit a queue-style model better, where several consumers split the work on one partition, but the old structure had no way to express that. KIP-932: Queues for Kafka removes that constraint with a new group type: the Share Group. ...

Here’s a rundown of KIP-848: The Next Generation of the Consumer Rebalance Protocol, which reached GA in Kafka 4.0. Its two central goals: cut rebalance downtime to nearly zero when consumer group membership changes, and move most of the responsibility for rebalancing from the client to the broker. Background The consumer group rebalancing protocol had been around for eight years, and it was running into structural limits. The biggest one was a thick-client design that put too much responsibility on the client. When a bug showed up in consumer group rebalancing, fixing it meant fixing the client, and if you’re running a cloud service, you can’t force your users to patch their own clients. Since most of the logic ran on the client side, diagnosing problems from server-side logs alone was often impossible. ...

Limitations of Traditional Scalar State Machine Parsers The JSON parsing algorithms we commonly use are based on scalar state machine parsers. Scalar parsers read the input string byte by byte, parsing it through state transitions within the state machine. For example: When encountering a quotation mark ("), it indicates the start of a string. When encountering a colon (:), it indicates that a value is expected next. Below is a simplified pseudo-code representation of how a scalar parser works ...

This post explains how coroutines work, referencing the design proposal Kotlin Proposals - Coroutines. Coroutine The proposal describes a coroutine in one sentence as an instance of suspendable computation. The essential trait of a coroutine is its ability to suspend. So what exactly does “suspendable” mean? Suspension According to the proposal, suspendable means a coroutine can pause execution on the current thread, yield the thread so another coroutine can run, and later resume—possibly on a different thread. ...