How Spotify Actually Works

Every December, hundreds of millions of people open Spotify Wrapped within roughly the same few hours, and the app has to instantly know exactly how much of The Weeknd you played at 2am in March. That single feature quietly depends on hundreds of independent services, each built and owned by a small, self-contained team, none of which needed to ask anyone else for permission to ship their part of it.

That team structure isn't an HR chart detail it's the actual architecture. Spotify's engineering organization and its technical system were built to mirror each other almost exactly, and understanding one means understanding the other. This article looks at how that came to be: the "squad" model that shaped the whole backend, a genuinely dramatic outage caused by a literal undersea cable, and the years-long migration off homegrown infrastructure and onto Google Cloud. A good amount of this draws directly from Spotify's own engineering blog, alongside independent write-ups and interviews filling in the surrounding story.

The Squad: An Organizational Unit That Became an Architectural One

Spotify's own engineering blog, in a foundational 2013 post called Backend Infrastructure at Spotify, lays out a principle that shaped everything downstream: every development team, called a squad, should be autonomous always able to move independently of other squads, even when a dependency between two teams exists. The blog is direct about what drives this: as the company grows, a huge amount of effort goes into eliminating dependencies between teams and stripping out unnecessary complexity, specifically to protect development speed.

A talk on modeling microservices at Spotify, covered on DZone, describes the actual size and scope of a squad concretely: a group of roughly five to nine people who own something end-to-end. The example given is a squad owning the search function meaning that one small team owns the user interface for search across every client, iOS, Android, browser, all of it, plus the backend services actually powering it.

A restaurant where the same small team that designs a dish also cooks it, plates it, and handles complaints about it directly, rather than a chef designing a recipe that a separate kitchen staff cooks, that a separate waitstaff serves, and that a separate manager handles complaints about when something goes wrong with that dish, there's no confusion about who owns fixing it, because one team owned the entire thing from idea to plate.

Spotify's squad model:
  Squad "Search" → owns search UI on iOS, Android, web
                 → owns the backend services powering search
                 → owns the Hadoop jobs powering search-related data
  One team, one feature, end-to-end, across every layer of the stack

Partitioning the Problem So Failures Stay Contained

The same 2013 engineering blog post explains the deeper reasoning behind this structure: any architecture handling Spotify's volume of users needs to partition the problem, and Spotify does this primarily by feature. Nearly every screen and view across every client belongs to a specific squad, and this is the important architectural payoff if one feature fails, the other features remain independent and continue working.

A large apartment building where a plumbing issue in one unit doesn't cut off electricity to the rest of the building, because the systems were deliberately kept separate rather than run through one shared, central point of failure a bug in the recommendation squad's service showing up on your home screen doesn't take down your ability to search for a song or play your saved playlists, because those are entirely separate services, owned by entirely separate teams, with no shared point where one failing necessarily drags the other down with it.

A more recent write-up on Medium by Parijat Biswas, describing how this plays out for a feature like Wrapped, makes the same point in practice: if the specific service responsible for generating your mood categories has a temporary issue, the rest of Wrapped still works. Each microservice is optimized for the particular kind of workload it actually handles high throughput, heavy computation, or fast, low-latency reads rather than every service being forced to compromise around a single, generic set of assumptions.

The Day a Shark (Maybe) Nearly Took Down Spotify

Here's a genuinely great story about why architecture decisions sometimes come from the strangest possible triggers. A retrospective on Okoone's blog describes an incident from around 2015, when Spotify was supporting roughly 35 million users on PostgreSQL: a transatlantic data cable connecting Spotify's London and U.S. data centers failed, and some engineers reportedly suspected a shark bite as the cause.

Whatever actually severed that cable, what mattered was the underlying database architecture's fragility being exposed by the incident. The same source frames Spotify's response as the real lesson: scaling isn't just a technical afterthought, it's a genuine operational decision, and it was this specific incident that pushed Spotify to move from PostgreSQL to Cassandra, a distributed database explicitly designed to keep functioning even when part of the network connecting its nodes becomes unreliable or unreachable.

A single family keeping their only copy of an important document in one specific filing cabinet, in one specific house, only to have a random accident destroy that one house and only then realizing important documents probably should have been kept in more than one place all along. A database that assumes a reliable, always-available connection between two distant data centers is making a similar bet, and Spotify's transatlantic cable incident is exactly the kind of real-world event that exposes how risky that bet actually was.

Before: PostgreSQL, largely dependent on a stable link
  between London and U.S. data centers
  → Transatlantic cable fails → real operational pain
 
After: Cassandra, a distributed database built specifically
  to tolerate exactly this kind of network partition
  → A severed link degrades gracefully, rather than breaking outright

Leaving the Data Center Business Entirely

This next chapter is a genuinely large undertaking: moving an entire company's backend, built on its own hardware across four data centers worldwide, onto someone else's cloud. A report on DataCenterDynamics describes the scale of what was being left behind: a mix of Spotify's own and rented data centers, blended with a bit of AWS, being consolidated entirely onto Google Cloud Platform.

A piece from Virtual Tech Gurus explains the actual reasoning Spotify gave for the move: the company didn't want to remain in the data center business at all, citing both the cost involved and a lack of the specialized expertise needed to keep improving that infrastructure themselves, while Google's strength in big data services was a genuine draw in the other direction. The same piece notes a detail that shows this wasn't an all-or-nothing bet: Spotify kept its actual audio files stored on Amazon S3 and continued delivering music worldwide through Amazon's CloudFront CDN, even while moving the rest of its backend to Google. Spotify's own big data architect, quoted in the same piece, put the reasoning simply: Amazon was kept specifically for content delivery, and Spotify intended to keep doing that.

Splitting a business that outgrows managing its own real estate into two clean decisions: hand off the physical office buildings to a property management company that's actually good at that job, while still keeping your own trusted, established courier service for physically shipping your product to customers, because that particular piece was already working well and didn't need replacing. Spotify didn't treat "move to the cloud" as a single, uniform decision it kept the part of its infrastructure (content delivery) that was already working well, while moving the part that had become an operational burden (data center management).

Building Your Own Orchestrator, Then Retiring It

Here's a detail that says a lot about how early Spotify actually was to some of these ideas. A Kubernetes case study, published on Kubernetes' own site, notes that Spotify had containerized microservices running across its fleet of virtual machines since 2014, using a homegrown, open-source container orchestration system it built itself, called Helios well before Kubernetes had become the dominant standard it is today.

The same case study describes what happened once Kubernetes matured: rather than stubbornly sticking with a tool they'd built themselves purely out of a sense of ownership, Spotify made the switch, with Site Reliability Engineer James Wen noting that Kubernetes's own APIs and extensibility features made integrating with Spotify's existing legacy infrastructure straightforward. The same source notes migration accelerated through 2019, though stateful services the trickier category, involving actual persistent data rather than easily-replaceable request handling needed considerably more work than the stateless ones did.

A carpenter who built their own excellent hand tool years before a mass-produced version existed, and later, once that mass-produced version genuinely became better-supported and more broadly maintained than their own one-off creation, switches over to it without any wounded pride about having built the original themselves the goal was always having the best available tool, not proving that their homemade one was permanently the right choice.

Redesigning the Data Pipeline Underneath a Live, Running System

A post on Spotify's own engineering blog, titled Changing the Wheels on a Moving Bus, captures the genuine difficulty of this kind of migration in its title alone. The post explains that Spotify logs an enormous range of data listening history, A/B testing results, page load times through a system called the Event Delivery Infrastructure, or EDI, and that in 2016, alongside the broader move to Google Cloud, this entire pipeline was redesigned from the ground up.

A related Google Cloud post on the migration from Kafka to Pub/Sub fills in the destination of that redesign directly, and a further breakdown on Happtiq's blog puts real scale numbers behind the resulting system: Pub/Sub, the event delivery service Spotify moved to, now handles over one trillion requests per day at very low latency, while BigQuery, used for ad hoc querying, processes more than 10 million queries and scheduled jobs every month.

Swapping out the entire plumbing system of a building while the building stays fully occupied and every faucet keeps running the whole time the fix has to happen underneath a system that can't simply be switched off while the work is being done, which is exactly the challenge the blog post's own title is winking at.

Keeping Streaming Smooth on a Bad Connection

A more consumer-facing piece of this architecture, described on Sprintzeal's blog, covers Adaptive Bitrate Streaming dynamically adjusting audio quality based on a listener's current internet speed, so someone on a weak connection gets a lower bitrate stream instead of constant buffering, while someone on a fast connection gets the full, high-quality audio. The same piece reports Spotify streams over 200 petabytes of data every month, with this adaptive approach alone estimated to save around 25% in bandwidth costs, alongside preloading small audio chunks in advance specifically so playback starts within milliseconds of hitting play, rather than visibly loading first.

A driver on a winding mountain road automatically shifting into a lower gear on a steep incline and back up to a higher one once the road flattens out, constantly adjusting to current conditions rather than sticking rigidly to one fixed gear regardless of the terrain that's the same instinct behind adaptive bitrate streaming, just applied to audio quality instead of a car's transmission.

Putting It Together

None of the individual pieces here are exotic inventions on their own. Microservices are a widely used pattern. Cassandra and Kubernetes are both mature, well-documented technologies. Adaptive bitrate streaming is an established technique used across the entire streaming industry. What makes Spotify's story genuinely interesting is how tightly the organizational structure and the technical architecture ended up mirroring each other small, autonomous squads owning features end-to-end, matched by a system deliberately partitioned along those exact same lines, so that a team's independence and a service's independence were never really two separate problems to solve, but the same one, solved once.

What This Means Going Forward

As Spotify keeps expanding well past music into podcasts, audiobooks, and increasingly personalized, algorithmically curated experiences delivered to hundreds of millions of people at once, the same underlying tension keeps showing up in a new form: how do you keep hundreds of independent teams moving fast without the resulting system becoming an unmanageable sprawl nobody can see the shape of anymore. The lesson underneath Spotify's story isn't really about Cassandra, Kubernetes, or Google Cloud specifically. It's the same one running through this entire series: a genuinely resilient, fast-moving system usually isn't the product of one clever piece of technology, but of an organization that deliberately shaped its teams and its architecture to reinforce each other, long before any severed cable or cloud migration ever put that decision to the test.