Segment, the startup Twilio bought last fall for $ 3.2 billion, was just starting to take off in 2015 when it ran into a scaling problem - it was growing so fast, the tools it had built to process marketing data on its platform were starting to surpass the original system design.
Inaction would have led the company to hit a technological wall, managers feared. Every early stage startup craves growth and segment was no exception, but he also had to start thinking about how to make his data platform more resilient or reach a point where he could no longer handle the data he was moving through the system. It was - in a real sense - an existential crisis for the young business.
The project that came out of their efforts was called Centrifuge and its purpose was to move data through the Segment data pipes where customers needed it quickly and efficiently at the lowest operating cost.
Segment's engineering team began to think hard about what a more robust and scalable system would look like. As it turned out, their vision would evolve in a number of ways between late 2015 and today and, with each iteration, they would make a leap in efficiencies in allocating resources and processing data on the go through its systems.
The project that came out of their efforts was called Centrifuge and its purpose was to move data through the Segment data pipes where customers needed it quickly and efficiently at the lowest operating cost. This is the story of how this system came about.
Growing pains
Systemic problems became evident as they often do when customers started complaining. When Tido Carriero, Segment's chief product development officer, boarded at the end of 2015, he was tasked with finding a solution. The problem was with the design of the original system, which like many early iterations of startups was designed to bring the product to market with little consideration for future growth and the technical debt payment was due.
"We had [designed] our initial integration architecture in a way that simply wasn't scalable in many different ways. We have seen tremendous growth and our CEO [Peter Reinhardt] he came to me maybe three times in a month and reported about various scaling challenges that customers or our partners had warned him about, "Carriero said.
The good news was that it was attracting customers and partners to the platform in no time, but it could all have collapsed if the company hadn't improved the underlying system architecture to support robust growth. As Carriero reports, it was a stressful time, but having come from Dropbox, he was actually able to understand that it is possible to completely redesign the company's technology platform and live to tell it.
"One of the things I've learned from my past life [at Dropbox] It's when you have a problem that is so central to your business, at some point you start to realize that you are the only company in the world experiencing this problem at this type of scale, "he said. For Dropbox that was related to the storage and per Segment was processing large amounts of data at the same time.
In the build-versus-buy equation, Carriero knew he had to build his way out of the problem. There was nothing out there that could fix Segment's unique scaling problems. "Obviously this led us to believe that we really need to think about it a little differently, and that's when our Centrifuge V2 architecture was born," he said.
Building the imperfect beast
The company began measuring system performance, processing 8,442 events per second at the time. By the time he started building V2 of his architecture, that number had grown to an average of 18,907 events per second.
