In growing companies, we often deal with hard limitations on resources and time. In the market, there is an explosion of SaaS tools that enable a company to plug and play their "Business Intelligence" department. While in the open source world, software like Kafka or Luigi and Snowplow are available as the building blocks of large pipelines. 

As a result, more now than ever companies are facing a build vs. buy decision with a strong bias on SaaS for speed of implementation. Unfortunately, this decision often leads to a complete re-engineering of the platform when a company reaches the next level of capabilities or needs compliance. What happens when you have to make sub-second decisions with unstructured data coming from a 3rd party on a different continent? Or factor-in GDPR compliance?

In this talk, we would like to present a considerate and deep technical comparison between the various options available for data infrastructure in a growing company while considering money and human resources. We also want to introduce our "Not-Go-Back" practice at Curve and how we define the need for an evolutionary approach in data engineering decisions that is able to evolve with the size of the company without needing to re-architect the platform.