JAX DevOps Blog

How to enable a company to become “data driven”

7 Mar 2018

When is the right time to set up your own delivery pipeline?

When is the right time to set up your own delivery pipeline? Source: Shutterstock

How can a company become “data driven” while maintaining non-functional requirements in scope? In this interview, Pierre di Lauro, JAX DevOps speaker and Data Engineer at Curve presents the tools that are currently available on the market, explains why you should choose SaaS when open source softwares like Kafka, or Luigi and Snowplow are readily available and talks about the common issues with data when a company is growing.

JAXenter: Can you go over the build vs. buy dilemma that so many companies are facing?
Pierre: Given the large amount of services available in the space, companies are often facing the question whether they should build or buy a data analytics solution. Given the large amount of SaaS that are now populating the market,the cost and the ease of access, this seems to be an obvious choice. Unfortunately though, after choosing a vendor without framing its specifics and reducing the logic offloaded, there is a high risk of getting locked-in and having to rebuild everything from scratch after a few years when the company grows and the requirement changes.

JAXenter: Why choose SaaS when open source softwares like Kafka, or Luigi and Snowplow are readily available?
Pierre: Nowadays, startups are facing huge challenges around quick growth and development. In such an environment, you want to use your time and resources as efficiently as possible.
Building a reliable ETL or Data Streaming platform can take up to several months of development from multiple engineers and will require maintenance. These products are a mean to an end. The end goal is to provide the right insights for the company to rely on when taking important decisions.
Small companies need to focus on the end goal. SaaS like Mixpanel, Stitch, AWS Data Pipeline, AWS Glue and others will help you focus on the Data Warehousing, the business logic and delivering the right insights to help the business do the right choices.
As the company and the needs grow, you will have more time and resources to commit to the challenge of building your own pipelines.

When investing time in a project, always ask yourself: is this helping the business be where it needs to be in 2 years from now?

JAXenter: What happens when you have to make sub-second decisions with sending data to a 3rd party in a different continent? Or factor-in GDPR compliance ?
Pierre: We constantly keep compliance at the back of our head. Regulations become more and more present in this field and we always make sure we include this dimension in our engineering and design work.

JAXenter: Can you explain more about Curve’s “Never Go Back” motto?

Pierre: Always keep in mind the business point-of-view in your choices to make sure we will bring value to the business in the short-term and long-term.
When investing time in a project, always ask yourself: is this helping the business be where it needs to be in 2 years from now?
Keeping the long-term objective in mind helps making sure the time spent is not lost.

 

 

JAXenter: What can attendees expect from your talk?

Pierre: The focus of the talk will be around the definition of a data strategy in a startup from an engineering point of view. How to enable a company to become “data driven”, while maintaining non-functional requirements in scope.The attendees will learn what tools are available in the market, what are the common issues with data when a company is growing and how we overcome them. We are also introducing our solution based on Snowplow, the open source event-based analytics platform that we’re running.

JAXenter: How can other companies make this decision for themselves?
Pierre: It should ask itself some basic questions : Is the software critical to current operations or your plans for business growth? Is it core business ? Or else, is it a core differentiator ? Is there an existing, off-the-shelf solution that does the job well enough for now then you should probably go? What is the total cost of ownership over the lifecycle of the software? The answer to those questions should balance one option over the other.

Thank you very much!

JAX DevOps Sessions by Pierre di Lauro:

Tuesday, April 10 2018
18:00 – 18:50

Behind the Tracks

BUSINESS & COMPANY CULTURE
the process of becoming fully agile
CLOUD PLATFORMS
Cloud-based & native apps
DOCKER & KUBERNETES
Docker, Kubernetes, Mesos & Co
CONTINUOUS DELIVERY
Build, test and deploy agile
MICROSERVICES
Maximize development productivity
Business & Company Culture

Business & Company Culture

Cloud Platforms

Cloud Platforms

Docker & Kubernetes

Docker & Kubernetes

Continuous Delivery

Continuous Delivery

Microservices

Microservices

Monitoring & Diagnostics

Monitoring & Diagnostics