Building the Backend: Data Solutions that Power Leading Organizations
Welcome to the Building the Backend Podcast! We’re a data podcast focused on uncovering the data technologies, processes, and patterns that are driving today’s most successful companies. You will hear from data leaders sharing their knowledge and insights with what’s working and what’s not working for them. Our goal is to bring you valuable insights that will save you and your team time when building a modern data architecture in the cloud. Topics will span from big data, AI, ML, governance, visualizations, and best practices for enabling your organization to be data-driven. If you are a chief data officer, data architect, data engineer, data analyst, and those building the backend data solutions then HIT SUBSCRIBE!
Building the Backend: Data Solutions that Power Leading Organizations
Optimizing Spark in the Cloud - with Jean-Yves Stephan
•
Travis Lawrence
•
Season 1
•
Episode 26
This episode features Jean-Yves Stephan Co-Founder & CEO @ Data Mechanics (recently Acq. by Spot by NetApp), during our discussion we talk about optimizing Spark to run in the cloud at a low cost.
Top 3 Value Bombs:
- Running Spark CAN be expensive but there are ways to reduce your current operating costs by 50-75% by smart automations (i.e. tune for node type, memory and CPU).
- Spot instances can lower your costs by utilizing unused instances.
- Creating serverless architectures and using containers will allow for more flexibility with deployment models and scalability.