This episode features Jean-Yves Stephan Co-Founder & CEO @ Data Mechanics (recently Acq. by Spot by NetApp), during our discussion we talk about optimizing Spark to run in the cloud at a low cost.
Top 3 Value Bombs:
- Running Spark CAN be expensive but there are ways to reduce your current operating costs by 50-75% by smart automations (i.e. tune for node type, memory and CPU).
- Spot instances can lower your costs by utilizing unused instances.
- Creating serverless architectures and using containers will allow for more flexibility with deployment models and scalability.