www.pydata.org Over the past years, the compute landscape has got much more fragmented and heterogenous: GenAI needs access to various types of GPUs, sometimes leveraging vertical scalability, sometimes horizontal. The demand for CPU-based compute has got more diverse as well, as vertically scaling, high-performance data engines like Arrow and DuckDB have reduced need for inefficient approaches based on horizontal scaling. On top of this, the competition amongst clouds and specialized compute providers is getting more intense, motivated by customer demands for cost-efficiency. Since its inception, Metaflow, which was originally open-sourced by Netflix in 2019, has been built to address diverse compute needs. Instead of proposing a new universal compute paradigm like Spark, which requires bespoke libraries, Metaflow integrates with various compute substrates and providers, including all major clouds. Recently, Metaflow gained support for large-scale distributed workloads, including distributed training on large GPU clusters. In this talk, we give an overview of the changing landscape for compute and describe how open-source Metaflow allows Python developers leverage various compute platforms easily. PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R. PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases. 00:00 Welcome! 00:10 Help us add time stamps or captions to this video! See the description for details. Want to help add timestamps to our YouTube videos to help with discoverability? Find out more here: https://github.com/numfocus/YouTubeVideoTimestamps
|
