Apache Submarine
Projects | | Links:
Apache Submarine (Submarine for short) is an End-to-End Machine Learning Platform to allow data scientists to create end-to-end machine learning workflows. On Submarine, data scientists can finish each stage in the ML model lifecycle, including data exploration, data pipeline creation, model training, serving, and monitoring.
Abstract
Apache Submarine is an open-source, cloud-native machine learning platform that is part of the Apache Software Foundation. For more information, please visit the following links.
Experience
I dedicated myself to refactoring the deployment flow of Submarine and developing a multi-tenancy mechanism. Our goal was to provide users with isolated environments for developing ML models. We designed a Kubernetes Custom Resource (CR) to help create these isolated environments. The CR can be thought of as a bundle of Kubernetes resources that are associated with a specific user or group. To manage these CRs, we implemented the “Submarine Operator” to control their lifecycle and monitor their status. Additionally, I implemented end-to-end tests to evaluate the functionality. So far, I have had over 20 pull requests merged into the released version.
In addition to my contributions to the codebase, I also made contributions to the community. I noticed that many potential contributors were missing information about the community, so I dedicated myself to expanding the community in Taiwan. Last year, I gave speeches at universities and shared my experiences with Apache Submarine. I also invited senior contributors to the Apache Foundation to share their experiences with multiple projects. Furthermore, I connected with Professor Hung-Yi Lee to support the project and helped organize people who were interested in it. As a result, more than ten contributors engaged with the community and brought new prospects to it.
References
Reflections
I enjoyed contributing to open-source. It is rare to have the opportunity to develop such a large application with experienced engineers from around the world while in college. However, joining the open-source community truly broadened my perspective and strengthened both my hard and soft skills. I hope that more potential engineers in Taiwan will discover the benefits of open-source and become contributors in the future.
Last but not least, shout out to Kai-Hsun Chen and Byron Hsu for inviting me to the community!
