Bringing Agile and Scrum to Data Science and Machine Learning
Fri Jan 13 2023Software Engineering and Data Science; while both fields require experience using code and technology to drive business value, Agile works well for one, but it can be challenging for the other. As an agile consultant in the innovation space, these are my thoughts on how to reap the benefits of Agile while making sure your Data Scientists are not nerfed.
Why Agile is Great For Developing Software
Agile prioritizes delivering a minimum viable product as soon as possible, and then continuously refining and adding to it based on feedback from customers and stakeholders. This allows teams to respond to changes and new information in real-time, resulting in a better end product that more accurately meets the needs of the users.
Software behavior, is quite deterministic. It's easy to come up with and estimation, have a solid set of requirements, and also prioritize what features are are import and and what are not.
With the competent team of product and engineers, you can always add new features ever week where its gains can be realized by the users.
Why Agile tends to Suck for Research
Doing Machine Learning and Data Science is not a building paradigm. It's a research paradigm. This involves a different way of thinking, and most importantly, its probabilistic.
- The ask is often too broad
- Research takes time
- One is deterministic, the other is probabilistic
Showlife Cycle
Stake holders are unsatisfied by the lack of transparency, or commitment to value Data Scientists are
Why it doesn't have to
What can you do
- Show aspects of the lifecycle
- Embrace the scientific method
- Democratize bubble up knowledge to others in you team
- Automate, DEV OPS
- Invest time in tooling that can make it easier for you to report
- Leave room in your backlog to pivot
- Think about stories as running an experiment
- Hire and use junior engineers to develop these experiments