Don’t Make Data Scientists Do Scrum - Sophia Yang, Towards Data Science

This resource first appeared in issue #105 on 16 Jan 2022 and has tags Strategy: Project Management, Managing A Team: Data Teams

On the one hand, research computing and data projects, especially the intermediate parts between “will this even work” and “put this into production”, often map pretty well to agile approaches - you can’t waterfall your way to research and discovery.

On the other hand, both the most uncertain (“Will this approach even work?”) and the most certain (“Let’s install this new cluster”) components are awkward fits to most agile frameworks, even if in partially different ways. The most uncertain parts are basically 100% research spikes, which short-circuit the usual agile approach; the most certain parts you don’t want a lot of pivoting around. And both ends of the spectrum benefit from some up-front planning.

Here Yang, who’s both a data scientist and a certified scrum master, argues against using scrum to organize data scientists, whose work is generally firmly on the “uncertain” side of the spectrum.

The argument is:

You need flexibility
Data scientists do not work on a single “product”, and work is often not an increment
Data scientists do not always need a product owner, and..
taking the ownership away from the data scientist is often not helpful
The definition of “Done” varies across projects
Sprint review doesn’t work

This doesn’t mean agile approaches aren’t useful, but they need some grounding in the nature of these more research-y efforts. This is true of some research software development efforts, too. Models like the Team Data Science Process or CRISP-DM or the like are worth investigating - not necessarily for verbatim adoption for a research process, but for getting a bit more nuance and structure.