Should you use Agile for data science (or research?)

This resource first appeared in issue #2 on 17 Jan 2020 and has tags Strategy: Research to Development Maturity Ladder

One of the big differences between research computing and regular software development or IT has always been the open-endedness of the work, and in particular that we’re often starting in a “will this even work” mode rather than building something we know is doable and it’s just a matter of discovering and validating the user requirements (something that Agile approaches help a lot with.)

There was an interesting blurb in R&D Today abut the Manhattan project as an example of a project where everything was just too new to able to form hypotheses and perform experiments; they just had to go through a stage of trying a lot of things, noting the outcomes, and slowly getting to the point of finding more solid ground.

In data science and particularly ambitious startups there is an increasing realization that you can’t just agile your way from nothing; you have to go through that same casting about phase first. That means actually mixing stagegate approaches (waterfall! gasp!) with agile, or going through separate R&D, Design and then Engineering/Productionizing phases. Basecamp has a process around it - transitioning away from “R&D”. I think as data analysis becomes more mainstream, research software development may become more and more similar to other areas of software development.