Successful delivery without scope: managing R&D projects

With the rapid growth of artificial intelligence applicability in all areas of business, creative and unique technological solutions are one of the key factors that can define companies’ financial stability and development perspectives. That is why R&D projects are not perceived anymore as an after-hours playground for enthusiasts, now that it has become clearer that R&D can determine competitiveness in the market. However, that makes stakeholder’s expectations grow even faster. Therefore, how do we find a balance to give the R&D team time to try-and-fail as well as ensure the output that brings actual business value?

Balancing expectations

Recently, I've had an opportunity to lead a multi-stream R&D project. My team consisted of six independent and senior developers, and the budget assumed ten weeks of undisturbed work. That seemed like potentially a great project; however, there were also some important constraints I had to consider (which are quite common across the industry):

• The exact scope was not defined at all. What I got was one or two sentences of description for each stream of work, something like e.g., find options for automated marketing video validation. Just, what does the validation mean in this context? What are the KPIs? What are the unknown unknowns? The requirements were very high-level and defined only the area of research and potential business application, nothing more. Understanding these requirements was part of the project.
• As there was a budget and resourcing involved, the main stakeholder expected not only a research summary, but also some working pre-MVP implementation as an output. Something that was already working and useful enough that could justify further investment in the area.
• At the same time, our stakeholders were difficult to reach due to their limited availability and involvement in various other projects. There was no chance to discuss ideas with them at our daily meetings.
• Finally, the developers joining my project were promised by their managers that they would learn something new and spend an intensive but creative and comfortable time in between regular projects.

As you see, managing all expectations was not an easy task. Some of these requirements may even be perceived as contradictory at first glance. Despite this, I was able to meet these goals based on three main principles: frequent reviews, open communication, and letting the team fail.

Forget about JIRA

In an R&D project, the first rule of successful delivery is that you do not talk about the delivery. I mean you need to consider it as a team/project lead, but you need to make sure the team is not under pressure to deliver a final solution. Otherwise, people would quickly review their positions and change their mode from "try and explore" through "follow the usual path" to "stay in your comfort zone" and "have something to show". And such a mind-switch would move us immediately from R&D to BAU with no way back. It is also important to reorganize and reduce the effort spent on maintaining and using advanced Scrum and Kanban-inspired project management tools. There are at least a few reasons to skip them:

• The initial scope is rarely well-defined.
• Estimates are difficult to calculate (and often replaced by time-boxed actions).
• Scope is dynamic and adaptive (changing based on exploration progress).
• Working in sprints is not efficient as we need to update and review much more often.

Of course, we need some timeline visualization, especially as the time (10 weeks) was in fact the only clear constraint, and therefore the most crucial dimension to illustrate. What I proposed was the following chart, where columns represent time, rows streams of work, and a single chunk of work is a one-sentence-long text box:

Of course, fewer project management tools do not mean less monitoring of progress and solutions. Therefore, it is all about moving the accents and effort to support team creativity rather than the process itself. In particular, the sizes (length) of the task boxes on the diagram above are not based on the precise, end-to-end effort estimation, but rather on time we want and can spend on investigating these aspects.

Let people try and fail

It was Thomas Edison who supposedly once said: I have not failed. I have just found 10,000 ways that will not work. Real science and the most efficient R&D projects are always about trial and error. If the expected output from your project is precisely stated, and all the user stories are already defined, then it is more a regular software development cycle with some options to investigate, rather than typical R&D. Therefore, in such cases, some of the suggestions here may not be relevant. Even so, for every R&D project, it is crucial to let teams experiment and try new paths, even if some of them lead nowhere. Sometimes, there is even no other option other than simply to follow such a path to see the dead end (as you cannot predict it in advance). Such an investigation, although it may be perceived only as a failure at first glance, brings some key information about the available options and allows the team to collect unique knowledge and experience.

What is also important to highlight, an R&D project requires some very specific skills and mindset; it may happen that a developer experienced in regular scrum work may find an R&D project quite challenging and not comfortable from the self-management perspective while another person, usually overwhelmed with project processes, may suddenly become an R&D star. Although an R&D project is often characterised by very specific ways-of-working with limited management tools and creative freedom, this needn’t mean chaos and random activities. We still need to lead the team and manage it, just differently. In my case, the 10-week timespan was the key constraint; therefore, it was important to review and re-plan often, while keeping the space for some tech deep dives.

As the complexity of R&D requests may be often difficult to estimate, we were working on time-boxed tasks, usually planned for 3-4 days. After that time, we reviewed the output internally within the team and brainstormed the topic to identify potential further ideas. After two or three such cycles, we presented the outputs to the stakeholders. During the meeting, each concept was either parked (still being noted down carefully for potential future insights) or agreed as interesting for a further, future cycle of investigation, or already confirmed as a scope for some MVP-targeted implementation. Such balancing allowed us to continue the research, while having some already implemented tasks.

Communication is everything

Open and free communication is crucial for any R&D project. This is not surprising if we realise that autonomy and free access were fundamental to modern academia. Away from politics and courtly intrigues, the first scientists were truly able to spread their wings and release their creative potential. Of course, the role of the project lead is to coordinate the work and discuss the effort with the team to check what is more efficient and suitable in a long-term plan. Even so, frequent brainstorming sessions are the key to reviewing the results achieved so far, and pushing the scope in a good direction. Make sure to have the entire team involved and stimulated to share their thoughts. This is especially important in case of remote and geographically distributed teams. Furthermore, what I found a real game changer was to avoid closed silos and stick to single communication chat (e.g. on MS Teams or Slack) across all the sub-teams instead of having a separate channel for any topic. The reason was clear: if a topic on the chat is not related to a person's current work, they skip it, but at the same time:

• They are free to raise any idea or comment around the topic being raised even if not directly involved (as no-one is really an I-know-everything expert in R&D projects; every single voice like that may be super valuable).
• They feel part of the team and have access to the big picture.
• Looking at other topics for a while may be refreshing and inspiring in their current work.
• They directly and indirectly follow the big picture and on-going scope.
• The potential rotation across the R&D teams is much easier then.
• The risk of FOMO (fear of missing out) is reduced. There is no feeling that something much more exciting is happening just around the corner. No secrets there.

And finally, this is to support so-called osmotic communication that is observed indirectly, usually within some open spaces company cultures: even without direct call or highlight, people seating around are aware of all the important things and crucial issues in their projects. Especially nowadays, in the post-pandemic world with many companies deciding to stay in a remote-work mode forever, open and general channels are key to maintaining such osmotic communication to a certain degree.