Converting a project from a waterfall to an iterative approach
Proponents of iterative development techniques -- especially practitioners and consultants who have spent years mastering the Rational Unified Process and related iterative methods for software development -- are often critical of the traditional, or "waterfall," method without stopping to think why waterfall is still prevalent in today's software development organizations. In coming up with project plans for IT solutions we have been asked to design and build for our clients over the years, we have been involved with a number of projects that initially started out using a waterfall approach. These solutions consisted of developing new applications and building out new infrastructure.
In each case there were a number of good reasons for starting off with a waterfall approach, including:
- Waterfall had been successfully used in the past
- We could reuse some of the assets from other waterfall projects
- Our team (that is, our client's staff) would be comfortable with the waterfall approach
- Our team would want all their design work signed off before development occurred
- Our team would be working with other teams who were adhering to waterfall (specifically the team designing and building the infrastructure for our overall solution)
Despite these reasons, using the Rational Unified Process®, or RUP®, on all these projects made more sense than using waterfall. In each case, we were able to change our project plan and overall approach to accommodate iterative practices. This article is based on that experience, and we will walk you through some of the methods we used to go from one approach to another.
What was the original plan, and why?
The original plans usually included a number of design phases where we would outline the solution for the client. Since these projects were going to have multiple releases of software, the initial design phase covered overall design for the software. We would then have a design phase for each release. Each design phase would become more and more detailed as our clients became clearer on their requirements and the design got to where the team could start construction.
Why did RUP make more sense?
With a number of our projects, after reviewing our waterfall plan with the clients, it appeared that a RUP-oriented project plan would make more sense that our waterfall plan, for a number of reasons, including these:
- Clients wanted results sooner. Some clients didn't want to wait until our extensive design work was complete before we started developing. They had some requirements and wanted they wanted to see working application code that they could show to their stakeholders as soon as possible.
- Clients didn't or couldn't provide all the high level requirements before the due date of the first release. Sometimes client couldn't provide all the requirements before the end of the planned design phase, either due to conflicts or other constraints (e.g., a pending decision that won't occur until sometime in the future).
- Clients wanted tightly controlled project timelines and budgets. Despite uncertainty with requirements and other aspects of the project, some clients wanted a tightly controlled project plan. Because each phase of our RUP projects were timeboxed, we were able to accurately state what the project schedule was, resources were required for each phase, and thus what the project cost would be at the end of each phase.
- Clients and developers wanted to attack risk soon in the project. For some of our clients, providing working application code sooner helped alleviate for them the risk that the application couldn't be delivered in time. For some of our development teams, delivering working application components sooner in the project, especially application components related to new technologies, helped them deal with the risk of developing the application.
- Clients wanted the ability to cut off the project if funding changed. By providing the high-value functionality up front in the project, a RUP approach would allow our clients the flexibility of maximizing the value they could get with as much budget as they could get or maintain for the project. If we went with a waterfall approach, we might have a great design for many releases but then find ourselves with only enough funding for one release.
What did we change?
In restructuring our project plans from waterfall to RUP, there are a number of things we changed:
- Project structure. Changing the project structure was a key change. We went from our waterfall phases, with high level design, overall design, release design, build, test, deploy, release-design, build, test, deploy -- to RUP phases, with multiple iterations of elaboration, followed by multiple phases of construction. We might repeat this, and follow it with one phase of transition.
- Timeboxing. Each of the iterations was timeboxed before we started it. If we determined there was not enough time in an Elaboration or Construction iteration to complete our work, we would push the work out to the next iteration. This differed from how we approached phases in our waterfall projects, in which we might extend the phase to complete design, build, testing, or deployment.
- Use of resources. In our waterfall approach, we had resources in the design phases that were no longer in the build/test/deploy phases. In the RUP approach, we kept resources on throughout. On some projects, we had one person performing different roles in different phases.
- Early development. We might start developing parts of the application almost immediately, even if we were in an Elaboration iteration and design wasn't complete. In our original waterfall plan, development was not to start until after design, but in our RUP projects, we tackled risk and provided value by developing some components very early. Specifically, in some cases we started to develop the user interface components in the first few weeks of the project. This allowed us to provide frequent presentations of the application's components, which made our clients happy. It allowed us to get agreement from them regarding the requirements. It also allowed us to continuously verify the quality of the application (i.e., that the application we were building met the client requirements). It also reduced risk to the development team by allowing them to start working on technologies that were new to them (e.g., the user of a persistence framework that they hadn't use before).
What did we keep the same?
While there were a number of things we changed in going from waterfall to RUP, we didn't throw out everything from our original plans.
Linkages to other activities. On all of our projects that were going from waterfall to RUP, there were a number of supporting projects and subprojects going on concurrently. In our original waterfall project we had a critical path that aligned key activities in our project with key activities in their project. When our project moved to RUP, we noted the dates for the key activities in their projects, and then set up new activities in our project to align with theirs.
Roles and resources. We kept the same roles and resources for our projects, even though the structures of the projects changed. We still wanted to get the same amount of design, development and testing performed with the same types of people. As well, some roles not associated with application development (e.g., Infrastructure development) were retained.
Deliverables and other documentation. A number of documents we planned to produce on our waterfall project we still planned to produce on our RUP project. Clients still wanted key deliverables (e.g., Master Test Plan), regardless of our approach. Likewise, we produced a document that told the team building the infrastructure on every project exactly how to set up the servers, software, etc., and we did this regardless of whether or not we used RUP or waterfall.
Amount of effort required. While the effort to build the solution shifted as a result of going from waterfall to RUP, the amount of effort to do the work stayed roughly the same, regardless of the approach.
What was the result?
When going from waterfall to RUP, we found that:
- RUP helped us manage requirements and achieve results even when the clients didn't know initially everything they wanted. Changes needed to be tightly managed and additional phases added to accommodate new requirements. Timeboxing and multiple iterations helped us manage change. Multiple iterations in the Elaboration phase meant that if we couldn't complete our work in one iteration due to the timebox, then we could transition the work to the next iteration if the client still wanted it. Likewise, if we discovered during Elaboration that there is more work involved than we planned for, we could manage the change by shifting some of the work to another iteration. Or we could even add additional iterations if the client felt there was value in pursuing it further. If the client was limited by budget, they could instead hold off on adding iterations, since as they typically had already received significant value from the iterations that had been completed.
- Multiple iterations in the Construction phase also helped with requirements management. By developing parts of the application in an iterative way, we were able to have clients confirm their requirements in each iteration. If the application wasn't the way they pictured it, we could change it in the next iteration. This also helped with scope management.
- RUP helps us continuously verify quality. Because we delivered code at the end of each iteration, RUP allowed testing to start early, and it reduced the risk of major defects being identified much later in the project.
- Modeling visually helps clients discover what they want and helps us understand what they want. Indirectly, it also helps us develop the relationship with our clients. They feel more involved and in control of the direction of the development of the application, and this gives them a great sense of comfort that the application will be completed on time and will contain what they want.
- Using component architecture allowed us to deliver standalone, functioning software to clients for earlier testing. It was also useful in helping the development team break down and assign the work.
When should you consider using RUP instead of waterfall?
The following are some indicators for when you should consider using RUP instead of a waterfall based approach:
- The client has difficulty with aspects of the waterfall approach. For example, the client argues that the time it will take to see the result of design effort is too far in the future. Or the client complains that they need more flexibility or they want to tackle risk early. Or when the client wants to see working components early on in the project that aren't simply prototypes or proofs of concept, but actual working code that will become part of the overall application, RUP offers a framework for that style of development.
- The client is currently using one or more Rational tools. If so, then they already see the value of IBM Rational software and the approach it brings. You can persuade the client that using RUP will further enhance the value of these tools.
- Software development is clearly a key part of the project, and/or the software is object-oriented. If your project is mainly a network redesign or a server consolidation project with very little software development, you might be better sticking with a traditional waterfall approach. Likewise, if most of the software development will be in a scripting or procedural language, traditional methods may well suffice.
- The client has stated they are unsure of parts of their requirements. This could be due to some unresolved issues, or a related project that has not started, or some other area of delay or uncertainty that would prevent the project from being able to completely gather all requirements in one phase.
If the majority of the answers to these questions is "yes", you should consider using RUP for your project.
What are the top things you should do?
- Get an expert in RUP to help you. A RUP expert can provide guidance in transitioning from waterfall to RUP. They will have experience in using RUP and help you identify and plan for any challenges that arise from the transition.
- Learn the value of RUP and iterative development. As you make this transition, you may be asked: is this just change for change's sake? So take courses on RUP, leverage existing assets (this is another area in which a RUP expert can help), and read articles, especially case studies, of other successful examples of how RUP was used.
- Take advantage of the integrated IBM Rational tool set. The RUP framework provides guidance on how to use the tools with the processes via tool mentors. While the Rational tools and processes can be used independently of each other, when used together they can become extremely powerful, since the whole becomes more than the sum of its parts.
- Adopt RUP best practices and be guided by the spirit of RUP. Restructuring your project plan is not enough. For success, you need to be guided by the spirit of RUP1 and you want to follow RUP best practices for the best possible outcome.
- Know your client and your client's culture in order to achieve a successful implementation of your RUP project. Some clients may not want to adopt all or certain parts of RUP, even if all signs point to the benefits they would achieve. Be sensitive to this and adopt accordingly.
Conclusion
There are many reasons why waterfall development methods have been part of our projects, at least initially. Occasionally, we have come onto a project that was bid and won by a team that is used to using a waterfall approach. In other cases, the IT professionals planning the project repeated the same approach they had used for similar projects in the past. In other cases, a client will run all their projects using waterfall, either implicitly or using their own custom methodologies. In all these cases, waterfall presents problems to some extent, and if the client insists on that approach we typically struggle to make it work.
Even with a good waterfall plan, greater benefits can be achieved by switching to a plan based on the Rational Unified Process. To make this transition, you need to look for signs that tell you the client and the project is a good candidate for the change. If the signs are there, before you proceed, it helps to know what you should consider changing as well as what you might want to leave as is. However, with some judicious changing, you can have a project that marries the best of both plans and achieves greater results.
Notes
1 See Per Kroll's article, "The Spirit of RUP," at http://download.boulder.ibm.com/ibmdl/pub/software/dw/rationaledge/dec01/TheSpiritoftheRUPDec01.pdf