10 Keys to Agile Transformation
I have had the opportunity to lead an Agile team through a complete Agile transformation from a waterfall methodology and wanted to share some of my experiences of both success and failures. Agile adoption continues to gain traction and will continue to evolve over time. There are many different flavors of Agile that organizations have adopted, and thus many misconceptions of how it should work. Here are my critical success factors for Agile.
1. Agile Transformation Top Down Approach
Agile must be driven from the top of the organization. This not only includes the IT organization but business as well. This must be a significant paradigm shift in order to be transformational. I had the opportunity to see this first hand as a result where a much smaller company acquired part of a Fortune 500 company which had stringent guidelines for IT projects and the overall business sector was in the insurance industry which is heavily regulated. I honestly had my doubts and wasn’t sure it could be successfully implemented. I was wrong. In looking back on this successful transformation, I truly saw how it should operate. There was a clearly defined approach and methodology which also allowed flexibility when needed.
The organization experienced transformation from a large structure to a small and nimble team. This transformation did result in a significant downsizing but it was necessary to help achieve the objective. It was a great opportunity for those involved to see how real change happens. From the CEO on down, you could see how this change affected how the organization was lead and managed. This happened across both business and IT which is unique. In my opinion, the acquisition helped force a cultural change that would not have happened with the previous management structure in place.
2. Agile Transformation Requires Continuous Training and Coaching
As a part of my Agile Transformation journey I had the opportunity to see how critical Agile coaching and training is needed. It is not something where a few people in the organization can get trained or certified as Scrum Masters or Product Owners in a few weeks of boot camp training. It must be constant. There must be trainers who are a part of the organization that can work with the various teams to sit and coach them on what will work best within their specific agile team. This must continue as there are always different challenges and obstacles that will present themselves over a period of time. I simply cannot express how critical this point is needed.
IT groups love training, unfortunately it is usually something that is not constant and is generally limited to a few people who go off to week long conferences and then use the train the trainer approach and expect that to transform an entire organization. The train the trainer approach is this case simply would not work. In addition, training budgets are constantly getting squeezed due to cuts and that is usually the first thing to go when things get tight.
3. Agile Transformation requires a Mindset change
The definition of transformation is: “a thorough or dramatic change in form or appearance”. I believe this sums it up well. It is important to see how this transformation occurs. Leaders have to set the vision and everyone must change how they think. This requires a high degree of self awareness and patience. It is not something that is going to happen overnight and will take a few years of consistent practice to make this shift.
This is a challenge. This change must occur in each individual. Since we are all individuals, there will be some people who can make the mindset shift, and others who will not be able to adapt to the transformation. This can be quite painful and it will affect individuals differently. It is important that leadership is adequately trained on how to handle these situations. If people within the structure cannot change their thinking, then Agile will not be successfully implemented. In general from an IT perspective, when it comes to technology, change is the one constant. We are always required to learn new technical skills to keep up and evolve what is used to meet business objectives. Agile Transformation should not be any different.
4. Agile Transformation Requires Dedicated Product Teams
With the 3 previous steps in place, the next items will be more centric to the transformation at the team level. Agile requires dedicated teams which are initially established. This transformational change will usually result in organizational changes. Our organization went through some dramatic changes and it really helped to set the tone on how things operated. Without this structural change it would have been more challenging to adopt.
In my personal example, I was leading a QA team. The QA management which was previously in place went away and my role was changed to lead one of the product teams. Instead of managing QA resources, I led a team of a business analyst, developers, and QA resources. My role shifted into a Scrum Master but I also had HR responsibilities over the team which helped maintain control from both perspectives. In the previous organization there were agile teams, but there were PM Managers, BA Managers, Development Managers, and QA Managers. Each manager had their own criteria and definition of success which made it extremely difficult to operate in a true Agile model.
The IT support teams such as Architecture, Infrastructure, Database, and Service Desk which would support the product teams as needed. Ideally, there should be some consistency in this area where the supporting resources can be the same ones who support the teams. Especially with something like Architecture. You need the same expertise from a technology stack perspective to build consistency. For Infrastructure and Database support it is not as critical. If there are anticipated changes needed in upcoming sprints, it is always a good idea to give them a heads up, so they can allocate resources for the work. It isn’t fair to them to open a ticket and tell them that work must be done today in order to keep the sprint on schedule.
5. Agile Transformation Requires Planning
Planning is critical component of Agile. The first step is the creation of a Product Roadmap. This is needed so the Agile team will have guidance on what is needed in order to have a picture of what things will be needed by the business in the upcoming quarters. This requires involvement and participation from the business. In our organization, the Product Owner worked to create that roadmap with support from IT and the business. Our organization had the Product Owners as a part of IT, but those could have easily been structured within the Business. In several of our product teams, the Product Owners were in the business in the previous organization. This helped to build consistency.
Our organization was embracing Agile, but we were also transforming our IT systems. This was a combination of brand new applications, previous systems from both companies, and plans to decommission those redundant systems. I have also been a part of transformations where large systems integrators come in to mostly waterfall organizations and leverage Agile to transform IT systems.
Product Roadmap
Product Roadmaps generally change over time. Business priorities change with changes in the marketplace, and that often results in roadmap changes. Initially we started in laying out a 3 month roadmap and that grew over time to where we had a 6 month roadmap in place. In general, anything over 6 months will be a stretch and it is too far out to accurately predict what might happen. Product Roadmaps are important because they help the team know what activities need to be completed, and it can help reduce technical debt.
Backlog Grooming
Once the Product Roadmap is established, the Product Owner can work with the product team to start building the backlog. We met with the Product Owner twice a week initially in order to get the backlog built up and then we reduced that time to 1 time per week once we had sufficient items to work on. Planning must be a constant activity.
Estimation
Estimation is a critical component of the planning process and must be done as a part of Backlog grooming. During this critical phase, the team is going to need some outside support from an Agile coach, especially if the team is new and has never estimated using the Agile approach. Once the Backlog had a few stories, we began learning about the estimation process. We used the Fibonacci approach to estimate our work. This include the following story points: 1, 2, 3, 5, 8, and 13. If we initially estimated the user story to be over 13 points, then we would break down that user story in to smaller user stories.
Planning poker was used as a part of our estimating process. We had the majority of the team in one central location. Those resources sat together. That is an ideal situation, but often not realistic. We had some additional resources who were located in a separate time zone and other resources who were offshore. This created some challenges for the team as a whole, but we were able to refine the process over time. During the Backlog planning session we used planning poker cards in the room and video conferencing enabled those offsite resources to also provide their estimates. At first, we were all over the place with our estimates, but over time, as a team we became more consistent. We had a process in place where the Agile team would each provide their own individual story point estimate, and then we would dialog until we could agree on the same point value. This process was done over and over again during our backlog sessions to ensure we felt comfortable with the estimate and would slot that into an upcoming planned release.
Definition of Done
Another aspect of planning which we created was defining our Definition of Done. We initially created our first version, and determined what items would need to be completed before we could formally close out the sprint. We were able to leverage our Agile coach and Product Owner who helped us determine what was needed. For each Agile team, your Definition of Done will be a little different. Some of the items on our list included: development code complete, code peer review, test execution, product demonstration, and no critical or high defects. We would review this list at the beginning and end of each sprint to determine if the team met their objectives. Over a period of time this was refined, but didn’t change a whole lot from sprint to sprint.
For our first sprint, we took at the Product Backlog and determined how many user stories we could implement. We started off slow, because we needed to learn how the overall process worked and wanted to successfully complete what we had planned. During the Sprint 1 planning session, we determined what our capacity would be and accounted for any planned PTO as a part of the capacity and took that time out. There will always be other unplanned activity such as unplanned leave, production support, and support of other product dependencies that will come up. This needs to be considered. For our team, we also supported to products in production, so we generally allocate one resource for that effort. We then reviewed the user stories and story points again and agreed to go ahead and start Sprint 1.
6. Agile Transformation has 4 Critical Events
The Agile process has 4 critical ceremonies or events that must be consistently done. Those 4 events are Sprint Planning, Daily Scrum, Sprint Review, and Retrospective.
Sprint Planning
The first event is Sprint Planning. This occurs at the beginning of the sprint. The Scrum Master is responsible for facilitating the Sprint Planning session. The Product Owner is also involved and helps to answer any questions that the team has regarding the planned User Stories. At that point the Agile team determines how much work they can absorb during the Sprint.
Daily Scrum
The Daily Scrum is a meeting that lasts for 15 minutes or less. Each Agile team member is responsible for providing an update on what they worked on the previous day, planned work for the current day, and any roadblocks they need help with resolution. The Scrum Master and Product Owner can also participate, but can be optional attendees.
Sprint Review
The Sprint Review is scheduled prior to the completion of the sprint which is usually the last few days. This meeting includes the Scrum Master, Product Owner, Business Stakeholders, and the Agile Team. The Agile team reviews what was completed during the sprint and gets feedback from the Business Stakeholders on what adjustments might be needed for upcoming sprints.
Sprint Retrospective
The Sprint Retrospective is generally scheduled on the last day of the sprint. This meeting includes the Scrum Master, Product Owner, and Agile team. The purpose of the meeting is to determine what worked well within the sprint and what process improvements can be made. This meeting should be an open, and honest dialog between the group in order to get the most productivity from the team in future sprints. When we had our first few meetings, the team identified a list of 3 process improvement changes they would make in the next sprint. The team did well in getting the first 1 or 2 done, but often struggled to complete all 3 consistently. We adjusted expectations and agreed that we would take 1 of the items and focus on doing that really well. This change allowed the team to be more successful in the long run.
7. Agile Transformation Embraces Failure
Agile Transformation needs to allow the Agile team to fail and learn from those failures. It is especially important that leadership allows the team to fail and doesn’t punish the team when they do. There were many times where the team failed, and learned to make adjustments as they moved forward. There are two examples the team experienced where one was positive and the other was negative.
The first example involved a situation where we were starting to plan and prepare for Sprint 3. We had slowly increased our sprint points in Sprints 1 and 2 and the team was extremely confident they were willing to dramatically increase our capacity. We all agreed that we should give it a try since we had proven we had been successful so far and had a lot of work to get done over the next year in order to decommission applications. We went ahead and set the target and began our sprint. From what we had experienced to that point, things during weeks one and two were going to plan and our burndown was in line with expectations. It was not uncommon for us to see a dramatic decrease in week 3 due the fact that the more complex items in the sprint required longer to develop and test. However, in week 3 we didn’t get closure on several user stories so we weren’t able to meet the target. I applauded the team for their efforts and we regrouped and decreased the overall story points for the following sprints. This valuable lesson allowed the team to make some process improvements over the next several sprints, and then we will able to meet that goal we had set previously.
The second example involved a situation where we had a negative experience and failure wasn’t embraced. As a part of our system transformation, we used a brand new product. The team was learning it on their own, but we weren’t given the proper amount of training that was needed to learn all the features. We developed and tested a complex user story and deployed it into production. Everything was working properly, so we began focusing on the additional work. A few months later, one of our customers notified us they weren’t getting the information they needed from us, so we began to investigate. We ended up having to go back and provide the updated information to that customers and other customers which needed similar information. The root cause of the issue was a configuration setting was not set properly to resend the information if a transmission failure happened due to network communication issues. With the proper understanding of how the application we were using worked with training, we could have avoided this critical mistake. It was clearly a big miss and we treated it with the upmost importance, but the team was always reminded of that failure by upper management and we were consistently being asked to ensure everything was thoroughly tested before it was deployed, which impacted our overall velocity. The team lost confidence for a while in their capabilities with the tool and that had a negative experience for several sprints until we were able to regain the trust of upper management.
8. Agile Transformation Requires Consistent Process Improvements
I am a huge advocate of process. With a QA background, I understand how important having a defined and consistent process is. Without process, things are often unplanned, chaotic, and prone to failure. Some Agile experts will disagree with me on this point. I am not saying we have to have a bulky, slow process and that it is so inflexible that process changes cannot be made. I learned through the Agile Transformation process that processes can be lean and that process improvements are needed to gain additional velocity sprint over sprint. The Agile team moved from a very slow and methodical process where it would take projects 8 to 10 months or longer to deploy into production. This mindset change took time. It allowed us to start slowly adapting but we were able to consistently make changes that would make us faster as a team, while still having consistent and predictable software releases. Once simple adjustment on a process can dramatically change how teams operate. With gradual changes over a year, we were able to double our sprint velocity and consistently deliver on our sprint goals.
9. Agile Transformation Requires Team Synergy
In the Agile model, one of the most important elements is the importance of the team and how they operate as a single unit. When the initial team was setup, there were team members from both sides of the previous organizations including an offshore component. The team didn’t know what each other’s capabilities were, so it was challenging figuring out what the team could do as a whole unit. This took time. The beauty of this Agile transformation was that we learned as a team what each person’s individual strong points were and we focused on those to maximize the overall output of the team. While we had team members with certain defined roles, we were flexible to adapt when needed. For example, if the Business Analyst was done with their work, they would help the testers on the team. This greatly helped to eliminate the silo mentality and everyone was willing to pitch in and help for the good of the team. High functioning Agile teams need to work together and stay together. In the previous organization, that would have not happened, and distinct teams would have pointed out the other teams weaknesses and failures. In addition, after the project was put into production, that Agile group would be disbanded and the next project would spin up a brand new team with different resources and it took time for the teams to build that synergy. With Agile if you do either of those you will fail.
10. Agile Transformation Requires Flexibility
One of the most important aspects of Agile Transformation is flexibility. Agile requires constant adjustments. Things move at a rapid pace with Agile. Priorities are constantly moving and thus the individual and the team needs to be flexible. Flexibility at times can be frustrating, because this might often result in rework. The teams who are the most flexible, will the the ones that are the most successful.
I hope this information has been helpful. Each Agile approach might be different and organizations use Agile in a different way. I encourage you to learn and grow and gain additional experience with the Agile Methodology.