Last year, I was tasked with creating a roadmap to raise the DevOps Maturity of our product development teams.
I needed to assess how far we went in our transformation journey and decide what more to accomplish.
Therefore I needed a tool or methodology to perform the evaluation and gather feedback from everyone.
With my Industrial Engineering background, I decided to repurpose a Lean Manufacturing methodology that enabled me to complete the task.
Lean Six Sigma practitioners have been working on a toolbox to evaluate the adoption of their practices for the past three decades. Without a doubt some lessons to learn from that.
This article aims to revisit the methodology called SMART Maturity Assessment to create a similar tool that would facilitate tracking a company or team's progress through their digital transformation journey and adoption of DevOps culture.
I will walk you through the steps that lead you to evaluate your organization's maturity and create a roadmap to advance further in your journey.
More About The SMART Maturity Assessment Methodology
The activity I recommend follows the Improvement Kata Model.
SMART Maturity Assessment is just a tool you will use in your Kata's planning phase (see illustration below). As the person who leads the transformation, your mindset should be the following:
- First, Identify core practices in the establishment of the DevOps methodology. It should help you understand the direction or challenges that need to be taken into consideration.
- Second, Use a methodology (SMART) to grasp the current situation and establish the following targets that would best benefit the company by improving its maturity in some core practices.
More About SMART Itself
The manufacturing world was facing the challenge of creating a “satisfactory and ongoing trajectory towards an effective application of Lean principles.”
Consequently, a novel Maturity Model and assessment tool called SMART has been developed as part of a Lean Transformation Toolkit (Lean T2) within the LeanPPD project funded by the European Commission (NMP-2008–214090).
SMART is an acronym for the five levels of maturity used in the methodology: Start, Motivate, Apply, Review, Transform.
The levels of the SMART Maturity Model and Assessment adapted to DevOps are the following:
- Level 1 (Start): The organization does not implement DevOps practices in its product development.
- Level 2 (Motivate): The organization is aware of the DevOps practices and is starting to implement them.
- Level 3 (Apply): The organization applies DevOps practices in its product and process development and has achieved a basic implementation level.
- Level 4 (Review and Improve): The company is applying DevOps practices in its product and process development and continuously measures its results to implement improvement actions.
- Level 5 (Transform): The organization applies DevOps best practices and shares them internally and externally along the complete value chain.
The result of the evaluation can be easily represented as a Radar Chart. With time and effort, you should see the area covered in the diagram grow. It is a great way to keep track of your progressions over time.
When Should I Use SMART?
I recommend using this evaluation if you have been working on your digital transformation for at least a year.
If you are looking for an initial evaluation, there are better models you can find. They should offer you a set of specific questions to answer and a list of actions or directions to start with, leading to a better understanding of the DevOps culture.
The way I see this, this version is more like a retrospective exercise at the organization level that provides not only an evaluation but also specific insight proper to your business that you can work to improve. This is in the spirit of the Improvement Kata Model.
SMART in the DevOps World
The original SMART tool in the Lean T2 toolkit evaluates 14 lean practices under four perspectives.
Thus, the assessment integrates the results of 56 qualitative Lean approaches – the four perspectives being Time, Cost, and Quality Measurement (1), New Product Development Process (2), Tools (3), and Skilled People (4).
Similarly, we can identify any number of practices in the DevOps world to feed our SMART tool.
Steve Smith introduces in his talk 15 techniques involved in the transformation of continuous delivery that I took as a reference in my first implementation of SMART.
I personally stopped at those 15 items and dropped the four perspectives mentioned in the original method.
I did not judge them as relevant in my context because it would make the evaluation process heavier. This is the list of items I have evaluated:
- Version Control
- Automated configuration
- Automated infrastructure
- Deployment pipeline
- Branching Model
- Continuous integration
- Evolutionary architecture
- database migration
- test-driven development
- Acceptance Tests
- Smoke Test
- Exploratory Testing
- Non-functional test
- Dynamic test data
- Monitoring and alerting
- Canary deployment and Feature Toggling
As the evaluator (i.e. DevOps evangelist), it is not your role to determine the teams' maturity level.
It will be better if you aimed to increase adoption and ease the change management process.
Still, you need to lead the improvement Kata, meaning that you are an observer that will organize interviews and workshops to grasp the current situation.
You are accompanying everyone like a gentle wind pushing the boat in the right direction. Proceeding that way will better include your colleges in the journey.
Implementing SMART Maturity Assessment
Step 1: Team Lead and Product Owner Interviews
In the first phase, gathering initial feedback is essential.
It is the catalyst for improving people’s mindsets and empowering them with new techniques. Tech lead most likely already have an opinion on the improvements they would like to implement.
On the other hand, Product Owners need to be aware of product development’s best practices. They also need to understand why they should dedicate some development time to improve some technical aspects. They need to understand the business value of the proposed practices in the product development cycle.
It is your chance to bring the business, the development, and the operation on the same page.
During the interview:
- Explain what SMART Maturity Assessment is.
- Review each of the 15 items and try to place them in each of the maturity levels (1 to 5).
- Each time, open the conversation to gain as much insight as possible. Remember the planning step of the Improvement Kata model.
- Understand the direction or change . What is Canary deployment? What is the value of it?
- Grasp the current situation . What do we do concerning Version control? What can we improve? Is there a blocker that stops us from implementing improvements? Do we need to do more research on the topic to understand the best practices on the market?
- Establish the following target condition . Do we have an interest in getting better on that item? Do we have a use case for it? Do we have a project that would immediately benefit from it?
Don’t be afraid of admitting that you are at a maturity level 1 (Start).
Step 2: Team Retrospective
This exercise will help us gather feedback from everyone and have a more accurate evaluation of the maturity. Your Tech lead opinions may not reflect the team buy-in for some practices, or the knowledge among teams or across teams may be heterogeneous.
Therefore, the more people you involve in the process, the more accurate your evaluation will be.
1) I organized this exercise as a retrospective exercise. An excellent way to introduce the activity would be to highjack one of the retrospectives for each team. So get your post-it ready! (or a miro board during Covid time).
You do not need to introduce SMART yet. The exercise below approaches the assessment from a different angle to gather the maximum number of insights.
2) Create 3 cards or post-its for each of the 15 items – so 45 cards in total. We want to duplicate items so several team members can express themselves on the same topic.
Based on the previous step, do not hesitate to remove an item you identified as level 1. Those items may create confusion among your participants, and they will not provide meaningful insight.
3) Distribute your deck of the card so that everyone has 3 to 4 cards. Giving more cards would result in a prolonged exercise. You can count on the fact that we duplicated each card and that you will repeat the exercise with several teams to converge to a clear picture of the situation.
4) Ask the participant to write comments on a post-it for each item they have been given — following these rules:
- You need to write at least 3 post-its per card.
- You must write positive feedback on a green post-it.
- You must write negative feedback on a red post-it.
- If you know very little about the card, you can write a single post-it that says you don’t know anything about it or we did not implement anything yet.
Set a 10 min timer for people to create the post-its.
The participant’s post-its should look like this:
5) Draw the following figure on the board. We are going to try to evaluate our 15 items on a scale from unrefined as charcoal to shiny as diamond.
6) In turn, ask the participants to stick cards and post-its on the board. Ask them:
Do we shine like diamond or are we unrefined like coal?
As a general rule:
- If you mainly wrote red post-its (negative feedback), you should be closer to the pile of charcoal.
- If you mainly wrote green post-its (positive feedback), you should be closer to the diamond.
7) Discuss the items with the group when laying down the post-its. This is the time to have group conversations and take some notes.
8) You can now explain what is SMART Maturity Assessment and tell participants what you will do with all their insight (keep reading, so I tell you what to do with all the post-it collected). A good conclusion would be to create the Radar Chart for this activity:
- Charcoal count as level 1
- Diamon count as level 3
- Everything else counts as level 2
Step 3: Compiling the Results
Create a table in your favorite tool (Confluence, Excel, or others).
Compile the maturity score for the interview and retrospective exercise. Create an average score across your organization; this is your maturity level from which you will create the final Radar Chart.
Report and compile all the insight collected alongside the SMART Maturity Assessment score. They will be helpful to develop your road map.
Why do we score items between 1 and 3 in Step 2: team retrospective exercise and not between 1 and 5?
Since it is the first time you do the exercise, we can assume that you did not review any process. Remember that to reach level 4, you need to measure results to implement improvement actions continuously. It may be the case, and if some member mentioned in their post-it that you have KPI ready for that item, then go ahead of your earn your level 4 maturity level for that item (Congratulations).
Step 4: Deciding on a Road Map
Creating the road map base on SMART is as simple as asking: How do I want my Radar Chart to look like in a year from now?
The levers to achieve and sustain changes are based on the maturity level:
- Moving from Start to Motivated involves making a presentation, creating a workshop to promote best practices associated with an item. It could also mean you need to encourage professional training, conferences, or searching for specific skills in future recruitment.
- Moving from Motivated to Applied requires some Epic or Stories to implement changes.
- Moving from Applied to Reviewed forces you to implement KPIs, document the process, dedicate time to review the tools in place continuously.
- Moving from Reviewed to Transformed symbolizes that you have established a “Bible” of how you do X in company Y.
Final Thought
I noticed a lot of interest in the DevOps world for repurposing lean manufacturing toolbox. VSM (Value Stream Mapping) is very popular at the moment. This eagerness for “lean” tools is the main reason that leads me to share my experience with another tool. On top of that, I believe that this tool provides a strong structure that drives the change management process.
Good luck!
