Kanban, Scrum and Continuous Delivery: Where do they fit in?
Since the day it began, Scrum is well known for short term project delivery. The strength of Scrum is in its speed and flexibility, much like a sprint running. Within this philosophy, Scrum development process is even called Sprint.
On the other hand, the Continuous Delivery (CD) is known for its long distance running that keeps on producing the code on the pipeline through several “pit-stop” before it achieves the state of “releasable.” CD is well-known for its high-productivity.
The continuous delivery, as the name implies, produces the codes continuously, regardless the code will be implemented in the software or not. Meanwhile Scrum only produces the code which listed on the backlog (Scrum initial documentation).
Let us see how they fit in together and deliver the product that focus on customer as Scrum emphasizes, with the Continuous Delivery high productivity.
MAKE SCRUM WORK FOR DEVOPS AND OTHER CD TASKS
The idea of this post came after reading the following article:
The link above talks about Infrastructure and how Infrastructure tasks can fit into Scrum. The truth is that they probably don't.
Scrum focus on deliver value to user. Therefore, any work that does not deliver directly any value to the user, but is nonetheless key to a successful delivery of working software, might not fit in Scrum.
This is due to the Scrum backlog that requires a user story to be told and the development will run based on that backlog.
SPRINT COMMITMENTS TO AUTOMATION OF PROCESSES
In order to understand the issue, we need look back at the ideas of Sprints and commitments. It is a tool to allow the business to prioritize the user stories. And they get to know what will be deployed to Production in the next iteration. In other word, they know what value will be delivered to the business and the users.
We can question though the purpose of committing to sprints for works that do not deliver direct value to the business or the users. For example, automating the deployment of the component X is necessary but it is hard to assign a business value to the component, since component X is the value itself. So the only advantage to using Sprints, in this case, is to keep the illusion that we are keeping track of the progress.
SPRINTS ARE NOT FLEXIBLE ENOUGH WHEN THE SPRINT IS RUNNING
Another issue can arise: sprints are not flexible enough. As a matter of fact, they are not meant to be flexible. We only develop user stories that are in the Sprint, without add anything, nor remove anything.
However, sometimes, something goes wrong. For instance the deployment is broken, the testing framework has become too slow or a new process needs to be automated.
WHY PLANNING THE UNPLANNABLE ?
The issues listed above cannot always be planned during a planning session. There might be unplanned situations arise, for example:
We are not sure yet that we are going to introduce a new component during the Sprint, and the deployment of this component needs to be automated
Developer adds 100 new automated tests and suddenly the test suite is running too slow. It takes an hour to get the test reports which becomes a major bottleneck for the team (which relies 100% on the tests before releasing)
An automated deployment that use to work stops working. It seems a new version has broken the build, but nobody saw it coming
In some cases, we don't know yet what is going to come in the next 2 or 3 weeks at an automation and delivery level. Most often, it is not worth delaying any further and waiting for the next Sprint planning. The Continuous Delivery mechanism works like a car factory. If any of the operation of the assembly chain were to fail, the entire production machine is stalled. Why should you keep producing cars when you cannot put all the screws together?
In the TQM (Total Quality Management) which adopted as Lean Management in America, when the assembly chain noticed an error, there is only one solution: stop the production, fix the error and then resume your work. This is called Poke Yoke, which means error-proofing in Japanese language. The idea of this method is to stop the production when error is noticed, quickly fix the error before the error becomes the defect and defect lead to failure. Using this method, we are able to reduce the severity of error and reducing the amount of cost spent to correct the codes.
In software development, the implementation of Poke Yoke is: stop coding, fix the deployment and resume your work.
This Poke Yoke requires all code to be inspected before delivery to the next stage (pipeline) of Continuous Development. Normally the CD pipeline consists of: Coding, Unit Testing, Integration, Acceptance Testing and Deployment.
Note that it's not recommended to let developers write too much code when that same code cannot be tested. Nobody wants to have to do big-bang commit and testing once everybody resumes their work.
HOW DOES ONE PHRASES A "TECHNICAL USER STORY"
Another time when developers struggle is the Sprint planning and phrasing technical user stories. Some bad examples:
As a tester I want to make the test run faster so that I get faster feedback
As a sysadmin I want to refactor the script that automates the configuration of our web servers
User stories are great because they start with "as a user" and force you to think about what you are going to build for the user.
However, when user stories do not serve directly the user, you will have hard time writing it. This is the time to stop writing that kind of user stories and find solution to “technical user stories.”
"HIDE" THE WORK IN USER STORIES
The first option is to "hide" CI/Infrastructure tasks into other user stories.
Say we need to need to automate the deployment of the component X and Y. We can add a few more tasks in User Story X, and maybe a few more in the story Y.
Unfortunately, that approach does not always work:
you are wasting your time doing this
your story Y is not independent and is waiting for story X
Since component X and Y is not a user, which we cannot make a story out of component. Therefore for some cases, this might work, but not for others.
A CONTINUOUS TECHNICAL USER STORY
For another project we had a continuous technical user story. The challenge is to explain the Product Owner why your team needs to spend 10 points on a technical story that does not provide any business value.
Unless the Product Owner used to be a software engineer, you will have a hard time justifying it.
A CONTINUOUS DELIVERY SUPPORT TEAM
In a recent Enterprise size project, we ended up having a "project support team" for any Continuous Delivery work. The CI/Infrastructure/Data Migration effort was so huge; so that we had to separate it out from the main development stream.
In fact we were running a data and infrastructure migration project as well as building an asset search website.
That Continuous Delivery team was autonomous and would reprioritize almost daily, depending on the new issues and the long terms tasks. Their goal was to make it possible to test and release the software at all time. They understood the priorities: serve first, maintain second, and improve later. They need to justify the value of works they have been doing by some management tools.
Kanban is a Japanese term for “Sign-Board.” It is a visual management tools that started in the 1970’s world-famous Toyota Production System (TPS). Its main purpose is to view, track and monitor the process in the production line, This production line in CD is called pipeline. In Agile world, Scrum adopted Kanban as one of their artifacts: Taskboard.
In Kanban board, it divides the status of Work In Progress according to the stage of the work. Starting from the Planning to Execution. In Scrum, the stages are User Requirement – To Do – In Progress – To Verify – Done.
The Kanban board will show exactly the job to do and the workflow instantaneously. A Kanban board is useful for more adaptability when urgent situation arises. In a way, the CI/Infrastructure is there to support the developers. One has to be ready to jump in and take responsibility. That means to make the situation as his priority to fix and improve (like support engineers do for other business units).
Kanban helps limiting the Work In Progress and allows one to prioritize the work nonetheless. Thus, when the situations arise, we can add the situation into the Kanban board so every team member can see the situation, and anyone can take action upon the situation immediately.
SCRUM IS NOT SUFFICIENT BUT ...
Scrum seems easy to adopt for pure software development tasks that are serving users. However, when we are facing a complex situation, we propose to implement Kanban as solution to deal with such situations, such as the following:
you are running a very technical project like Data migration or Infrastructure transformation
you have automation tasks that are required but the tasks do not serve the user directly
you have very technical tasks and are having a hard time phrasing user stories
Depending on the size of the project or the team, Kanban might be of a great help. In any case don't try to use user stories when you are not building a feature, but rather supporting the delivery of the feature.