Do we really need task estimation in Scrum

I came accross this question on the LinkedIn Scrum Practitioner group, asked by one of the Certified Scrum Master. (http://www.linkedin.com/groups?home=&gid=52030)

Why we have two level of estimation in scrum? What purpose do task estimates serve?

This topics pinch me as I've been pondering about that for a little while:

why do we do tasks estimation and is it a good idea ?

COMPARIN FOCUS FACTOR AND TASK ESTIMATION

I also came accross this blog post recently, from one of my acquantainces:http://www.agilebuddha.com/agile/how-to-do-effective-capacity-planning-on-the-scrum-team/

The author explains the process of calculating team capacity for a given Sprint using Focus Factor, instead of total number of hours available.

For 5 people working 8 hours a day for 2 weeks :

That means the total working hours will be 5x8x10 = 400 hours!

Estimated planning for this capacity will be a disaster. It will lead to team working over time, rushing towards the end, quality cuts and low team morale.

Now let us take into account the Focus Factor of 6 hours per day:

Traditionally, project managers used 6-6.5 hours as planned hours in a day for project execution. Focus factor is team ability to remain focussed on the sprint goals without any other distractions.

The idea here is to sum up all the hours and find out how big the user story is.

Based on the capacity calculated above, you find out how many tasks and user stories can fit into a given the Sprint. Except that such estimations are far from accurate. On top of this, using tasks estimation instead of user stories does not make it more reliable.

Now come the question: Is estimating tasks in hours a good idea at all?

PROS AND CONS FOR TASK ESTIMATIONS

Let's see what people on the Scrum LinkedIn Practitioner group have to say.

PROS:

So here is my top list of arguments for those in favor of tasks estimations:

• Task level estimation is used in the daily burndown. It is a great way to show the team where they are each day. It may take more than a day to complete a user story and there isn't a way to see the progress on that story unless you track the task hours.
• Tasking helps the team to plan the actual work and provides transparency.
• Task estimation allows progress tracking, and supports decision making. Will the stories really fit in the iteration (i.e. a sanity check)?
• When it comes to sprints, burning down story points may lead to very large steps where it appears nothing is done, then suddenly a large drop. This frightens people and makes them feel uneasy as generally people want to know if they are somewhat on track.
• If the developers are highly experienced and mature as an agile team, they can get away with relative sizing using story points. But as this is seldom the case (most teams consisting of mixed abilities/experience levels), task breakdown is welcome.
• From a team perspective, the main reason to have estimates is to be able to detect when things start going off track. It tells us that we need to look and possibly make a decision to go back on track

CONS:

And here is my top list of arguments against tasks estimations:

• Dual estimation using points and time is overkill (most of the time). You have to keep in mind that both of them are estimates, thus a guess on how big or much time/effort will be taken up with them.
• The major risk of doing task estimation is it leads to falsify direct relationship, that initiate people to start to see between story points and time (time sum of all tasks = story point value of story).
• Do we already have a time measurement in scrum: the sprint timebox. In order to find how many story points can a sprint fit in, we have to find the answer by doing, not by guessing
• In order to start a sprint you only need enough work for a few days in advance. You don't need to break down everything from the beginning
• There's a reason many scrum masters forego the time estimations (on any level): they are imprecise. If your story point estimation is not useful enough to draw a reliable velocity, then definitely the time estimates won't provide more details.
• As a very rough indicator I would say each person in the team should be completing 2-4 items a day and marking them as done. More than that is too much administration which will slow the team down, and completing items less than that is too big and does not facilitate good communication and getting stuff done.
• The Scrum Guide never uses the word task. It does use the word estimate in combination with product backlog items. So yes, according to the Scrum Guide there is estimation of Sprint Backlog Items
• Scrum actually does not define how or when estimation should be done, but merely state that there is an estimation.
• The more time you spend on estimation, the more it highlights the dysfunction of pressure which results in technical debt.

From the above, we cannot be convinced estimating tasks is something developers should systematically do. Let's see if there are alternatives solutions to support story sizing and sprint forecasting.

ALTERNATIVES OPINIONS

Here are some ideas, presented by LinkedInders that can be taken into consideration:

• What I favour doing is breaking tasks into similar size, approximately half-day to day-long chunks (day long is definitely the upper limit IMHO). Then I look at burn-down by counting the number of tasks yet to complete. I don't care whether they are 4 hour or 8 hour; I don't even want to try and estimate to that level. It all averages out.
• If you have a lot of very small stories where each one takes a couple hours and there are not many tasks on each, then by all means burn down the stories. If however, you have only a few stories and lots of task, burn down the count of tasks.
• Task estimates are about developers planning technical work. It is less than an out-facing estimate, and more a way to plan how to approach each task, what would "normal time spent", and when to go for help.
• Story sizing is not really about creating estimates. Ideal days, story points and relative sizes do not directly translate into hours and days. These are more for creating an understanding for the PO and the rest of the organization how much work each piece of the backlog entails.
• We live, eat and breath by the clock and estimating the time is so ingrained in us, that we naturally want to know and predict the future, I would say the majority of people and as I count myself here, I cannot resist estimating in some form or another, even if it is a 2 second gut feeling.

MY PERSONAL OPINION

Below is what I answered to my peers on the topic. This is what I learned from 3 years doing tasks estimates:

In my opinion, when we do task estimation, we tend to put too much power into the estimations. The maths are so easy that we start believing in them: 4 tasks of 2 hours + 3 tasks of 4 hours + 1 task of 8 hours. = 28 hours! Easy, right ? No… Most of the times it never happen , even when we try to use buffer techniques it still never works out

And the PO or SM to say:

so that fits into the sprint, right ? if we look at the hours...

The answer to the question is not. Let us answer the question regarding the team: do you believe you and the team are able to achieve all this into a sprint ? Put away the hours and tasks, let us use our gut feeling. We will know that the sprint length is the only valid time period.

Although I must say breaking down a story into tasks has been a useful exercise to brainstorm and think about the solution. It might not be something we want to do all together though. Another approach is for a couple of developers to spend a couple of hours doing solution design.

I've been doing tasks estimations for so long (3 years) and I've stopped believing in it. It is often a waste of developer's time, and leads to frustration:

is it 3 hours, is it 2.5 hours ? or maybe 3.5 hours ???

Ok, let's simplify, only 4 and 8 hours tasks are allowed. Even that does not feel right: we ask hours estimations and then start to round up and put constraints on what should be a very simple task.

One thing I understood: if you depend on task estimations, maybe your user stories are too big and need to be broken down.

In the end I recommend no more than 5 user stories sizes. Something like:

• small
• large
• xlarge
• unknown : might lead to a spike for investigation

DON'T ALWAYS DO TASK ESTIMATIONS, INSTEAD BREAK DOWN YOUR USER STORIES INTO SMALLER CHUNKS

Here are, my take on the topics above based on my experience with task estimation:

• Do task break down as a team if it helps brainstorming and designing the solution
• Do not systematically do task estimation (putting too much power into task estimations might actually lead to technical debt and lower quality)
• Use small user stories, not epics: a user story is an entity that can be tested independently. For example, replace "As a user I want to manage products" with 3 stories: "as a user I want to be able to create products" + "as a user I want to be able to edit products" + " as a user I want to be able to delete products"

What about you, would you use task estimation ?

Kanban, Scrum and Continuous Delivery: Where do they fit in?

Since the day it began, Scrum is well known for short term project delivery. The strength of Scrum is in its speed and flexibility, much like a sprint running. Within this philosophy, Scrum development process is even called Sprint.

On the other hand, the Continuous Delivery (CD) is known for its long distance running that keeps on producing the code on the pipeline through several “pit-stop” before it achieves the state of “releasable.” CD is well-known for its high-productivity.

The continuous delivery, as the name implies, produces the codes continuously, regardless the code will be implemented in the software or not. Meanwhile Scrum only produces the code which listed on the backlog (Scrum initial documentation).

Let us see how they fit in together and deliver the product that focus on customer as Scrum emphasizes, with the Continuous Delivery high productivity.

MAKE SCRUM WORK FOR DEVOPS AND OTHER CD TASKS

The idea of this post came after reading the following article:

"How does a Scrum team account for infrastructure tasks in the planning meeting?"http://programmers.stackexchange.com/questions/61529/how-does-a-scrum-team-account-for-infrastructure-tasks-in-the-planning-meeting

WHERE IS THE BUSINESS VALUE ?

The link above talks about Infrastructure and how Infrastructure tasks can fit into Scrum. The truth is that they probably don't. 

Scrum focus on deliver value to user. Therefore, any work that does not deliver directly any value to the user, but is nonetheless key to a successful delivery of working software, might not fit in Scrum.

This is due to the Scrum backlog that requires a user story to be told and the development will run based on that backlog.

SPRINT COMMITMENTS TO AUTOMATION OF PROCESSES

In order to understand the issue, we need look back at the ideas of Sprints and commitments. It is a tool to allow the business to prioritize the user stories. And they get to know what will be deployed to Production in the next iteration. In other word, they know what value will be delivered to the business and the users.

We can question though the purpose of committing to sprints for works that do not deliver direct value to the business or the users. 
For example, automating the deployment of the component X is necessary but it is hard to assign a business value to the component, since component X is the value itself. 
So the only advantage to using Sprints, in this case, is to keep the illusion that we are keeping track of the progress.

SPRINTS ARE NOT FLEXIBLE ENOUGH WHEN THE SPRINT IS RUNNING

Another issue can arise: sprints are not flexible enough. As a matter of fact, they are not meant to be flexible. We only develop user stories that are in the Sprint, without add anything, nor remove anything. 

However, sometimes, something goes wrong. For instance the deployment is broken, the testing framework has become too slow or a new process needs to be automated.

WHY PLANNING THE UNPLANNABLE ?

The issues listed above cannot always be planned during a planning session. There might be unplanned situations arise, for example:

• We are not sure yet that we are going to introduce a new component during the Sprint, and the deployment of this component needs to be automated
• Developer adds 100 new automated tests and suddenly the test suite is running too slow. It takes an hour to get the test reports which becomes a major bottleneck for the team (which relies 100% on the tests before releasing)
• An automated deployment that use to work stops working. It seems a new version has broken the build, but nobody saw it coming

In some cases, we don't know yet what is going to come in the next 2 or 3 weeks at an automation and delivery level. Most often, it is not worth delaying any further and waiting for the next Sprint planning. 
The Continuous Delivery mechanism works like a car factory. If any of the operation of the assembly chain were to fail, the entire production machine is stalled. Why should you keep producing cars when you cannot put all the screws together?

In the TQM (Total Quality Management) which adopted as Lean Management in America, when the assembly chain noticed an error, there is only one solution: stop the production, fix the error and then resume your work. This is called Poke Yoke, which means error-proofing in Japanese language. The idea of this method is to stop the production when error is noticed, quickly fix the error before the error becomes the defect and defect lead to failure. Using this method, we are able to reduce the severity of error and reducing the amount of cost spent to correct the codes.

In software development, the implementation of Poke Yoke is: stop coding, fix the deployment and resume your work. 

This Poke Yoke requires all code to be inspected before delivery to the next stage (pipeline) of Continuous Development. Normally the CD pipeline consists of: Coding, Unit Testing, Integration, Acceptance Testing and Deployment.

Note that it's not recommended to let developers write too much code when that same code cannot be tested. Nobody wants to have to do big-bang commit and testing once everybody resumes their work.

HOW DOES ONE PHRASES A "TECHNICAL USER STORY"

Another time when developers struggle is the Sprint planning and phrasing technical user stories.
Some bad examples:

• As a tester I want to make the test run faster so that I get faster feedback
• As a sysadmin I want to refactor the script that automates the configuration of our web servers

User stories are great because they start with "as a user" and force you to think about what you are going to build for the user.

However, when user stories do not serve directly the user, you will have hard time writing it. This is the time to stop writing that kind of user stories and find solution to “technical user stories.”

POSSIBLE SOLUTIONS

"HIDE" THE WORK IN USER STORIES

The first option is to "hide" CI/Infrastructure tasks into other user stories.

Say we need to need to automate the deployment of the component X and Y. We can add a few more tasks in User Story X, and maybe a few more in the story Y.

Unfortunately, that approach does not always work:

• you are wasting your time doing this
• your story Y is not independent and is waiting for story X

Since component X and Y is not a user, which we cannot make a story out of component. Therefore for some cases, this might work, but not for others.

A CONTINUOUS TECHNICAL USER STORY

For another project we had a continuous technical user story. The challenge is to explain the Product Owner why your team needs to spend 10 points on a technical story that does not provide any business value.

Unless the Product Owner used to be a software engineer, you will have a hard time justifying it.

A CONTINUOUS DELIVERY SUPPORT TEAM

In a recent Enterprise size project, we ended up having a "project support team" for any Continuous Delivery work. The CI/Infrastructure/Data Migration effort was so huge; so that we had to separate it out from the main development stream.

In fact we were running a data and infrastructure migration project as well as building an asset search website.

That Continuous Delivery team was autonomous and would reprioritize almost daily, depending on the new issues and the long terms tasks. Their goal was to make it possible to test and release the software at all time. They understood the priorities: serve first, maintain second, and improve later. They need to justify the value of works they have been doing by some management tools.

KANBAN

Kanban is a Japanese term for “Sign-Board.” It is a visual management tools that started in the 1970’s world-famous Toyota Production System (TPS). Its main purpose is to view, track and monitor the process in the production line, This production line in CD is called pipeline. In Agile world, Scrum adopted Kanban as one of their artifacts: Taskboard.

In Kanban board, it divides the status of Work In Progress according to the stage of the work. Starting from the Planning to Execution. In Scrum, the stages are User Requirement – To Do – In Progress – To Verify – Done.

The Kanban board will show exactly the job to do and the workflow instantaneously. A Kanban board is useful for more adaptability when urgent situation arises. In a way, the CI/Infrastructure is there to support the developers. One has to be ready to jump in and take responsibility. That means to make the situation as his priority to fix and improve (like support engineers do for other business units).

Kanban helps limiting the Work In Progress and allows one to prioritize the work nonetheless. Thus, when the situations arise, we can add the situation into the Kanban board so every team member can see the situation, and anyone can take action upon the situation immediately.

SCRUM IS NOT SUFFICIENT BUT ...

Scrum seems easy to adopt for pure software development tasks that are serving users. However, when we are facing a complex situation, we propose to implement Kanban as solution to deal with such situations, such as the following:

• you are running a very technical project like Data migration or Infrastructure transformation
• you have automation tasks that are required but the tasks do not serve the user directly
• you have very technical tasks and are having a hard time phrasing user stories

Depending on the size of the project or the team, Kanban might be of a great help. In any case don't try to use user stories when you are not building a feature, but rather supporting the delivery of the feature.