// Response Windows
When talking about bugs and issues, we talk about the relation between opened and resolved issues to give us general trends as to the overall quality of the project and the group effort being employed in the pursuit of shippable quality. The key being the overall numbers of issues as defined by urgency & severity from critical to minor. This gives a good overall impression of where the project currently stands.
However, in bug count trending there is no inspection of the particular issues, how long they may have been in existence or how badly they have affected the ability of the team to undertake their daily work. Triage lists are often created to target issues but the overall shape of the data may allow certain issues to lurk for long periods without being addressed.
What if the QA of games was approached from the perspective of security vulnerabilities in software and what different information would that give us about the state of a project?
‘The window of vulnerability is the time from when the security hole was introduced or manifested in deployed software, to when access was removed, a security fix was available/deployed, or the attacker was disabled.
‘Vulnerability management is the cyclical practice of identifying, classifying, remediating, and mitigating vulnerabilities” This practice generally refers to software vulnerabilities in computing systems.’
Here the key issue is not how many bugs exist but how long it took for the bug to be addressed, giving an overall impression of the stability of the product and the responsiveness of the team in addressing issues. This may be a useful metric on the day to day usability of the product and the immediate quality of features.
Response windows may also form a useful guide as to the relative efficiency of the production pipeline. How long does it take the team to get a feature turned around? How long does a build or feature go without being in a workable state? How long do niggling issues persist without being dealt with? How good are individual teams at maintaining existing features? How quickly can the team get a build together?
If teams were to focus on the responsiveness of their ability to iterate content and address issues, how would that affect the perceived quality of the product and the developer’s willingness to change for the better in the opinion of their peers and customers?
Q – What might the data look like?
- Modelling the data as a distribution should match a normally distributed bell curve.
- The Response Time is the total time from the issue being introduced to being resolved.
- The Response Window is the area underneath the curve defining the time in which issues are addressed.
- Urgent bugs would hopefully trend to the left hand side of the distribution with a sharp peak and decay
- The lower the priority of issues the further to the right and flatter the expected peak.
Example Response Window graph for bugs (x= response time, y = number of issues):
Q – What would good news look like?
- Really urgent issues would be dealt with in a very short number of hours with very little tail to the distribution curve.
- Higher priority issues would be dealt with before lower priority issues.
- Not all bugs are urgent.
- Not all bugs are minimal.
Q – What would bad news look like?
- Urgent issues being further to the right of the graph than lower priority issues.
Q – What further data might be extracted?
- The time to triage, fix, test, & resolve issues
- Per component, strike team, feature or build data
Q – What actions could be taken to improve the response?
- Try to focus on more urgent issues to reduce the Response Window.
- Try and make the response windows as short as possible.
- Try to improve workflows or ramp up resourcing for the stages of the pipeline identified as the biggest contributors to the Response Time.