Ubuntu Quality – Key Performance Indicators
Recently there has been a huge push for quality in Ubuntu. The goal is to have a stable and usable daily image of the development version of Ubuntu. To do this, the QA teams have spent a lot of time writing tests and manually testing everything in Ubuntu to ensure this high level quality. The QA Team is also developing the QA Dashboard to display all of the results from the testing that is being performed. The goal of the QA Dashboard is to start at high level overviews of testing results, and drill down to the small details that make up the overall big picture to give users and developers the ability to quickly and easily be able to identify that a problem exists, determine the cause of the problem and then fix the problem. Certain tests or types of tests are part of what we call ‘key performance indicators’ or KPI. These key performance indicators will give someone a very quick overview of the quality of today’s Ubuntu image.
Of the types of testing that is being done, the smoke testing is the most important one. Smoke testing is the most basic level of what to test. Essentially, does it install and after it installs does it work? If the image doesn’t install, none of the other testing can be completed. Infact, the daily builds part of cdimage now has a current and a pending. Current is the most recent image to pass the automated smoke testing whereas pending has not yet passed smoke testing. Because smoke test results are so important, it is now listed as a key performance indicator on the QA Dashboard. The Smoke Testing KPI shows each release that is currently being tested as well as the pass rate for the latest image of that release as well as which arches are being tests. It also will show you the number of bugs effecting a release by hovering over the pass rate.
The other test that is currently listed as a key performance indicator is Eventstat Wakeups testing. Essentially, this test is measuring the number of wakeup events each process has. The reason this is important is each wakeup prevents the kernel from going into low power mode, or idling. With the push towards cell phone and tablet devices battery life is a big deal. It is important that power consumption is properly measured and understood in order to make an operating system that performs well on these small factor devices which have a limited power source. The eventstat testing tells developers which processes are causing too many wakeups, allowing the developers to look into the root cause of all of the wakeup events. This KPI shows each machine that the tests are run on and the average number of total wakeups for all processes on that machine as well as the percent that it is higher or lower than the last image.
There are quite a few other types of power related tests that will hopefully soon be added as key performance indicators for the overall quality of Ubuntu. We have sort term goals of adding bootspeed testing results and memory usage as key performance indicators. If you have any questions about the QA Dashboard, feel free to jump into #ubuntu-quality on Freenode and ask away.