Risk Based Testing

Risk-based testing (RBT) is a methodology that can be described as prioritization of various requirements on the basis of the risk they carry and testing them according to priority of the risk. Risk is always associated with two terms- Impact and Likelihood. RBT plays a pivotal role when testing effectively keeping time and resources are stretched thin.
What is Risk and how it impacts software projects: Risk, in general term, can be defined as an uncertain damage or loss that is probable to occur in future. To be more specific Risk is probability or threat of a damage, injury, liability, loss, or other negative occurrence, caused by external or internal vulnerabilities and that may be neutralized through pre-mediated action. The Software Development Process has three major criteria or limiting factors- Budget, Time and Requirements.
A successful software project must satisfy all user requirements within estimated time and budget. Any uncertainty or possibility of loss may result in non conformance of any of these key factors, leading to overtime / over-budget or poor quality project.Software risks, which impact above three key factors, can be broadly categorized as:

• Requirement Risks: Lack of requirement analysis, requirements that are not in-line with customers’ needs, poorly defined requirements, ambiguous requirements, inadequate requirements, invalid or impossible requirements etc.
• Technical Risks: Complexity of architecture, technology change, configuration change, inadequate technical support/ knowledge, lack of domain knowledge etc.
• Schedule Risks: Insufficient budget, unrealistic time line, inadequate skilled resources, improper resource planning etc.

Why RBT is needed: We can term risk as an unwanted event that has unfavorable or negative consequences. In order to negate them or make them less severe, there occurs a need to identify various dimensions of software project risk and to build and test a model guided by theory that relates the risk dimensions to project performance. This gave rise to the need of Risk Based Testing.
Potential reason for adopting RBT can be the aggregated impact of limited resources – time, budget and human. On one hand, the project’s calendar doesn’t allow sufficient time for a thorough testing of all functions and on the other hand budget limits the number of skilled human / software resources. In such a scenario, test coverage of all minute part of application would not be possible. To overcome or balance this situation, focus should be put on those areas that represent the largest risk, if a fault occurred.
 Before we start discussing Risk Based Testing process, we must understand few basic concepts that altogether create a base for RBT.
1. Pareto Principle: In software, roughly 80% of the defects come from 20% of the LOC (Lines of code). This principle allows testers to concentrate their testing effort in this specific area (20%) where there are chances to get most of the deviations (defects). This is the best guideline to achieve respectable testing coverage within limited time, budget and resources.
2. Quality Attributes: Software quality attributes can be considered as the benchmarks that describe application’s intended behavior within the configuration / environment for which it was built. The quality attributes provide the measurement of the fitness and suitability of a product. Some of the quality attributes are: functionality, reliability, usability, efficiency, maintainability, portability.
3. Prioritizing Requirement on the basis of MoSCoW technique: In this technique, all the stakeholders of the application (Client, Project manager, Business Analyst etc.) reach a common understanding for the importance (priority) they place on each requirement.
The MoSCoW stands for:

• M – MUST have this / Must Test this.
• S – SHOULD have this / Should test this.
• C – COULD have this / Could Test this
• W – WON’T have this time / Won’t test this time.

Here, requirements are categorized on above four criteria. A Must have requirement is one, which has highest priority and that must be included in the delivery. A Should have Requirement is one, which is also critically needed for the application but can be implemented in later releases. Could have requirements are good to have features, which are less critical. Won’t have requirements are least critical, and are considered inappropriate for this delivery. These may be considered in later releases if time and budget permits
4.Quality Criteria: Quality criteria are an agreement between client and the vendor which states – what kind of defects in which numbers will be accepted by the client. To make a release to be possible, the application must satisfy the agreed upon quality criteria.

Risk Based Test Methodology
Since we have already discussed the building blocks for RBT, now we have to co-relate them to gain understanding of the RBT methodology. In a scenario where we have huge business requirements, short timelines, limited resources then we have to make sure that our testing must cover the most essential functions. Establishing each chunk of functionalities relative importance lets us sequence our construction of business and test scenarios to provide the greatest quality at the lowest cost.
The RBT process can be described by these major steps:

• Step1 – Define all requirements in terms of Risk associated to them
• Step2 – Based on risk assessment, prioritize the requirements
• Step3 – Plan and define tests according to requirement prioritization
• Step4 – Execute test according to prioritization and acceptance criteria.

Here we will see how the concepts, discussed in earlier chapters, ease the process of defining test strategy for a project having tight test schedule and budget with lots of requirements to be tested. The first two steps are common for development stream and test stream, however Step 3 and 4 are specifically dedicated to testing.
•Step 1 – Various risks associated with the applications are identified by the stakeholders of the project. The stake holders here should be a mix of business and technical team – that means, involving people from various departments and with various backgrounds: for example, the client, customers, business experts, technical experts, project manager, project Leader, users, developers and infrastructure representative. The risks can be identified in a number of ways whichever is considered suitable to the stakeholders:

• By using past experience
• Checklists of recurrent risks
• By holding workshop

The “impact analysis” for each identified risk is done in order to understand the consequences.

•Step 2 – Once all the possible risks and their impacts are analyzed, then project manager has to get the requirements prioritized. To achieve this- combination of Pareto Principle and various prioritization techniques (discussed above) are used by the stakeholders. Priority of the requirements should be agreed upon and should be updated in functional requirement document; the same should also be conveyed to the development and the test teams.

•Step 3 – Once we have the Requirements with priority tagged with them, test activities should be started keeping the priority of the requirements in mind. Business scenarios and test scenarios for the “must have” requirements must be derived first and then for “Should”, “Could” and for “Won’t have”, if time permits. Here while the test scenarios are being written, we must set priority to each test scenario. This priority can be set with the common understanding of tester and test lead/ manager. Each test case should have its priority – “Must test”, “Should test”, Could test”, “Won’t Test”- applying Pareto principle (here I have used MoSCoW priority just for example).
Wondering, how to have a “Could test” priority test case for a “Must have” requirement? Here pops the “quality attributes” & “quality criteria” in to picture. We have to consider which Quality attributes are in scope of testing and their respective weightage with respect to Quality Criteria set for the release. Test cases for the quality attributes with high weightage and having minimum limit in quality criteria should be given high priority and vise vice.
Figure – 2 – Requirement and Test Specification Diagram.
For Example:

• Case 1 – if Functional quality attribute has high weightage and quality criteria quotes Critical functionality – 0 defects, then this means that the application should be tested for functionality having no critical functional issues. Hence test cases representing this criterion should have “Must test” priority. A functional test case which covers the navigation or User Interface requirement will have “Could test” priority.
• Case 2 – If “Usability” quality attribute has high weightage and quality criteria quotes 0 Cosmetic – defects (Most unlikely case), then this means that the application should be tested for usability having no usability/cosmetic/ GUI issues. Hence test cases representing this criterion should have “Must test” priority.

•Step 4: If any of the identified risk realizes by this time (When test execution is scheduled), then there is quite good chances of schedule slippage from development side. In this case, since final deadline to customer can’t be changed, obviously the “Test execution Schedule” would be shrunken. In such scenario, Test manager again has to apply the Pareto principle and has to finalize the scope of testing in the reduced timeline that will ensure least risk and highest quality. Test scenarios/ Cases that have “Must test” priority would be executed first to ensure basic Quality criteria are met. Then if time permits, test cases with “Should test” priority would be executed and so on. Concept here is to test the most important features first, ensuring the bare minimum quality requirements, and then if time and budget permits then widening the scope of testing to lesser important requirements.

Test Techniques Helpful for carrying RBT:
Test techniques that have specific purpose of capturing issues from most error prone areas are the best suited for RBT:

• Boundary Value analysis
• Equivalence partitioning
• Decision tables
• Path Flow testing
• Some amount of exploratory / Experience based testing
• Others, based on type of testing need to be performed

Risk Based Test Reporting:
Risk based Test reporting provides all the Stakeholders to understand the Project’s current status in a better and more understandable way. Test manger produces a Test report that includes Rick coverage matrix, based on risks identified (discussed earlier). The main objective of risk based test reporting is to be able to display the test coverage of risks, i.e – which of the identified risks have been tested so far and which of them have not yet been.
These reports basically help in decision making for the release. If the release is made today, then what are the identified risks which might realize, since they are not yet addressed by the test team? At the same time – how safer are we, i.e – how many risks are covered by testing?
Advantages of RBT:
The Most important advantages of this approach are as follows:

• Linking the product risks to the requirements identifies gaps.
• With limited time, money and qualified resources, testing concentrates on the most important matters first, thus delivering the most optimal test.
• The focus lies on risks to the business instead of just the functionality of the information system.
• During testing, communication (Test Reporting) takes place in a language (risks) that all stakeholders understand, hence involvement of testing is greater in comparison to mere list of issues communicated.
• It offers a negotiating instrument to client and test manager alike when available means are limited.