Fundamentals of testing - Session 1A

e?Testing Principle - Testing is context dependentTesting is done differently in defferent contexts. For example, safety-critical software is tested from an e-commerce site.7Why do faults occur in software?software is written by human beingswho know something, but not everythingwho have skills, but aren’t perfectwho do make mistakes (errors)under increasing pressure to deliver to strict deadlinesno time to check but assumptions may be wrongsystems may be incompleteif you have ever written software ...Testing Principle - Testing shows presence Testing can show that defects are present, but cannot prove that there are no defects. Testing reduces the probability of undiscovered defects remaining in the software but, event if no defects are found, it is not a proof of correctness.8What do software faults cost?huge sumsAriane 5 ($7billion)Mariner space probe to Venus ($250m)American Airlines ($50m)very little or nothing at allminor inconvenienceno visible or physical detrimental impactsoftware is not “linear”: small input may have very large effect9Safety-critical systemssoftware faults can cause death or injuryradiation treatment kills patients (Therac-25)train driver killedaircraft crashes (Airbus & Korean Airlines)bank system overdraft letters cause suicide10So why is testing necessary?because software is likely to have faultsto learn about the reliability of the softwareto fill the time between delivery of the software and the release dateto prove that the software has no faultsbecause testing is included in the project planbecause failures can be very expensiveto avoid being sued by customersto stay in business11Why not just "test everything"?system has20 screensAverage: 10 fields / screen2 types input / field(date as Jan 3 or 3/1)(number as integer or decimal) Around 100 possible valuesTotal for 'exhaustive' testing:20 x 4 x 3 x 10 x 2 x 100 = 480,000 testsIf 1 second per test, 8000 mins, 133 hrs, 17.7 days(not counting finger trouble, faults or retest)Avr. 4 menus3 options / menu10 secs = 34 wks, 1 min = 4 yrs, 10 min = 40 yrs12Exhaustive testing?What is exhaustive testing?when all the testers are exhaustedwhen all the planned tests have been executedexercising all combinations of inputs and preconditionsHow much time will exhaustive testing take?infinite timenot much timeimpractical amount of timeTesting Principle - Exhaustive testing is impossibleTesting everything (all combinations of inputs and preconditions) is not feasible except for trivial cases. Instead of exhaustive testing, we use risks and priorities to focus testing efforts.13How much testing is enough?it’s never enoughwhen you have done what you plannedwhen your customer/user is happywhen you have proved that the system works correctlywhen you are confident that the system works correctlyit depends on the risks for your system14How much testing?It depends on RISKrisk of missing important faultsrisk of incurring failure costsrisk of releasing untested or under-tested softwarerisk of losing credibility and market sharerisk of missing a market windowrisk of over-testing, ineffective testing15So little time, so much to test ..test time will always be limiteduse RISK to determine:what to test firstwhat to test mosthow thoroughly to test each itemwhat not to test (this time)use RISK toallocate the time available for testing by prioritising testing ...}i.e. where toplace emphasis16Most important principlePrioritise testsso that,whenever you stop testing,you have done the best testingin the time available.17Testing and qualitytesting measures software qualitytesting can find faults; when they are removed, software quality (and possibly reliability) is improvedwhat does testing test?system function, correctness of operationnon-functional qualities: reliability, usability, maintainability, reusability, testability, etc.18Other factors that influence testingcontractual requirementslegal requirementsindustry-specific requirementse.g. pharmaceutical industry (FDA), compiler standard tests, safety-critical or safety-related such as railroad switching, air traffic controlIt is difficult to determinehow much testing is enoughbut it is not impossible19OverviewWhy testing is necessaryFundamental test processPsychology of testingRe-testing and regression testingExpected resultsPrioritisation of tests20Test Planning - different levelsTestPolicyTestStrategyCompany levelHigh LevelTest PlanHigh LevelTest PlanProject level (IEEE 829)(one for each project)DetailedTest PlanDetailedTest PlanDetailedTest PlanDetailedTest PlanTest stage level (IEEE 829)(one for each stage within a project,e.g. Component, System, etc.)21The test processspecificationexecutionrecordingcheckcompletionPlanning (detailed level)22Test planninghow the test strategy and project test plan apply to the software under testdocument any exceptions to the test strategye.g. only one test case design technique needed for this functional area because it is less criticalother software needed for the tests, such as stubs and drivers, and environment detailsset test completion criteriaTesting Principle - Early TestingTesting activities should start as early as possible in the software or system development life cycle and should be focused on defined objectives.23Test specificationspecificationexecutionrecordingcheckcompletionIdentify conditionsDesign test casesBuild testsPlanning (detailed level)24A good test caseeffectiveexemplaryevolvableeconomicFinds faultsRepresents othersEasy to maintainCheap to use25Test specificationtest specification can be broken down into three distinct tasks:1. identify:	determine ‘what’ is to be tested 	(identify test conditions) and 	prioritise2. design:	determine ‘how’ the ‘what’ is to 	be tested	(i.e. design test cases)3. build:	implement the tests 	(data, scripts, etc.)26Task 1: identify conditionslist the conditions that we would like to test:use the test design techniques specified in the test planthere may be many conditions for each system function or attributee.g.“life assurance for a winter sportsman”“number items ordered > 99”“date = 29-Feb-2004”prioritise the test conditionsmust ensure most important conditions are covered(determine ‘what’ is to be tested and prioritise)27Selecting test conditionsImportanceTimeBest set48First set28Task 2: design test casesdesign test input and test dataeach test exercises one or more test conditionsdetermine expected resultspredict the outcome of each test case, what is output, what is changed and what is not changeddesign sets of testsdifferent test sets for different objectives such as regression, building confidence, and finding faults(determine ‘how’ the ‘what’ is to be tested)Testing Principle - Defect clusteringA small number of modules contain most of the defects discovered during pre-release testing or show the most operational failures29Designing test casesImportanceTimeMost importanttest conditionsLeast importanttest conditionsTest cases30Task 3: build test casesprepare test scriptsless system knowledge tester has the more detailed the scripts will have to bescripts for tools have to specify every detailprepare test datadata that must exist in files and databases at the start of the testsprepare expected resultsshould be defined before the test is executedTesting Principle - Pesticide paradoxIf the same tests are repeated over and over again, eventually the same set of test cases will no longer find any new bugs. To overcome this "pesticide paradox", the test cases need to be regularly reviewed and revised, and new and different tests need to be written to exercise diffirent parts of the software or system to potentially find more defects.31Test executionspecificationexecutionrecordingcheckcompletionPlanning (detailed level)32ExecutionExecute prescribed test casesmost important ones firstwould not execute all test cases iftesting only fault fixestoo many faults found by early test casestime pressure can be performed manually or automated33Test recordingspecificationexecutionrecordingcheckcompletionPlanning (detailed level)34Test recording 1The test record contains:identities and versions (unambiguously) ofsoftware under testtest specificationsFollow the planmark off progress on test scriptdocument actual outcomes from the testcapture any other ideas you have for new test casesnote that these records are used to establish that all test activities have been carried out as specified35Test recording 2Compare actual outcome with expected outcome. Log discrepancies accordingly:software faulttest fault (e.g. expected results wrong)environment or version faulttest run incorrectlyLog coverage levels achieved (for measures specified as test completion criteria)After the fault has been fixed, repeat the required test activities (execute, design, plan)36Check test completionspecificationexecutionrecordingcheckcompletionPlanning (detailed level)37Check test completionTest completion criteria were specified in the test planIf not met, need to repeat test activities, e.g. test specification to design more testsspecificationexecutionrecordingcheckcompletionCoverage too lowCoverageOK38Test completion criteriaCompletion or exit criteria apply to all levels of testing - to determine when to stopcoverage, using a measurement technique, e.g.branch coverage for unit testinguser requirementsmost frequently used transactionsfaults found (e.g. versus expected)cost or timeTesting Principle - Absence-of-errors fallacyFinding and fixing defects does not help if the system build is unusable and does not fulfill the users' needs and expectations.39 Comparison of tasks	ClericalIntellectualone-offactivityactivityrepeatedmany timesGoverns thequality of testsGood toautomateExecuteRecordingPlanningSpecification40Summary: Key PointsTesting is necessary because people make errorsThe test process: planning, specification, execution, recording, checking completion41Thank YouAiTi Educationpublished by www.aiti.edu.vn@aiti_aptechaiti.edu.vn42