Static Testing - Session 3

Static TestingSession 31OverviewReviews and the test processTypes of reviewStatic analysis2People techniquesindividual:desk-checking, data-stepping, proof-readinggroup:Reviews (informal & formal): for consensusWalkthrough: for educationInspection (most formal): to find faultsStatic techniques do not execute code3Benefits of reviewsDevelopment productivity improvementReduced development timescalesReduced testing time and costLifetime cost reductionsReduced fault levelsImproved customer relationsetc.4Reviews are cost-effective10 times reduction in faults reaching test, testing cost reduced by 50% to 80%Freedman & Weinberg, Handbook of Walkthroughs, Inspections & Technical Reviewsreduce faults by a factor of 10Yourdon, Structured Walkthroughs25% reduction in schedules, remove 80% - 95% of faults at each stage, 28 times reduction in maintenance cost, many othersGilb & Graham, Software Inspection5What can be Inspected?policy, strategy, business plans, marketing or advertising material, contractssystem requirements, feasibility studies, acceptance test planstest plans, test designs, test cases, test resultssystem designs, logical & physicalsoftware codeuser manuals, procedures, training materialAnything written downcan be Inspected6What can be reviewed?anything which could be Inspectedi.e. anything written downplans, visions, “big picture”, strategic directions, ideasproject progresswork completed to schedule, etc.“Should we develop this” marketing options7What to review / Inspect?TestsTestsTestsTestsRequirementsDesignCodeFunctionsIntegration TestUnit TestAccept. TestSystem Test8Costs of reviewsRough guide: 5%-15% of development efforthalf day a week is 10%Effort required for reviewsplanning (by leader / moderator)preparation / self-study checkingmeetingfixing / editing / follow-uprecording & analysis of statistics / metricsprocess improvement (should!)9OverviewReviews and the test processTypes of reviewStatic analysis10Types of review of documentsInformal Review	undocumentedwidely viewed as useful and cheap (but no one can prove it!) A helpful first step for chaotic organisations.Technical Review:	(or peer review)includes peer and technical experts, no management participation. Normally documented, fault-finding. Can be rather subjective.Decision-making Review:group discusses document and makes a decision about the content, e.g. how something should be done, go or no-go decision, or technical comments11Types of review of documentsWalkthroughauthor guides the group through a document and his or her thought processes, so all understand the same thing, consensus on changes to makeInspection:formal individual and group checking, using sources and standards, according to generic and specific rules and checklists, using entry and exit criteria, Leader must be trained & certified, metrics required12Reviews in general 1Objectives / goalsvalidation & verification against specifications & standardsachieve consensus (excluding Inspection)process improvement (ideal, included in Inspection)13Reviews in general 2Activitiesplanningoverview / kickoff meeting (Inspection)preparation / individual checkingreview meeting (not always)follow-up (for some types)metrics recording & analysis (Inspections and sometimes reviews)14Reviews in general 3Roles and responsibilitiesLeader / moderator - plans the review / Inspection, chooses participants, helps & encourages, conducts the meeting, performs follow-up, manages metricsAuthor of the document being reviewed / InspectedReviewers / Inspectors - specialised fault-finding roles for InspectionManagers - excluded from some types of review, need to plan project time for review / InspectionOthers: e.g. Inspection/ review Co-ordinator15Reviews in general 4DeliverablesChanges (edits) in review productChange requests for source documents (predecessor documents to product being reviewed / Inspected)Process improvement suggestionsto the review / Inspection processto the development process which produced the product just reviewed / InspectedMetrics (Inspection and some types of review)16Reviews in general 5Pitfalls (they don’t always work!)lack of training in the technique (especially Inspection, the most formal)lack of or quality of documentation - what is being reviewed / InspectedLack of management support - “lip service” - want them done, but don’t allow time for them to happen in project schedulesFailure to improve processes (gets disheartening just getting better at finding the same thing over again)17Inspection is differentthe document to be reviewed is given out in advancetypically dozens of pages to reviewinstructions are "please review this"some people have time to look through it and make comments before the meeting (which is difficult to arrange)the meeting often lasts for hours"I don't like this" much discussion, some about technical approaches, some about triviadon't really know if it was worthwhile, but we keep doing itchunk or sampletraining, rolesentry criteria to meeting, may not be worth holdingRule violations, objective, not subjectiveno discussion, highly focused, anti-triviaonly do it if value is proven (continually)not just product, sources2 max., often much shorter18Inspection is more and betterentry criteria training optimum checking rateprioritising the words standards process improvement exit criteriaquantified estimates of remaining major faults per pagetypical reviewearly Inspectionmature Inspectioneffectivenessreturn on investment10 - 20%unknown30 - 40%6 - 8 hrs / Insp hr80 - 95%8 - 30 hrs / Insp hr19The Inspection ProcessSoftwareDevelopmentStage..PlanningKickoffIndChkMeetEditChangeRequestProcess ImprovementEntryNext SoftwareDevelopmentStageExit20At first glance ..Here’s a document: review this (or Inspect it)21Reviews: time and size determine rateTimeCheckingRateSize2 hrs?100 pages?50 pages per hourCheckingRate22Review “Thoroughness”?ordinary “review” - finds some faults, one major, fix them,consider the document now corrected and OKmajorminorminor23Inspection: time and rate determine sizeTimeCheckingRateSize2 hrs?Optimum:1 page* per hour2 pages (at optimum rate)* 1 page = 300 important words24Inspection ThoroughnessInspection can find deep-seated faults: all of that type can be corrected but needs optimum checking rate25Inspection surprisesFundamental importance of Rulesdemocratically agreed as applyingdefine major issues / faultsSlow checking ratesStrict entry & exit criteriaFast logging ratesAmount of responsibility given to author26OverviewReviews and the test processTypes of reviewStatic analysis27What can static analysis do?A form of automated testingcheck for violations of standardscheck for things which may be a faultDescended from compiler technologya compiler statically analyses code, and “knows” a lot about it, e.g. variable usage; finds syntax faultsstatic analysis tools extend this knowledgecan find unreachable code, undeclared variables, parameter type mis-matches, uncalled functions & procedures, array bound violations, etc.Remember: static techniques do not executethe code28Data flow analysisThis is the study of program variablesvariable defined* where a value is stored into itvariable used where the stored value is accessedvariable is undefined before it is defined or when it goes out of scope*defined should not be confused with declaredx = y + zIF a > b THEN read(S)x is defined, y and z are useda and b are used, S is defined29Data flow analysis faultsn := 0read (x)n := 1while x > y do	begin	 read (y)	 write( n*y)	 x := x - n	endData flow anomaly: n isre-defined without being usedData flow fault: y is usedbefore it has been defined(first time around the loop)30Control flow analysisHighlights:nodes not accessible from start nodeinfinite loopsmultiple entry to loopswhether code is well structured, i.e. reduciblewhether code conforms to a flowchart grammarany jumps to undefined labelsany labels not jumped tocyclomatic complexity and other metrics31Unreachable code exampleMacro definitions (different for different platforms the code runs on) 	Buffsize: 1000 	Mailboxmax: 1000	IF Buffsize 60% THEN Print "pass"ELSE Print "fail”ENDIFPseudo-code:doifr=r+1endinitifrespassfailend35Other static metricslines of code (LOC)operands & operators (Halstead’s metrics)fan-in & fan-outnesting levelsfunction callsOO metrics: inheritance tree depth, number of methods, coupling & cohesion36Limitations and advantagesLimitations:cannot distinguish "fail-safe" code from programming faults or anomalies (often creates overload of spurious error messages)does not execute the code, so not related to operating conditionsAdvantages:can find faults difficult to "see"gives objective quality assessment of code37Summary: Key PointsReviews help to find faults in development and test documentation, and should be applied earlyTypes of review: informal, walkthrough, technical / peer review, InspectionStatic analysis can find faults and give information about code without executing it38Thank YouAiTi Educationpublished by www.aiti.edu.vn@aiti_aptechaiti.edu.vn39