- a methodology called STEP™, which was created by Dr. Bill Hetzel and Dr. David Gelperin as a way to implement the original IEEE-829 Standard for Test Documentation.
- The [[Software Testing]] Engineering Process (STEP) was developed out of a need for a methodology that not only specified what testing documents needed to be created, but also how to create them and develop the necessary processes for their use.
- STEP doesn't establish absolute rules, but rather provides guidelines that can be modified to meet the needs of software engineers.
- [[Testing History]]
- [[Preventive Testing]]
- Testing is often seen as difficult due to factors such as ambiguous and incorrect requirements, tight time schedules, and the complexities of the software itself. Despite these challenges, testing is a crucial part of the software development process, often accounting for a significant portion of the development budget.
## STEP Methodology
The Systematic Test and Evaluation Process (STEP) was first introduced in 1985 as part of the course material for the Systematic Software Testing seminar series. It has since been revised many times and field-tested through consulting engagements and the shared experience of many individuals and organizations. STEP is built upon the foundation of the IEEE Std. 829-1983 Standard for Software Test Documentation and subsequently updated based on the latest version (IEEE Std. 828-1998) of this standard and the IEEE Std. 1008-1987 Standard for Software [[Unit Testing]]. While retaining compatibility with these standards, this methodology has grown in scope and now stands as one of the leading models for effective software testing throughout the industry.
### Key Point
Much of this section was reprinted from the STEP Guide with permission from Software Quality Engineering.
### Scope and Objectives of STEP
STEP covers the broad activity of software evaluation. Evaluation is defined as that sub-discipline of software engineering concerned with determining whether software products do what they are supposed to do. The major techniques employed in evaluation are analysis, review and test. STEP focuses on testing as the most complex of the three, but stresses overall coordination and planning of all aspects of evaluation as a key to success. It stresses the prevention potential of testing, with defect detection and demonstration of capability as secondary goals.
#### Key Point
Evaluation is defined as the sub-discipline of software engineering concerned with determining whether software products do what they are supposed to do.
- Early views saw testing as a phase that occurred after software development, or "something that programmers did to get the bugs out of their programs."
- The more modern view sees testing as a process to be performed in parallel with the software development or maintenance effort (refer to [Figure 1-2](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/section-7.xhtml#ch01fig02 "Figure 1-2: Views of Testing")) incorporating the activities of planning (determining risks and selecting strategies); analysis (setting test objectives and requirements); design (specifying tests to be developed); implementation (constructing or acquiring the test procedures and cases); execution (running and rerunning the tests); and maintenance (saving and updating the tests as the software changes).
Figure 1-2: Views of Testing
This lifecycle perspective of testing represents a major change from just a few years ago, when many equated testing with executing tests. The contribution of planning, analyzing, and designing tests was under-recognized (and still is by many people), and testing was not seen as really starting until tests started running. Now we understand the evaluation power of test planning and analysis. These activities can be more powerful than test execution in defect prevention and timely detection. We also understand that an accurate interpretation of the situation when "all tests are running successfully" requires a clear understanding of the test design.
The lifecycle model for testing that has emerged borrows heavily from the methodology we've grown accustomed to for software. Considering that a test set is made up of _data_ and _procedures_ (which are often implemented as executable test programs), it should not come as a surprise that what it takes to build good software is also what it takes to build good testware!
### Elements of STEP
STEP draws from the established foundation of software methodologies to provide a process model for software testing. The methodology consists of specified tasks (individual actions); work products (documentation and implemented tests); and roles (defined responsibilities associated with groups of tasks), as shown in [Figure 1-3](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/section-7.xhtml#ch01fig03 "Figure 1-3: Elements of STEP"), packaged into a system with proven effectiveness for consistently achieving quality software.
Figure 1-3: Elements of STEP
The STEP methodology is not tool dependent and does not assume any particular test organization or staffing (such as independent test groups). It does assume a development (not a research) effort, where the requirements information for the product and the technical design information are comprehensible and available for use as inputs to testing. Even if the requirements and design are not specified, much of the STEP methodology can still be used and can, in fact, facilitate the analysis and specification of software requirements and design.
#### Key Point
Even if the requirements and design are not specified, much of the STEP methodology can still be used and can, in fact, facilitate the analysis and specification of requirements and design.
### STEP Architecture
[Figure 1-4](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/section-7.xhtml#ch01fig04 "Figure 1-4: Activity Timing at Each Level of Test") shows how STEP assumes that the total testing job is divided into levels during planning. A level represents a particular testing environment (e.g., unit testing usually refers to the level associated with program testing in a programmer's personal development library). Simple projects, such as minor enhancements, may consist of just one or two levels of testing (e.g., unit and acceptance). Complex projects, such as a new product development, may have more levels (e.g., unit, function, subsystem, system, acceptance, alpha, beta, etc.).
Figure 1-4: Activity Timing at Each Level of Test
STEP provides a model that can be used as a starting point in establishing a detailed test plan. All of the components of the model are intended to be tailored and revised, or extended to fit each particular test situation.
The three major phases in STEP that are employed at every level include: planning the strategy (selecting strategy and specifying levels and approach), acquiring the testware (specifying detailed test objectives, designing and implementing test sets), and measuring the behavior (executing the tests and evaluating the software and the process). The phases are further broken down into eight major activities, as shown in [Table 1-3](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/section-7.xhtml#ch01table03 "Table 1-3: STEP Activities & Their Locations in This Book").
#### Table 1-3: STEP Activities & Their Locations in This Book
|[**Step 1**](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/section-10.xhtml#ch02lev3sec1 "Form a Brainstorming Team")|**Plan the Strategy**|**Covered In**|
|---|---|---|
|P1|Establish the master test plan.|[Chapters 2](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/chapter-2-8.xhtml#ch02 "Risk Analysis") and [3](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/appendix-D-80.xhtml#ap04lev2sec3 "Introduction")|
|P2|Develop the detailed test plans.|[Chapter 4](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/chapter-4-18.xhtml#ch04 "Detailed Test Planning")|
| [**Step 2**](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/section-10.xhtml#ch02lev3sec2 "Compile a List of Features") | **Acquire the Testware** | |
| ---- | ---- | ---- |
| A1 | Inventory the test objectives (requirements-based, design-based, and implementation-based). | [Chapter 5](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/chapter-5-23.xhtml#ch05 "Analysis and Design") |
| A2 | Design the tests (architecture and environment, requirements-based, design-based, and implementation-based). | [Chapter 5](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/chapter-5-23.xhtml#ch05 "Analysis and Design") |
| A3 | Implement the plans and designs. | [Chapter 6](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/chapter-6-30.xhtml#ch06 "Test Implementation") |
| [**Step 3**](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/section-10.xhtml#ch02lev3sec3 "Determine the Likelihood") | **Measure the Behavior** | |
| ---- | ---- | ---- |
| NOTE: Chapters 8, 9, and 10 cover the testing organization, the software tester, and the test manager, respectively. Chapter 12 provides a review of critical testing processes. | | |
| M1 | Execute the tests. | [Chapter 7](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/chapter-7-35.xhtml#ch07 "Test Execution") |
| M2 | Check the adequacy of the test set. | [Chapter 7](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/chapter-7-35.xhtml#ch07 "Test Execution") |
| M3 | Evaluate the software and testing process. | [Chapter 11](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/chapter-11-59.xhtml#ch11 "Improving the Testing Process") |
### Timing of STEP Activities
STEP specifies when the testing activities and tasks are to be performed, as well as what the tasks should be and their sequence, as shown in [Figure 1-5](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/section-7.xhtml#ch01fig05 "Figure 1-5: Activity Timing at Various Levels of Test"). The timing emphasis is based on getting most of the test design work completed before the detailed design of the software. The trigger for beginning the test design work is an external, functional, or black box specification of the software component to be tested. For higher test levels (e.g., acceptance or system), the external specification is equivalent to the system requirements document. As soon as that document is available, work can (and should) begin on the design of the requirements-based tests.
Figure 1-5: Activity Timing at Various Levels of Test
The test design process continues as the software is being designed and additional tests based on the detailed design of the software are identified and added to the requirements-based tests. As the software design process proceeds, detailed design documents are produced for the various software components and modules comprising the system. These, in turn, serve as functional specifications for the component or module, and thus may be used to trigger the development of requirements-based tests at the component or module level. As the software project moves to the coding stage, a third increment of tests is designed based on the code and implementation details.
#### Key Point
The goal at each level is to complete the bulk of the test design work as soon as possible.
Test inventory and design activities at the various levels overlap. The goal at each level is to complete the bulk of the test design work as soon as possible. This helps to ensure that the requirements are "testable" and well thought out and that defects are discovered early in the process. This strategy supports an effective software review and inspection program.
Measurement phase activities are conducted by level. Units are executed first, then modules or functions are integrated and system and acceptance execution is performed. The sequential execution from small pieces to big pieces is a physical constraint that we must follow. A major contribution of the methodology is in pointing out that the planning and acquisition phases are not so constrained; and furthermore, it's in our interest to reverse the order and begin to develop the high-level test sets first - even though we use them last!
The timing within a given test level is shown in [Figure 1-6](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/section-7.xhtml#ch01fig06 "Figure 1-6: Activity Timing at Various Levels of Test") and follows our natural expectation. Plans and objectives come first, then test design, then implementation, then finally execution and evaluation. Overlap of activities is possible.
Figure 1-6: Activity Timing at Various Levels of Test
### Work Products of STEP
Another aspect of the STEP process model is the set of work products produced in each phase and activity. STEP uses the word ["testware"](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/appendix-A-76.xhtml#Testware "Testware") to refer to the major testing products such as test plans and test specification documents and the implemented test procedures, test cases, and test data files. The word ["testware"](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/appendix-A-76.xhtml#Testware "Testware") is intentionally analogous to software and, as suggested by [Figure 1-7](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/section-7.xhtml#ch01fig07 "Figure 1-7: Parallel, Mutually Supportive Development"), is intended to reflect a parallel development process. As the software is designed, specified, and built, the testware is also designed, specified, and built.
Figure 1-7: Parallel, Mutually Supportive Development
These two broad classes of work products support each other. Testware development, by relying on software work products, supports the prevention and detection of software faults. Software development, by reviewing testware work products, supports the prevention and detection of testware faults.
STEP uses IEEE standard document templates as a recommended guideline for document structure and content. [Figure 1-8](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/section-7.xhtml#ch01fig08 "Figure 1-8: Template for Test Documents from IEEE Std. 829-1998. The templates for many IEEE documents are presented in this book, but we recommend that you purchase the complete guidelines from the IEEE at— www.ieee.org") lists the documents that are included in this book.
Figure 1-8: Template for Test Documents from IEEE Std. 829-1998. The templates for many IEEE documents are presented in this book, but we recommend that you purchase the complete guidelines from the IEEE at— www.ieee.org
> ###### **IEEE Std. 829-1998 Standard for Software Test Documentation Template for Test Documents**
>
> ###### **Contents**
>
> | | |
> |---|---|
> |1.|Test Plan
Used for the master test plan and level-specific test plans.|
> |2.|[[Test Design]] Specification
Used at each test level to specify the test set architecture and coverage traces.|
> |3.|Test Case Specification
Used as needed to describe test cases or automated scripts.|
> |4.|Test Procedure Specification
Used to specify the steps for executing a set of test cases.|
> |5.|Test Log
Used as needed to record the execution of test procedures.|
> |6.|Test Incident Report
Used to describe anomalies that occur during testing or in production. These anomalies may be in the requirements, design, code, documentation, or the test cases themselves. Incidents may later be classified as defects or enhancements.|
> |7.|Test Summary Report
Used to report completion of testing at a level or a major test objective within a level.|
Implementations are the actual test procedures to be executed along with their supporting test data and test files or test environments and any supporting test code that is required.
### Roles and Responsibilities in STEP
Roles and responsibilities for various testing activities are defined by STEP. The four major roles of manager, analyst, technician, and reviewer are listed in [Table 1-4](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/section-7.xhtml#ch01table04 "Table 1-4: Roles and Responsibilities").
Table 1-4: Roles and Responsibilities
|**Role**|**Description of Responsibilities**|
|---|---|
|Manager|Communicate, plan, and coordinate.|
|Analyst|Plan, inventory, design, and evaluate.|
|Technician|Implement, execute, and check.|
|Reviewer|Examine and evaluate.|
These roles are analogous to their counterpart roles in software development. The test manager is responsible for providing overall test direction and coordination, and communicating key information to all interested parties. The test analyst is responsible for detailed planning, inventorying of test objectives and coverage areas, test designs and specifications, and test review and evaluation. The test technician is responsible for implementation of test procedures and test sets according to the designs provided by the analyst, for test execution and checking of results for termination criteria, and for test logging and problem reporting. The test reviewer provides review and oversight over all steps and work products in the process.
The STEP methodology does not require that these roles be filled by different individuals. On small projects, it's possible that one person may wear all four hats: manager, analyst, technician, and reviewer. On larger projects and as a test specialty becomes more refined in an organization, the roles will tend to be assigned to different individuals and test specialty career paths will develop.
#### Key Point
On smaller projects, it's possible that one person may wear all four hats: manager, analyst, technician, and reviewer.
### Summary of STEP
STEP has been introduced through Software Quality Engineering's (SQE) Systematic Software Testing classes to hundreds of organizations. It's a proven methodology offering significant potential for improving software quality in most companies.
Key differences between STEP and prevalent industry practices are highlighted in [Table 1-5](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/section-7.xhtml#ch01table05 "Table 1-5: Major Differences Between STEP and Industry Practice"). First is the overall goal of the testing activity. STEP is prevention oriented, with a primary focus on finding requirements and design defects through early development of test designs. This results in the second major difference of when major testing activities are begun (e.g., planning timing and activity timing). In STEP, test planning begins during software requirements definition, and testware design occurs in parallel with software design and before coding. Prevalent practice is for planning to begin in parallel with coding and test development to be done after coding.
Table 1-5: Major Differences Between STEP and Industry Practice,
|**[Methodology](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/appendix-A-76.xhtml#Methodology_Test "Methodology (Test)")**|**Focus**|**Planning Timing**|**Acquisition Timing**|**[Coverage](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/appendix-A-76.xhtml#Coverage "Coverage")**|**Visibility**|
|---|---|---|---|---|---|
|STEP|Prevention & [[Risk management]]|Begins During Requirements Definition|Begins During Requirements Definition|Known (Relative to Inventories)|Fully Documented & Evaluated|
|Prevalent Industry Practice|Detection & Demonstration|Begins After Software Design|Begins After Software Design (or Code)|Largely Unknown|Largely Undocumented with Little or No Evaluation|
Another major difference between STEP and prevalent industry practices is the creation of a group of test cases with known coverage (i.e., mapping test cases to inventories of requirements, design, and code). Finally, using the IEEE documents provides full documentation (i.e., visibility) of testing activities.
#### Key Point
In STEP, test planning begins during software requirements definition and testware design occurs in parallel with software design and before coding.
STEP also requires careful and systematic development of requirements and design-based coverage inventories and for the resulting test designs to be calibrated to these inventories. The result is that in STEP, the test coverage is known and measured (at least with respect to the listed inventories). Prevalent practice largely ignores the issue of coverage measurement and often results in ad hoc or unknown coverage.
A final major difference lies in the visibility of the full testing process. Every activity in STEP leads to visible work products. From plans, to inventories, to test designs, to test specs, to test sets, to test reports, the process is visible and controlled. Industry practice provides much less visibility, with little or no systematic evaluation of intermediate products.
These differences are significant and not necessarily easy to put into practice. However, the benefits are equally significant and well worth the difficulty and investment.
#### Key Point
_[Calibration](https://cdn2.percipio.com/1707466290.b163b888783bb66f9dd5d6ad376a827cf0eded40/eod/books/6411/OEBPS/appendix-A-76.xhtml#Calibration "Calibration")_ is the term used to describe the measurement of coverage of test cases against an inventory of requirements and design attributes.