You Can’t Evaluate a Test Tool by Reading a Data Sheet
Datasheets generally have the same appearance. “Industry Leader,” “Unique Technology,” “Automated Testing,” and “Advanced Methods” are the same catchphrases. “Bar Charts,” “Flow Charts,” “HTML reports,” and “Status Percentages” are among the similar screenshots. It dulls the mind.
How does software testing work?
We all who have tested software know that testing comes in various forms. In this work, we will only use three terminologies to keep things simple:
System testing, unit testing, and integration testing
Everyone performs some system testing to use the system to complete some of the same tasks as the final users. You’ll note that we stated “some” rather than “all.” Unexpected and thus untested input combinations that an application runs into in the field are one of the most frequent causes of apps being fielded with defects.
Even fewer people than integrated testing do unit testing. If you’ve ever conducted integration or unit testing, you’re probably aware of the quantity of test code required to separate a single file or set of files from the rest of the program. It is not unusual for the portion of test code created to exceed the amount of application code being tested at the most rigorous levels of testing. As a result, these testing levels are typically used for mission- and safety-critical applications in industries including aviation, medical technology, and railway.
What Exactly Is “Automated Testing”?
Every tool being sold in this market will tout “Automated Testing” as an advantage because it is well known that performing unit and integration tests manually costs a lot of money and takes a lot of time. How about “automatic testing,” though? To various people, automation might signify different things. The promise of “automated testing” is taken by many engineers to mean that they may click a button and receive either a “green check,” signifying that their code is correct, or a “red x,” suggesting failure.
Regrettably, there is no such tool. More importantly, would you want to utilize it if it were available? Consider this. What does a device tell you your code is “Ok” mean? Does that imply that the code is nicely formatted? Maybe. Does that imply that it complies with your code requirements? Maybe. Does that tell that your code is sound? Not!
Fully automated testing is neither feasible nor desirable. The labor-intensive, computational aspects of the testing process should be handled by automation. This frees up the software engineer to perform testing tasks of higher value, such as creating better and more thorough tests.
When assessing tools, it makes sense to ask, “How much automation does this tool provide?” When a company tries to determine an ROI for tool investment, this is a significant gray area and the primary source of confusion.
How Test Tools Work
Generally speaking, test tools offer a wide range of functions. Vendors will use different names for various instruments; some may lack certain features. We have selected the following terms for the “modules” that could be present in the test tools you are assessing to provide a typical frame of reference:
Parser: The tool can comprehend your code thanks to the parser module. After reading it, it creates an interim representation for the code (usually in a tree structure). Similar to what the compiler does, essentially. The result, or “parse data,” is generally saved in an intermediate language (IL) file.
CodeGen: The code generator module creates the test harness source code using the “parse data” feature.
Even though the test harness isn’t a part of the tool itself, decisions about its architecture impact all of the other aspects. Therefore while evaluating a device, the harness architecture is crucial.
Compiler: The test tool can call the compiler using the compiler module to compile and link the test harness components.
Target: The target module makes it simple to execute tests in many different runtime environments, including those that support emulators, simulators, embedded debuggers, and commercial RTOS.
Preconditions and expected values (pass/fail criteria) for test cases can be set up using the test editor’s advanced graphical user interface (GUI) or scripting language.
Coverage: The user can obtain information on the sections of the code run by each test using the coverage module.
Reporting: The reporting module enables the compilation of project documentation from the gathered data.
CLI: A command line interface (CLI) enables the tool to be called from scripts, make, and other automated processes, further automating the tool’s operation.
Regression: The regression module enables tests written for an application version to be rerun against newer versions.
Integrations: Leveraging your investment in a test tool through integrations with third-party solutions might be an intriguing strategy. Static analysis, requirements management, and configuration management tools are frequently integrated.
In later sections, we’ll go into more detail on how to assess each of these modules in your potential tools.
Degrees of Automation / Types of Test Tools
We have divided test tools into the following general categories because not all devices offer all the capabilities or modules mentioned above and because each agency’s level of automation varies greatly. Candidates for test instruments will fit into one of these groups.
The test data and logic needed to implement the test cases must be manually coded with “manual” tools, often producing an empty test harness framework. They frequently offer a scripting language and a collection of library functions that can be used to perform routine tasks like creating prepared test reports or test assertions.
While “Semi-Automatic” tools may add a graphical user interface to some Automated capabilities offered by a “manual” agency, hand-coding and scripting will still be necessary to test more intricate constructions. Also, some of the modules that an “automated” tool contains may be absent from a “semi-automated” device. Built-in assistance with target deployment, for instance.
“automated” technologies will address each of the functional domains or modules indicated in the previous section. The tools in this class will handle every language construct and a wide range of target deployments without requiring manual hand coding.
Minor tool variations
It’s crucial to assess and compare the test approach utilized in addition to tool features and automation levels. It is essential to create some straightforward test cases for each function in the class you are testing rather than just loading your code into the tool because this could obscure underlying flaws in the device. Builds a complete test harness using the tool? Do stubs all get generated automatically? Is it possible to specify test case parameters and global data using the GUI, or must you write code as you would if you were testing manually?
Target support varies significantly amongst tools in a similar manner. If a seller asserts, “We support all compilers and all targets out of the box,” take caution. These are code phrases: “You do all the work to make our tool operate in your environment.”
How to Assess Testing Tools
The information you should look into while evaluating a software testing tool is described in detail in the following few parts. Ideally, you should test each device in person to check this information.
We want to explain some of the conventions employed as most of this work is quite technical. We have a “Key Points” section to include specific things to think about, a title that highlights an issue to be considered, a discussion of why the problem is significant, and the section itself.
Terminology should also be mentioned while we are discussing conventions. The terms “function” and “unit” denote a C function or a C++ class method, respectively. Finally, remember that practically all tools can somehow support the points described in the “Key Points” sections; it is your responsibility to determine how automated, user-friendly, and comprehensive the assistance is.
Generator of Code and Parsers
Building a parser for C is quite simple, but building a complete parser for C++ is very challenging. How reliable and developed is the parser technology should be one of the topics that are addressed throughout the tool evaluation. Some tool suppliers employ commercial parser technology that they license from parser technology firms, while others have in-house parsers that they have created. Testing the tool with intricate code constructions that are a good representation of the code that will be used for your project may confirm the robustness of the parser and code generator.
Crucial Ideas
– Is the parser technology third-party or proprietary?
Which languages can I use?
– Are the C++ and C versions of a tool the same, or are they different?
– Are there limitations, or is the complete C++ language implemented?
– Does the tool support the most complex code we have?
The Test Pilot
The “primary software” directing the exam is the Test Driver. Here is a straightforward illustration of a driver that tests the sine function from the standard C library:
#include
#include
Integer Main ()
local float
sin (local) = 90.0;
If the local is more significant than 1.0, print (“My Test Passed!”);
alternatively print (“My Test Failed! n”);
deliver 0;
A “semi-automated” tool might provide some scripting language or straightforward GUI to enter the stimulus value for sine, even though this is a relatively simple example. A “manual” tool might need you to type (and debug) this small snippet of code by hand. A fully functional GUI for creating test cases, integrated code coverage analysis, an integrated debugger, and integrated target deployment are all features of an “automated” tool.
Do you know that there is a bug in this driver? The error is that the input angle for the sin function is given in radians rather than degrees.
Major Points
– Is the driver generated automatically, or must I write the code?
– Is it possible to test the following without using any code:
– Testing with various values
Testing Combinatorial
– Testing for Data Partitions (Equivalence Sets)
– Input value lists
– Expected value lists
– Expected values as exceptions
– Signal management
– Is arranging a series of calls to various methods within a single test possible?
Dependent Functions Stumbling
When you want to manage the values that a dependent function returns during a test, you must build replacements for those functions. To isolate the code being tested from other components of your application and more readily stimulate the execution of the unit or sub-system of interest, stumping is a highly crucial component of integration and unit testing.
To make a stub perform anything other than return a static scalar result, many tools require the manual development of the test code (return 0;)
Major Points
– Are stubs generated automatically, or must you write code for them?
– Are complicated outputs (classes, structures) automatically supported?
– Can the stub return a different value with each call?
– Does the stub record how frequently it was called?
– Does the stub maintain a record of the input parameters across calls?
– Can you stub calls to malloc and other standard C library functions?
Test Results
“Semi-automated” and “automated” tools implement test cases using one of two fundamental strategies. A “data-driven” architecture is one, and a “single-test” design is the other.
The test harness is developed for a data-driven architecture and supports all the functionalities specified in each tested unit. The tool sends the stimulus data across a data stream, like a file handle or a physical interface, such as a UART, when a test is to be executed.
When a test is run for a “single-test” architecture, the tool builds the test driver specific to that test, compiles it, and links it into an executable. Here are several things to consider: first, the single-test approach requires additional code creation, which increases test execution time; second, you create a different test harness for each test case.
This means a potential tool may seem to work in some hypothetical situations but may not function correctly in more challenging assessments.
Major Points
– Does the test harness use data?
– How long does it take to run a test case, taking into account any time required for code generation and compilation?
– Can the test cases be edited outside the test tool IDE?
– If not, have I played around with the tool in sufficient detail using challenging code examples to comprehend any restrictions?
Generation of Test Data Automatically
A certain amount of automated test case creation is offered by several “automated” solutions. Various methods are employed to do this. Several of these methods are described in the paragraphs that follow:
A function will be stressed by Min-Mid-Max (MMM) Test Cases tests at the limits of the input data types. Code written in C and C++ frequently does not defend itself against out-of-bound inputs. The engineer has a functional range in mind but often does not shield themselves from inputs outside that range.
Equivalence Classes (EC) tests construct “partitions” for each data type and choose a sample of values from each partition. The underlying premise is that matters from the same section will influence the application similarly.
Random Values (RV) tests will set combinations of random values for each function’s parameters.
To look at the distinct paths that exist within a procedure, Basic Paths (BP) tests use foundation path analysis. Branch coverage can be extensively increased automatically by BP tests.
The function that automatic test case development fulfills is the most crucial consideration. Automated tests help evaluate the application code’s robustness but not its accuracy. It would be best if you designed tests for accuracy based on what the program should do rather than what it does.
Integration of Compilers
Compiler integration serves two purposes. One goal is to make it possible for the test harness components to be automatically compiled and linked without requiring the user to understand the necessary compiler parameters. The third goal is to enable any language extensions specific to the compiler being used to be honored by the test tool. It is pretty usual for the compiler to include extensions that are not part of the C/C++ language standards, especially with cross-compilers. Some utilities use the method of #defining these extensions to null strings. This extremely shoddy method is particularly terrible because it modifies the object code that the compiler generates. Consider the following global extern with a GCC property, for instance:
((aligned (16))); extern int MyGlobal __attribute__;
The code will operate differently during testing than when it is deployed because the memory will not be aligned the same way if your candidate tool does not keep the attribute while defining the global object MyGlobal.
Major Points
– Is the test harness automatically compiled and linked by the tool?
– Does the tool support compiler-specific language extensions and implement them?
– What kind of interface (IDE, CLI, etc.) does the compiler have?
– Is there a way to import project settings from your development environment into the tool, or do you have to do it by hand?
– If the tool does import project settings, is this import functionality restricted to a specific compiler or compiler family, or is it a general purpose?
– Does the tool allow you to debug tests because it is integrated with your debugger?
Support for Embedded Target Testing
The term “Tool Chain” will be used in this section to refer to the entire cross-development environment, which includes the cross-compiler, target board, debug interface (emulator), and Real-Time Operating System (RTOS). It’s crucial to consider if the potential tools have robust target integrations for your toolchain and to be aware of any changes that would need to be made if you migrated to a different toolchain.
Understanding the target integration’s level of automation and resilience is also crucial. As previously stated, if a manufacturer claims that “we support all compilers and all targets out of the box,” they probably do. They imply that you must put in all the effort to make our tool function in your setting.
Comparing it to a comparing it to a comparing it to a comparing it to a comparing it to a comparing it to a comparing it to a comparing it to a comparing it to a comparing it to a comparing it to a comparing it to an a comparing it to a comparing it to an a…
Hardware accessibility complicates embedded target testing further. Frequently, the hardware is being created concurrently with the program, or the hardware is not readily available. The capability to begin testing in a native environment and go on to the actual hardware later is a crucial component. The tool artifacts should be hardware independent in theory.
Major Points
Does my toolchain have support? If not, can it be supported? Why is anything “supported”?
– Can tests be created on a host system and then used on a target system for testing?
– How does the target download the test harness?
– How is the host informed of the test results?
What target types, cross-compilers, and RTOSes are pre-supported?
Who creates the foundation for a new toolchain?
– Is there a user-configurable component of the toolchain integration?
Editor for Test Cases
You will undoubtedly spend most of your interaction time with a test tool in the test case editor. The time required to set up the test environment and the target connection should be minimal if the preceding topics discussed in this paper are truly automated. As stated, you should utilize the engineer’s time to create better, more thorough tests.
It is crucial to examine how difficult it is to set up test input and expected values for non-trivial structures. Every tool on the market now offers a simple method for configuring scalar values. Does your candidate tool, for instance, provide a clear and straightforward way to create a class? What about setting up an STL container in an abstract style, like a vector or a map? These are the things the test case editor should be used to evaluate.
There are two types of help, similar to the rest of this essay: manual support and automated support. While analyzing structures that might interest you, keep this in mind.
Major Points
– Are permitted scalar value ranges displayed?
Are array sizes displayed?
– Can Min and Max’s values be easily set using tags rather than values? If a type changes, keeping the test’s integrity is crucial.
– Do unique floating point numbers (like NaN and +/- Infinity) support
– Can you perform combinatorial tests, changing the values of 5 parameters in a range and asking the tool to test every possible combination?
– Is the editor “base aware” so that values in different bases like hex, octal, and binary can be easily entered?
– Can you quickly enter absolute tolerances for floating point numbers, such as +/- 0.05, and relative tolerances, such as +/- 1%?
– Is importing test data from other sources, such as Excel, simple?
Coverage of Code
All “automatic” tools and most “semi-automated” tools offer a built-in code coverage feature that lets you view metrics that illustrate the application’s portion performed by your test cases. This data is presented in tables by several tools. Some display flow diagrams, while others display annotated source lists. While tables are helpful as summaries, an annotated source listing is ideal for achieving 100% code coverage. The source code file will be displayed in this listing, along with colorations for covered, partially covered, and uncovered constructs. This makes it simple to identify the extra test cases required to achieve 100% coverage.
It’s critical to comprehend how more instrumentation will affect your program. There are two factors to take into account: the first is the growth in object code size, and the second is the run-time overhead. Knowing if your application is memory or real-time-constrained is crucial (or both). This will assist you in concentrating on the component that is most crucial to your application.
Major Points
-How much does the code size for each kind of instrumentation increase?
– How much longer does each sort of instrument take to run?
– Can your “make” or “build” system integrate instrumentation?
– How are the user’s coverage results displayed? Are there only tables of metrics, or are there annotated listings with a graphical coverage browser?
– How is the target information about coverage retrieved? Is the procedure adaptable? Is RAM capable of buffering data?
– Do you support the statement, branch (or decision), and MC/DC coverage?
– Can various forms of coverage be recorded during a single execution?
– Is it possible to share coverage data between different test environments? For instance, might system testing coverage be integrated with unit and integration testing coverage?
– Can you visualize the control flow through your application without a debugger using the coverage data to go through the test execution?
– Can all coverage for each test run be obtained in a single report?
– Can the tool be certified for DO-178B and its intended use as a medical device?
Testing for Regression
While choosing a test instrument, there should be two main objectives. Saving time while testing is the main objective. We assume you concur with that if you’ve read this far. The secondary aim is to use the generated tests throughout the application life cycle. This means that the effort and resources put into creating tests should produce tests that can be reused as the program evolves and are simple to configure. The most critical items to assess in your potential tool are what precise information needs to be “kept” to conduct the same tests later and how to test repetition is managed.
Major Points
What file(s) must be configuration maintained to perform a regression test? Is the tool’s Command Line Interface (CLI) comprehensive and well-documented? Are these files binary or plain text? This impacts your ability to utilize a diff tool to assess changes over time. Do the tool’s generated harness files require configuration management? Are tools for configuration management integrated? Make a unit test, rename a parameter, and rebuild your test environment. When will this be finished? Is it difficult?
> Does the software enable statistical graphs and database technology for trend analysis of test execution and code coverage over time? Is it possible to automatically test several code baselines using the same test cases? Is distributed testing available, allowing for the execution of some test cases over multiple physical machines to accelerate testing?
Reporting
The majority of solutions offer comparable reporting. They should, at the very least, produce a report that is simple to interpret and includes the inputs, anticipated results, actual results, and a comparison of the predicted and actual values.
Major Points
What are formats of output supported? HTML? Text? CSV? XML? \s> Is it easy to acquire both an overarching (project-wide) report and a more in-depth report for a single function? Is the report’s content editable by users?
> Is it possible to customize the report format?
Integration of Additional Tools
No matter how excellent or practical a given tool may be, it must be able to function in a multi-vendor context. Big businesses have frequently invested money in acquiring small companies to provide “the tool” to perform any task for any customer. The intriguing point is that the whole is frequently much less than the sum of the parts with these huge tool suites. Businesses often combine 4-5 reasonably excellent tiny tools into one clunky, useless one.
Major Points
> Which tools does your potential device already integrate with, and can the user add further integrations?
Other Functions that Would Be Ideal for a Testing Tool
The functionality covered in the parts before should be present in any tool used for automated testing. We will mention some desirable traits in the following sections, along with explaining why each aspect is significant. The degree to which these elements apply to your specific project may vary.
Many units are under test and proper integration testing.
Unit testing is expanded upon by integration testing. You must assemble the components of a functional process to combine the units used to test interfaces between them. Many solutions assert that they can support integration testing by connecting the object code for actual units with the test harness. The test harness executable is built using several files, but there is no way to activate the functions of these additional units using this technique. In a perfect world, each test case would allow you to start any part, in any order, within any team. The application’s hidden assumptions and faults will typically be found by testing the unit interfaces. In fact, for projects without a history of unit testing, integration testing could be a good starting step.
Major Points
Can I build sophisticated test scenarios for these classes where we trigger a series of operations across several units within a single test case? > Can I incorporate numerous teams in the test environment? Can I collect metrics for code coverage across many units?
Adaptive Stubbing
You can dynamically turn on and off specific function stubs by using dynamic stubbing. This enables the creation of tests for single functions with stubs for all other parts (even if they exist in the same unit as the function under test). This is an excellent feature for complex code because it makes testing much more straightforward.
Major Points
Stubs can be selected at the unit level or the function level. Can function stubs be enabled or disabled for each test case? The function stubs (see items in the previous section) are they generated automatically?
Thread testing at the application and library levels (System Testing)
One of the difficulties with system testing is that the fully integrated application’s test stimulus may call for a user to press buttons, toggle switches, or type at a console. The inputs may be considerably trickier to control if the application is embedded. Imagine you could perform integration testing-style stimulation on your fully integrated application at the function level. This would enable you to create intricate test scenarios that solely rely on the application’s API.
You can test in this manner using some of the more modern tools. This testing method also has the advantage of not requiring the source code to test the application. All you need is the API definition (generally the header files). With the help of this methodology, testers can do system testing in an automated and scriptable manner.
Development that is test-driven and agile (TDD)
Testing will be incorporated earlier than ever in the development process, thanks to test-driven development. You build your tests before your application code instead of writing application code first and your unit tests as an afterthought. This is a well-liked new method of development that requires testing frequently and upfront. Your automated tool should enable this testing technique if you employ an Agile Development methodology.
Integration with Requirements Tools in Both Directions
A test tool should link with a requirements management tool if you care about connecting requirements with test cases. If you are interested in this functionality, the interface must be bi-directional so that information about test cases, such as test name and pass/fail status, can be pushed back to your requirements database when requirements are tagged to test claims. This will give you an idea of how thorough your requirements testing was.
Equipment Qualification
You must “qualify” the development tools used to create and test your application if you work in a regulated environment, such as commercial aviation or Class III medical devices.
The certification process entails defining what the tool must accomplish and conducting testing to demonstrate that it does so. A vendor should ideally have these products on hand and a history of clients who have used the qualifying data for your sector.
Major Points
Are materials for tool vendor qualification generated specifically for your target environment and toolchain available? What initiatives made use of these resources successfully? How is the content licensed? Have the qualification materials been successfully used to certify to DO-178B Level A if this is an FAA project? How are the materials adapted and authorized for a specific project? Have the tools been approved for “intended use” if it is an FDA project?
Conclusion
We hope the information in this paper will be helpful to you as you explore the options offered by test tool suppliers. For various projects, the proportional relevance of each issue identified will vary. Our last recommendations are:
> Test the potential tools on a sample of your application’s code that reflects its complexity. Use the same toolchain that will be utilized for your project to evaluate the possible tools. Ask about some of the concerns presented to the vendor’s long-term clients in this article. Inquire about the technical support staff for the device. Test them out by contacting their customer service with some inquiries (rather than their sales representative)
Finally, remember that almost every tool can somehow assist the things listed in the “Key Points” sections. You are responsible for assessing the support’s automation, usability, and completeness.
Concerning Vector Software
Vector Software, Inc is the top independent supplier of automated software testing solutions for creators of safety-critical embedded programs. The VectorCAST family of solutions from Vector Software manages and automates the challenging unit, integration, and system testing tasks. VectorCAST products support the programming languages C, C++, and Ada.
East Greenwich, Rhode Island 02818 (USA) 1351 South County Trail, Suite 310
Visit our website for additional details or to obtain a free trial.
Read also: Tips For Helping Your Parents Use Technology
