This chapter describes the Tester graphical user interface. It contains these sections:
When you run cvxcov, the main Tester window opens and an iconized version of the Execution View appears on your screen. It displays the output and status of a running program and accepts input. To open a closed Execution View, see "Clone Execution View" in "Admin Menu Operations".
There are two methods of accessing the Tester graphical user interface:
Type cvxcov at the command line with these optional arguments: -testname test to load the test; -ver to show the Tester release version; and -scheme schemename to set a predefined CASEVision color scheme.
Select "Tester" from the "Launch Tool" submenu in a WorkShop Admin menu (see Figure 9-1). The major WorkShop tools, the Debugger, Static Analyzer, and Build Manager provide Admin menus from which you can access Tester.
The main window and its menus are shown in Figure 9-2.
The current test is entered (and displayed) in the Test Name field. You can switch to a different test, test set, or test group through this field. To the right, the Type field indicates whether it is a Single Test, Test Set, or Test Group. You can select a test (test set or test group) from the "List Tests" dialog box under the Test menu, to appear in the Test Name field in the main window.
Test results display in the coverage display area. You select the results by choosing an item from the Queries menu. You can select the format of the data—text, call tree, or bar chart— from the Views menu. (Note that the Text View format is available for all queries, whereas the other two views are limited.)
The Query Type displays under the Test Name field, just over the display. It is followed on the far right of the window by the Query Size (number of items in the list). Headings above the display are specific to each query.
The Search field lets you look for strings in the coverage data. It uses an incremental search, that is, as you enter characters, the highlight moves to the first matching target. When you press <return>, the highlight moves to the next occurrence.
Apply is a general-purpose button for terminating data entry in text fields; you can use <return> equivalently. Both start the query.
Source lets you bring up the standard CASEVision Source View window with Tester annotations. Source View shows the counts for each line and highlights lines with 0 counts. By default Source View is shared with other applications. For example, if cvstatic performs a search for function A, the results of the query overwrite Tester query results that are in the shared Source View. To stop sharing Source View with other applications, set the following resource:
cvsourceNoShare: True |
Disassembly brings up the CASEVision Disassembly View window, called "Assembly Source Coverage", which operates at the machine level in a similar fashion to the Source View. This view is not shared with other applications.
Contribution brings up the Test Contribution window with the contributions made by each test so that you can compare the results. It is available for the queries "List Functions", "List Arcs", and "List Blocks". When the tests do not fit on one page, multiple pages are used. Use the Previous Page and Next Page buttons to display all the tests.
Sort lets you sort the test results by criteria such as function, count, file, type, difference, caller, or callee. The criteria available depend on the current query.
The status area displays status messages that confirm commands, issue warnings, and indicate error conditions. When you enter a test name in the Test Name field, the Func Name field appears (along with other items) in the status area for use with queries. Entering a function in this field displays the coverage results limited to that function only.
Additional items display in the area below the status area that change when you select commands from the Queries menu. These items are specific to the query selected. Some of these items can be used as defaults (see "Queries Menu Operations").
The Admin menu lets you perform general housekeeping concerning saving files, setting defaults, changing directories, launching other WorkShop applications, and exiting.
The Test menu lets you create, modify, and run tests, test sets, and test groups.
The Views menu lets you choose one of the following modes:
text mode, which displays results numerically in columns
graphical mode, which displays
functions as nodes (rectangles) annotated by results
calls as arcs (connecting arrows)
bar graph mode, which displays the summary of a test as a bar graph.
The Queries menu lets you analyze the results of tests. The Help menu is standard in all CASEVision tools.
All operations for running tests are accessed from the Test menu in the main Tester window. Figure 9-3 shows the dialog boxes used to perform test operations.
The Test menu provides these selections:
"Run Instrumentation" |
| |
The "Run Instrumentation" dialog box (see Figure 9-4) provides these fields:
| ||
"Run Test" |
| |
"Make Test" |
The "Make Test" dialog box (see Figure 9-6) provides these fields for tests, test sets, and test groups: | |
If you select Single Test, the following fields are provided:
| ||
If you select Test Set, the following fields and buttons are provided: | ||
For a test group (see Figure 9-7), the following field is added to the same fields and buttons used for a test set: | ||
"Delete Test" |
| |
"List Tests" |
| |
"Modify Test" |
|
The Views menu has three selections that let you view coverage data in different forms. The selections are:
"Text View" |
| |
"Call Tree View" |
| |
"Bar Graph View" |
|
The Queries menu provides different methods for analyzing the results of coverage tests. Each type of query displays the coverage data in the coverage display area in the main Tester window and displays items that are specific to the query in the area below the status area. When you set these items for a query, the same values are used by default for subsequent queries until you change them. You can set these defaults before the first query or as part of any query. For a single test or test set, all queries except "Describe Test" have the fields shown in Figure 9-14.
The Executable field displays the executable associated with the current coverage data. You can switch to a different executable by entering it directly in this field. You can also switch executables by clicking the Executable List button, selecting from the list in the Target List dialog box and clicking Apply in the dialog box.
The experiment menu (Expt) lets you see the results for a different experiment that uses the same test criteria.
Note: When you are performing queries on a test group, the Executable field changes to Object field and the Executable List button changes to Object List as shown in Figure 9-14. These items act analogously except that they operate on dynamically shared objects (DSOs). Refer to "Tutorial #4 — Analyzing a Test Group" for more information on test groups. |
The Queries menu (see Figure 9-16) provides these selections:
"List Summary" |
The Coverages column indicates the type of coverage. The Covered column shows the number of functions, source lines, branches, arcs, and blocks that were executed in this test (or test set or test group). The Total column indicates the total number of items that could be executed for each type of coverage. The % Coverage column is simply the Covered value divided by the Total value in each category. The Weight column indicates the weighting assigned to each type of coverage. It is used to compute the Weighted Sum, a user-defined factor that can be used to judge the effectiveness of the test. The Weighted Sum is obtained by first multiplying the individual coverage percentages by the weighting factors and then summing the products. The "List Summary" command causes the coverage weighting factor fields to display below the status area. Use these to adjust the factor values as desired. They should add up to 1.0. If you select "Bar Graph View" from the Views menu, the summary will be shown in bar graph format as shown in Figure 9-13. The percentage covered is shown along the vertical axis; the types of coverage are indicated along the horizontal axis. | ||
"List Functions" |
"List Functions" enables the sort menu that lets you determine the order in which the functions display. Only the sort criteria appropriate for the current query are enabled, in this case, "Sort By Func", "Sort By Count", and "Sort By File" as shown in Figure 9-18. The Search field scrolls the list to the string entered. The string may occur in any of the columns. This is an incremental search and is activated as you enter characters, scrolling to the first matching occurrence. Entering a function in the Func Name field displays the coverage results limited to that function only in the display area. The Filters button displays the Filters dialog box, which lets you enter filter criteria to display a subset of the coverage results. There are three types of filters: Function Count, Block Count (%), and Branch Count (%). For blocks or branch coverage, use the toggles described below. Following each label is an operator menu to define the relationship to the limit quantity entered. Each filter type has a text field for entering the desired limit. The limits for Block Count and Branch Count are percentages (of coverage) and can also be entered using sliders. Two toggles are available for including branch and block counts. Both appear as actual counts followed by parentheses containing the ratio of counts to total possible. If you select "Call Tree View" from the Views menu with a "List Functions" query, a call graph displays (see Figure 9-19). The call graph displays coverage data graphically, with functions as nodes (rectangles) and calls as arcs (connecting arrows). The nodes are color-coded according to whether the function was included and covered in the test, included and not covered, or excluded from the test. Arcs labeled N/A connect excluded functions and do not have call counts. If you hold down the right mouse button over a node, the node menu displays, including the function name, coverage statistics, and standard node manipulation commands. If you have a particularly large graph, you may find it useful to zoom to 15% or 40% and look at the coverage statistics through the node menu. | ||
"List Blocks" |
Be careful before listing all blocks in the program, since this can produce a lot of data. Entering a function in the Func Name field displays the coverage results limited to that function only in the display area. | ||
"List Branches" |
The first column shows the line number in which the branch occurs. If there are multiple branches in a line, they are labeled by order of appearance within trailing parentheses. The next two columns indicate the function containing the branch and the file. A branch is considered covered if it has been executed under both true and false conditions. The Taken column indicates the number of branches that were executed only under the true condition. The Not Taken column indicates the number of branches that were executed only under the false condition. Branch coverage can be sorted only by function and file. Entering a function in the Func Name field displays the coverage results limited to that function only in the display area. | ||
"List Arcs" |
Entering a function in the Func Name field displays the coverage results limited to that function only. The Caller and Callee toggles let you view the arcs for a single function either as a caller or callee. You do this by entering the function name in the Func Name field and then clicking the appropriate toggle, Caller or Callee. | ||
"List Argument Traces" |
where:
For more information on the instrumentation file, see "Single Test Analysis Process". The Arguments column shows the calling function with its argument. Type indicates the type of the argument. Range shows the minimum and maximum values if TRACE bounds was selected; otherwise, it shows the end of the range selected with a short line (-) substituted for the opposite end of the range. Entering a function in the Func Name field displays the coverage results limited to that function only in the display area. | ||
"List Instrumentation" |
Function List toggle shows the functions that are included in the coverage experiment. Ver allows you to specify the version of the program that was instrumented. The latest version is used by default. Executable displays the executable associated with the current coverage data. You can switch to a different executable by entering it directly in this field. You can also switch executables by clicking the ExecutableList button, selecting from the list in the dialog box, and clicking Apply in the dialog box. | ||
"List Line Coverage" |
| ||
"Describe Test" |
| ||
"Compare Test" |
The comparison data displays in the coverage display area. The basic types of coverage display in the Coverages column. Result 1 and Result 2 display the results of the experiments specified in the Expt1 and Expt2 fields, respectively. Results are shown as the counts followed by the coverage percentage in parentheses. The values in the Result 2 column are subtracted from those in Result 1 and the differences are shown in the Differences column. If you want to view the available experiments, click the Expt: menu. You can also compare the differences in function coverage by clicking the Diff Functions toggle. Figure 9-28 shows a typical function difference example. |
The Admin menu is shown in Figure 9-29.
The Admin menu provides these selections:
"Save Results" |
| |
"Clone Execution View" |
| |
"Set Defaults" |
| |
"Launch Tool" |
If any of these tools are not installed on your system, the corresponding menu item will be grayed out. |
"Exit" closes all Tester windows.