Table of Contents
- Part 1: The Very Basics
- Overview
- Starting TotalView
- Environment Setup
- Compiling Your Program
- Starting TotalView
- TotalView's Basic Look and Feel
- Primary Windows
- Dialog Boxes
- Mouse Usage
- Menus
- Accelerator Keys
- Scrolling, Resizing and Memorizing
- Process and Thread State Codes
- X Resources
- TotalView's Basic Functions
- Viewing Source Code
- Setting a Breakpoint
- Controlling Execution
- Diving
- Viewing and Modifying Data
- Text Editing and Searching
- Saving Window Contents
- Getting Help
- Exiting TotalView
- Part 2: Common Functions
- Loading Executables
- Attaching / Detaching Processes
- Viewing a Core File
- Expression Evaluation
and Code Fragments
- More on Action Points
- Setting Executable Command
Arguments
- Setting Source Code Search
Paths
- Setting stdin, stdout, and stderr
- Signal Handling
- Displaying Your Program's Call Tree
- Debugging Memory Problems
- Setting Preferences
- Visualizing Array Data
- Command Line Interpreter (CLI)
- Part 3: Debugging Parallel Programs
- Process/Thread Groups
- Debugging Threaded Codes
- Overview
- Finding Thread Information
- Selecting a Thread
- Execution Control for
Threaded Programs
- Viewing and Modifying Thread
Data
- Debugging OpenMP Codes
- Overview
- Debugging OpenMP Programs
- Debugging MPI Codes
- Overview
- Starting an MPI Debug Session
- Selecting an MPI Process
- Controlling MPI Process
Execution
- Viewing and Modifying
Multi-process Data
- Displaying Message Queue
State
- Debugging Hybrid Codes
- Overview
- Debugging Hybrid Programs
- Batch System Debugging
- Topics Not Covered
- References and More Information
TotalView Part 1: The Very Basics
|
|
What is TotalView?
- TotalView is a sophisticated software debugger product of
Etnus LLC.
- Used for debugging and analyzing both serial and parallel programs.
- Especially designed for use with complex, multi-process and/or
multi-threaded applications.
- Without question, the most popular HPC debugger to date.
- Has been selected as the Department of Energy's
Advanced Simulation and
Computing (ASC) program's debugger.
Key Features of TotalView:
- Designed to handle most types of HPC parallel coding
- Supported on most HPC platforms (in the US)
- Provides both a GUI and command line interface
- Can be used to debug programs, running processes, and core files.
- Includes memory debugging features
- Provides graphical visualization of array data
- Includes a comprehensive built-in help system
- And more...
Supported Platforms and Languages
- Etnus TotalView is supported on most major U.S. HPC platforms.
- Ports of TotalView to other platforms (NEC, Hitachi, Fujitsu, etc)
are available from non-Etnus sources.
- For the most up-to-date platform related information see "Platforms"
on the Etnus web site.
- Supported languages include the usual HPC application languages:
- C/C++
- Fortran77/90
- Mixed C/C++ and Fortran
- Assembler
How to Obtain TotalView:
- LC provides TotalView on all of its production platforms.
- Non-LC users can obtain TotalView software and documentation from Etnus's
web site: www.etnus.com.
Software can be downloaded for free for a trial period. All
documentation may be downloaded for free anytime.
Environment Setup
LC takes care of most of this for users.
Path Variable:
TotalView should be in the default path of LC users. If you prefer a
version different than the default, then you can begin navigating
(usually) from the /usr/global/tools/tv/ directory to
find your preferred version. For example:
alias totalview /usr/global/tools/tv/toolworks/totalview.7.1.1-0/bin/totalview
License Manager File:
Taken care of for LC users. Otherwise...
The TotalView license is typically served under the FLEXlm
license manager. You may need to specify the
LM_LICENSE_FILE
environment variable to point to the location of its license.dat file.
This location may vary from site to site.
Authorization:
Taken care of for LC users. Otherwise...
For multi-process programs, such as those using MPI, you will probably need
to insure that you are authorized on all the machines you intend to use. How
this is accomplished depends upon your site. In most cases it is through
the usual UNIX trusted hosts mechanisms. Minimally, you'll need to be able
to use rsh or a substitute command.
Xwindows:
OK, here's one you have to do for yourself at LC:
Because the TotalView GUI is an Xwindows application, you will need to
make sure that your Xwindows environment is setup correctly.
This may differ from machine to machine, depending upon such factors as:
- Your machine platform - Unix, Linux, Mac, Microsoft...
- The type of Xwindows software you have installed
- SSH software and X-tunneling
- Connectivity method between your local machine and the machine
where TotalView is running
- Network and access security
Compiling Your Program
-g:
- Like many UNIX debuggers, you will need to compile your program with the
appropriate flag to enable generation of symbolic debug information.
For most compilers, the -g option is used for this.
- TotalView will allow you to debug executables which were not compiled with
the -g option. However, only the assembler code can be
viewed.
Beyond -g:
- Don't compile your program with optimization flags
while you are debugging it. Compiler optimizations can "rewrite"
your program and produce machine code that doesn't necessarily match
your source code.
- Parallel programs may require additional compiler flags.
Starting TotalView
A Variety of Ways:
TotalView can be started in a number of different ways, depending upon
whether you want to:
- debug an executable file
- attach to a running process
- debug a core file
- change the debugger's default appearance/behavior
Some Examples:
Command / Action |
- totalview
- Starts the debugger. You can then load a program or
corefile, or else attach to a running process.
|
- totalview filename
- Starts the debugger and loads the program specified by
filename.
|
- totalview filename corefile
- Starts the debugger and loads the program specified by
filename and its core file specified by corefile.
|
- totalview filename -a args
- Starts the debugger and passes all subsequent arguments
(specified by args) to the program specified by filename. The
-a option must appear after all other TotalView options on
the command line.
|
- totalview filename -remote hostname[:portnumber]
- Starts the debugger on the local host and the totalview debugger server
on the remote host hostname. Loads the program specified by
filename for remote debugging. You can specify a host name or
TCP/IP address for hostname, and optionally, a TCP/IP port number
for portnumber.
|
- totalview -fn 9x15
- Starts the debugger and uses the specified font.
|
More Information:
- The totalview command has a number of options
that may be used to control the GUI's behavior and appearance.
- Additionally, it is possible to customize TotalView's appearance and
behavior through "initialization files". These initialization files
can be global to a site and/or local to a user.
- See the TotalView documentation from Etnus.
TotalView's Basic Look and Feel
|
Primary Windows
Root Window:
- Will always appear when the TotalView GUI is started. If TotalView is
started without any arguments, it is the only window to appear.
- Provides an overview of all processes and threads, showing the TotalView
assigned ID, MPI rank, host, status and brief description/name for each.
- Allows sorting by ID, rank, host and status.
- Provides the ability to expand/collapse each process to view/hide any
included threads in that process.
- Has 6 Pull-down menus - File, Edit, View, Tools, Window, Help
(menus are discussed later)
Process Window:
- Usually (but not always) appears with the Root Window after
TotalView is started.
- By default, a single process window will display. For multi-process /
multi-threaded programs however, every process and every
thread may have its own Process Window if desired.
- Comprised of:
- Pull-down menus
- Execution control buttons
- Navigation control buttons
- Process and thread status bars
- 4 "Panes"
- Stack Trace Pane
- Shows the call stack of routines the current executable is running
- Selection of any routine shown in the call stack will automatically
update the Process Window with its information.
- Source Pane
- Displays source/assembler for the currently selected program or
function.
- Shows program counter, line numbers and any associated action points.
- Only "boxed" line numbers are eligible for debugging.
- Stack Frame Pane
- Displays the local variables, registers and function parameters for
the selected executable.
- Register abbreviations and meanings are architecture specific. See
the TotalView documentation for details.
- Action Points, Processes, Threads Pane
- A multi-function pane. By default, it shows any action points
(covered later) that have been set.
- May also select Processes to show attached processes or Threads to show
associated threads.
- The Action Points display is shown above. The
Processes and Threads displays are shown below.
Variable Window:
- Probably the most common window after the Root and Process windows.
- Comprised of a single pane with pull-down menus. Additional information
about the variable is shown. “More” and “Less” buttons allow you to
filter the amount of information shown.
- Appears when you dive (covered later) on a variable or select a menu item
to view variable information.
- Displays detailed information about selected program variables. Also
permits editing of variable data.
TotalView's Basic Look and Feel
|
Dialog Boxes
- TotalView has numerous dialog boxes that are used for a variety
of purposes:
- Solicit and confirm selections
- Display informational, warning and error messages
- Accept input
- Display and select options and preferences
- Display various types of information
- Dialog boxes vary in complexity.
A few representative dialog boxes are shown below.
TotalView's Basic Look and Feel
|
Mouse Usage
Much of your interaction with the TotalView debugger is through the use
of a three-button mouse. Each mouse button has a specific purpose,
described below.
NOTE: If you are using a machine that doesn't have a
three button mouse, you will probably need to configure the mouse you
have to emulate a three button mouse.
|
|
Mouse Button |
Purpose |
Description |
LEFT |
Select / Dive |
Single clicking on an object causes it to be selected and/or to
perform its action. Double-clicking allows you to dive into an
object. For example, double-clicking on an array object in the source
pane will cause a new window to pop open, showing the array's values. |
MIDDLE |
Paste |
Writes information previously copied or cut into the clipboard
at the cursor's position. |
Dive |
Display more information about an object |
RIGHT |
Context menu |
Pops-up a menu of common commands related to the object clicked on
(if applicable). |
TotalView's Basic Look and Feel
|
Menus
Two Types of Menus:
- Drop-down menus:
- Visible
- Appear along the top border of most windows
- Activated by clicking with the left mouse button
- Some menu selections may have submenus
- Pop-up menus:
- Hidden
- Activated by clicking on an object (such as a variable, line number,
etc.) with the right mouse button
- Not all objects possess pop-up menus
- Menus are context sensitive - different windows will have different menus.
- Dimmed menu selections are either irrelevant or not available.
- TotalView has many menus – too many to show here. Only a few representative
menus are shown below – two drop-down menus and two pop-up menus.
TotalView's Basic Look and Feel
|
Accelerator Keys
Short Cut:
- In addition to selecting actions from menus, you can also use
TotalView's predefined accelerator keys to initiate most of the
debugger's common functions.
- Saves time by skipping menu navigation.
- You can always find out which accelerator key to use by viewing the
menu for the action - accelerator keys are shown on the right side of
the menu where applicable.
Examples:
TotalView's Basic Look and Feel
|
Scrolling, Resizing and Memorizing
Conventional Scrolling Behavior:
- Conventional scrollbars are used by most of TotalView's windows, pages
and panes.
- Scrolling can be accomplished by clicking and/or dragging with
with the left mouse button.
- The usual up-arrow, down-arrow, page up and page down keys can also
be used for scrolling.
Resizing Windows and Panes:
- All windows can be resized in the usual Xwindows fashion by dragging
window borders with the mouse to a new size/position.
- The Process Window panes can be also be resized by clicking and
dragging on any resize widget.
Memorizing Windows:
- The "Window" menu (if present) will allow you
to save the position and size of that window, or all windows.
- A convenience feature for those who like to have their TotalView sessions
customized.
- Resized panes inside a window are not memorized.
|
 |
TotalView's Basic Look and Feel
|
Process and Thread State Codes
- TotalView uses colored single character State Codes to describe process and
thread status information. These are also called Status Codes.
- These codes appear in several places. One example is the Root Window,
shown below.
- The table below lists TotalView's state codes.
State Code |
Description |
B |
Stopped at a breakpoint |
E |
Stopped because of an error |
H |
In a Hold state |
K |
Thread is executing within the kernel |
M |
Mixed - some threads in a process are running and some not |
R |
Running |
T |
Thread is stopped |
W |
At a watchpoint |
TotalView's Basic Look and Feel
|
X Resources
- Historically, TotalView was a typical XWindows based GUI. However,
since TotalView version 5, Etnus has been moving away from using X
resource specifications as a means to control the look and feel of
TotalView.
- Instead, users are being encouraged to set attributes and behaviors
through the Preferences Dialog Box (discussed later) and/or through
TotalView's tvd initialization files.
- Some X resources are still used, with the caveat they may be
"deprecated" - that is, they may not be supported in future releases.
Finding out the names of supported X resources is difficult, as they
have, for the most part, been removed from TotalView documentation
(with the exeception of the Visualizer X resesources).
TotalView's Basic Functions
|
Viewing Source Code
Source, Assembler or Both:
- The Process Window's Source Pane is used to display source code,
assembler code or both.
- TotalView will attempt to display the source code by default.
If it can not find the source, then it will display assembler.
- Assembler can also be displayed with symbolic addresses rather
than absolute addresses (default).
- To toggle between the different display modes:
- An example of both source and assembler is shown below.
Displaying Function / File Source Code:
- Complex applications can include many different source files and
many different functions. TotalView makes finding and displaying
the source code for any of these easy:
- A Function/File dialog box will then appear (below).
Enter the name of the function or file desired. If found, TotalView
will display its source in the Source Code Pane of the Process Window.
- Diving on a function will also cause TotalView to update the Source
Pane with that function's source.
- If the function name is ambiguous (there are multiple occurrences),
TotalView will open an Ambiguous Function Dialog Box and ask you to
select from a list of possible functions.
TotalView's Basic Functions
|
Setting a Breakpoint
What Is a Breakpoint?
- A breakpoint is the most basic of TotalView's action points used to
control a program's execution. It causes a process/thread to halt execution
at the line number - prior to executing that line number.
- Breakpoints can be set in source code and assembler code.
- For regular source, only "boxed" line numbers are eligible for breakpoints.
For assembler, only assembler instructions displaying a "gridget" are
eligible.
- TotalView has three other types of action points (discussed later):
- Process barrier point
- Evaluation point
- Data watchpoint
Several Ways to Set / Unset a Breakpoint:
- Method 1: The easiest way to set a breakpoint is to
simply click on a source code line number with the left mouse button.
A red STOP icon will then appear on the source line number, as shown
at right.
- Method 2: Right mouse click anywhere on the desired source line
until the pop-up menu appears (right). Then select Set Breakpoint.
- Method 3: First, click on a source line to select it (make sure
it's highlighted). Then use:
- Method 4: For any arbitrary line number, use the path below.
A dialog box will then open to prompt you for the line number.
- To unset the breakpoint, simply click on the red STOP icon or select
"delete" from the pop-up menu or Action Point menu.
|
 |
Viewing Breakpoints:
- TotalView displays breakpoint information in several locations, as
shown below:
- As a "STOP" icon on the selected source line number
- Within the Action Points Pane
- Within the Action Point Properties Dialog Box
- In the Process Window's status bars
- In the Root Window state code column
- Within the Threads Pane (not shown)
Breakpoint Options:
- TotalView provides a means for selecting how breakpoints behave
across multi-process / multi-threaded programs. This topic is
further discussed in Part 3: Debugging Parallel
Codes.
TotalView's Basic Functions
|
Controlling Execution
- Controlling the execution of a program within TotalView involves two
decisions:
- Selecting the appropriate command
- Deciding upon the scope of the chosen command
- Both of these are performed via the
Process
Window, and are discussed below.
Execution Control Commands:
- TotalView enables you to control program execution three different ways:
- Whichever of the three methods you choose, the same basic commands
apply. The table below describes the basic execution control commands.
Command |
Description |
Go |
Start/resume execution |
Halt |
Stop execution |
Kill |
Terminate the job |
Next |
Run to next source line or instruction. If the next
line/instruction
calls a function, the entire function will be executed and control will
return to the next source line or instruction (the function is
"stepped over"). |
Step |
Run to next source line or instruction. If the next
line/instruction
calls a function, the function will be "stepped into". Execution will
stop within the function. |
Out |
Execute to the completion of a function. Returns to
the instruction after the one which called the function. |
Run To |
Allows you to arbitrarily click on any source line and
then run to that point |
Next Instruction |
Similar to Next, but applies only to machine instructions |
Step Instruction |
Similar to Step, but applies only to machine instructions |
Hold/Release |
Hold ignores other commands to resume execution
Release allows other run commands to have effect |
Restart |
Restarts a running program, or one that has stopped without exiting |
Set PC |
Sets the Program Counter to a desired source line, machine instruction,
or absolute address |
Group, Process, Thread Command Scopes:
- For serial programs, execution scope is not an issue because there
is only one execution stream. For parallel programs, execution scope
is critical – you need to know which processes and/or threads your
execution command will effect.
- Most of TotalView's execution control commands can be applied at
the Group, Process or Thread scoping level. The right scope depends
upon what you want to effect.
- Command scope can be selected from the execution scope drop-down
menu located next to the execution control keys (shown above) or
for the appropriate command on the Group, Process and Thread
drop-down menus.
- Group scope:
- Executes the command on all processes within a specified group.
- Usually used for a multi-process parallel program.
- Can be used for a single process serial program (number of group
members = one).
- Process scope:
- For a multi-process program, executes the command on a single
selected process.
- Some subtleties in behavior exist if the process is multi-threaded,
but generally the command influences all threads owned by the process.
- Can be used for a single process serial program.
- Thread scope:
- Usually used to execute the command on a single thread of a
multi-threaded process.
- Behavior can differ between machine architectures.
- Can be used for single process, single threaded serial program also.
- Additional details about Group, Process and Thread commands usage are
discussed later in
Part 3: Debugging
Parallel Codes.
TotalView's Basic Functions
|
Diving
- TotalView enables you to view more detail about a data containing
object (such as an array variable) by "diving" into it.
- Diving can be accomplished by several different methods:
- Double left clicking on an object
- Right clicking on an object and then selecting Dive from
the resulting pop-up menu (if applicable)
- Selecting Dive from any window's View menu
(if applicable)
- Clicking on an object with the middle mouse button
- What happens when you dive on an object depends upon the object.
The table below describes most cases.
Object |
Where Object is Located |
What Happens |
Process or thread |
Root Window |
Process/thread is displayed in an existing Process Window. If none
exists, then a new Process Window appears for the selected
process/thread. |
Routine
| Process Window (in Stack Trace Pane) |
Stack Frame and Source Code panes in the Process Window
are updated with information for the selected routine. |
Subroutine |
Process Window (in Source Code Pane) |
Source code appears in the Process Window |
Pointer |
Process Window |
Referenced memory area appears in a new Variable Window |
Variable, array, address |
Process Window |
Variable contents appear in a new a Variable Window |
Element of an array or structure |
Variable Window |
Contents of element appear in the Variable Window.
Example of a "nested" dive. |
Source code line |
Process Window |
Pop-up menu appears |
Example:
Nested Dives and Undiving:
TotalView's Basic Functions
|
Viewing and Modifying Data
Viewing Data:
- TotalView allows you to view variables, registers, areas of memory and
machine instructions, as discussed below.
- Variables
- Method 1: Dive on any variable that appears in the Source Pane or
Stack Frame Pane of the Process Window.
- Method 2: Use the variable lookup command:
- Registers
- Dive on any register that appears in the Stack Frame Pane of the
Process Window.
- Memory Areas
- Use the variable lookup command:
- Then enter either a hexadecimal address (must start with 0x) for a
single location. Enter two hexadecimal addresses for a range.
- Machine Instructions
- Method 1: Dive into the address of an assembler instruction in
the
Process Window Source Pane. The instructions for the entire
function will display in a Variable Window.
- Method 2: Dive on the PC register in the Process Window Stack Frame
Pane. The instructions for the entire function will display in a
Variable Window.
- Leaving a Variable Window open allows you to perform runtime monitoring
of variables. TotalView will update its contents each time the program
is stopped.
Examples:
Modifying Variable Data:
- You can edit variables from within the Variable Window. Simply click on
the variable with the Select (left) mouse button. This will select
the variable for field editing.
- The Variable Window below demonstrates editing an array element.
Notice that the array element being edited is highlighted and
shows a field editor cursor.
- The modified variable has effect when the program resumes execution.
Arrays:
- For array data, TotalView provides several additional features:
- Displaying array slices
- Data filtering
- Data Sorting
- Array statistics
- Displaying Array Slices
- Used to display subsections of an array. Particularly useful if only a
small section of a large array is of interest.
- Can be entered in the Slice: field in the Variable Window.
- Syntax is lower_bound:upper_bound:stride and may be
specified for each dimension.
- Examples:
Fortran
| Slice: (1:5, 3:8)
|
C/C++
| Slice: [::2][1:20]
|
- Array Data Filtering
- Arrays containing data types of character, integer or floating
point can be filtered to display only desired data.
- Can be entered in the Filter: field in the Variable Window.
- Filtering can be:
- By arithmetic comparison
- For IEEE values
- By a range of values
- Within an expression
- See the TotalView documentation
for additional examples, syntax options and other important information.
- Examples:
Fortran
| Filter: .gt. 250
Filter: .eq. $nan
Filter: 7:512
|
C/C++
| Filter: >= 100
Filter: != $inf
Filter: 128:<1024
|
- Sorting Array Data
- Simply click on the Value bar in a Variable Window. The array will
sort in ascending order. Clicking again will cause it to sort
in descending order. Clicking a third time will return the array
to its original order.
- Note: Sorting takes place internal to TotalView and not actually
within your data.
- Array Statistics
- TotalView is able to display some basic statistics about an array.
- A window containing statistical information about your array will
appear - example below.
- See the TotalView documentation
for an explanation of the statistics fields.
Changing Variable Display Format:
- You can specify how your variable data should be displayed. This is
done through TotalViews Preferences dialog box.
- Select any one:
- The Preferences dialog box will appear. Select the Formatting tab to
change the way TotalView displays variables.
Changing Variable Data Types:
TotalView's Basic Functions
|
Text Editing and Searching
Text Editing:
- TotalView provides a basic field editor for use within certain debugger
fields and windows. Text which can be edited will be highlighted
and display a field editor cursor.
- Cutting and pasting can be accomplished by using the middle mouse
button or by selecting Cut, Copy, or Paste
from any window's
Edit pull-down menu.
Text Searching:
- Most TotalView windows will permit you to search for text strings.
Simply select Find from any window's
Edit pull-down menu.
- A dialog box will appear for you to enter the string to search for,
plus any search options, as shown at right.
- Select Find Again from the same
Edit menu to repeat a search.
|
|
TotalView's Basic Functions
|
Saving Window Contents
- Most TotalView windows enable you to save their contents as
ASCII text. You can also pipe the contents to UNIX shell commands.
- For windows with multiple panes, you have to save each pane individually.
- Make sure your mouse pointer is in the window or pane of interest. Then
select Save Pane from any window's
File pull-down menu.
A dialog box will then appear for your input, as shown below:
Saved contents of a Stack
Frame pane.
- Using the “Send To Pipe” option – allows you to direct the pane/window
contents to a UNIX shell command. The command is entered in the
File Name box. For example:
Unless specified otherwise, output from the command appears as stdout
in the window where you started TotalView.
TotalView's Basic Functions
|
Getting Help
- TotalView provides an extensive, web browser based online Help system.
- All primary TotalView windows have a Help pull-down menu
that includes access to Etnus's complete set of product documentation.
- Context sensitive help is available by left-clicking on an object and then
selecting "Help" from the Help pull-down menu, or hitting the F1
key.
- Additionally, many dialog boxes have a context-sensitive Help
button.
- The Help pull-down menu and general Help screen are shown below.
TotalView's Basic Functions
|
Exiting TotalView
- You can exit the debugger in several ways:
- From any window select File Menu > Exit
- Typing CTRL-q or CTRL-Q in any window
- Closing the Root Window via your window manager
- After selecting any of these ways to exit TotalView, you will be prompted
to confirm your choice to exit:
This concludes TotalView Part 1
Where would you like to go now?