Verification: Did I do the thing right?
Validation: Did I do the right thing?
You verify that the system meets its specifications.
You validate that the system meets its user's requirements
Q: Can you pass verification and not pass validation? How?
Q: Can you pass validation and not pass verification? How?
Testing (many varieties)
Formal proofs
Prototyping
Expert opinion?
Design reviews
Code inspections
Pair programming
Static Analysis
Dynamic Analysis
Many software engineering tasks require you to work with a system (or part of one) that you are not familiar with.
Probably will be working with other people
May be hired into an existing project
May be hired as tester/maintainer of code you didn't write
May be forced to find a bug in someone else's code
May not remember your own code!
Analysis over the dynamic behavior of a system
But....how is this different from testing?
Dynamic analysis cares less about the correct result and more about how it got there.
Profiling
Coverage tools
Memory allocation monitors
Assert
Printf's
Debugger?
A call graph is a graph of procedure calls in a program.
Static analysis can generate a static call graph, simply by recognizing the procedure calls in each procedure.
Typically, one node for each procedure, and only one arc between any two procedures.
This graph shows you all procedure calls that might occur.
Dynamic analysis can capture a trace of all procedure calls.
This trace represents a call tree -- i.e., every actual procedure call is represented by an edge (the call) and a node (the invoked procedure).
Procedures in the static call graph can be 0, 1, or many nodes in the call tree.
The dynamic call tree can be collapsed using varying methods:
The last one above should produce the static call graph if the data was generated by good coverage test cases.
A dynamic analysis only shows you what happened, not what could happen.
A dynamic analysis is only as good as the tests cases used to generate the dynamic behavior
Assertions are (generally) point statements inline in a program (and in the programming language) that are expressions about what should be true when the program reaches that point.
In C/C++, an "assert()" macro defined in assert.h -- allows the assertion to be compiled in or left out (for optimization).
Assert statements are evaluated at run-time. Thus, they do not provide guarantees of correctness.
Is assert a dynamic analysis?
Yes: they are monitoring the dynamic behavior of the program without regard to the correctness of the program output.
No: they are checking the output of a piece of the program and thus are analagous to white-box testing.
The best text on assertions is Meyer's _Object Oriented Software Construction_, in the "Design by Contract" chapter
Assertions can be very powerful tools. However, 99.9% of their use is for basic checks -- null pointer, array bounds, etc.
Why is this?
Assertions can be "dangerous", for functional and performance reasons.
But we saw this under static analysis?
Yes, but the language is non-commital as to how it is used
Last class --> the ideal is to only specify the properties of your system once (if ever!).
The toolset from the JML people includes a run-time checker (i.e., dynamic analysis) but not a static checker!
Available at http://pag.csail.mit.edu/daikon
Began as a PhD topic for Michael Ernst (now a professor at MIT)
Daikon instruments your program, collects the program data, and tries to find invariant properties.
Daikon invariant:
Instrumentation is hard!
Many different approaches
A nice website on Program Transformation.
Found at http://valgrind.kde.org They have a quick overview, and a longer technical description.
Part of the KDE project. Originally focussed on analyzing memory usage and finding memory errors.
But supports writing your own analyses on top of its framework.
Framework is an x86 virtual machine!
As a VM, it can do whatever it wants to the program
Unfortunately, 5-100X slowdown on the program
Found at www.dyninst.org.
Allows dynamic insertion of binary code into binary executing processes.
Dynamically compiles instrumentation code, copies it and its variables into allocated memory, and then patches the original code with jumps to the instrumentation.
Also supports a scripting-language interface (Tcl).