I. Unit testing with JUnit A. what is unit testing? ------------------------------------------ WHAT IS UNIT TESTING? def: *testing* is the process of showing that a program works for certain inputs def: a *unit* is a module, or small set of modules. In Java, a unit is a class or interface, or a small set of them. E.g., an interface and 3 classes that implement it. def: *unit testing* is testing of a unit. ------------------------------------------ Do you get more confidence by running more test data? B. goals and motivation ------------------------------------------ WHY UNIT TESTING? Code isn't right if it's not tested Practical: - most programmers rely on testing - e.g., Microsoft has 1 tester per developer - you could get work as a tester Divide and conquer: - split system into units - debug units individually - narrow down places where bugs can be - don't want to chase down bugs in other units Support regression testing: - so can make changes to lots of code and know if you broke something - can make big changes with confidence ------------------------------------------ C. how to do unit testing ------------------------------------------ HOW TO DO UNIT TESTING Build system in layers - start with classes that don't depend on others - continue testing building on already tested classes Benefit: - avoids having to write stubs - when testing a module, ones it depends on are reliable ------------------------------------------ How does low coupling help testing? How does high coupling hurt it? 1. example without JUnit ------------------------------------------ PROGRAM TO TEST public class ISqrt { /** Return an integer approximation to the square root of y. */ public static int isqrt(int y) { int guess = 1; while (guess * guess < y) { guess++; } return guess; } } ------------------------------------------ ------------------------------------------ CONVENTIONAL TESTING /** Test ISqrt. */ public class TestISqrtNoJUnit { /** Run the tests. */ public static void main(String[] args) { printTestResult(0); printTestResult(1); printTestResult(2); printTestResult(3); printTestResult(4); printTestResult(7); printTestResult(9); printTestResult(100); } private static void printTestResult(int arg) { System.out.print("isqrt("); System.out.print(arg + ") ==> "); System.out.println( ISqrt.isqrt(arg)); } } ------------------------------------------ ------------------------------------------ CONVENTIONAL TEST OUTPUT isqrt(0) ==> 1 isqrt(1) ==> 1 isqrt(2) ==> 2 isqrt(3) ==> 2 isqrt(4) ==> 2 isqrt(7) ==> 3 isqrt(9) ==> 3 isqrt(100) ==> 10 ------------------------------------------ What does this say about the code? Is it right? What's the problem with this kind of test output? 2. example with JUnit ------------------------------------------ import junit.framework.*; import junit.textui.*; /** Test ISqrt. */ public class TestISqrt extends TestCase { /** Run the tests. */ public static void main(String[] args) { TestRunner.run(suite()); } /** Test isqrt. */ public void testIsqrt() { // line 26 below assertEquals(0, ISqrt.isqrt(0)); assertEquals(1, ISqrt.isqrt(1)); assertEquals(1, ISqrt.isqrt(2)); assertEquals(1, ISqrt.isqrt(3)); assertEquals(2, ISqrt.isqrt(4)); assertEquals(2, ISqrt.isqrt(7)); assertEquals(3, ISqrt.isqrt(9)); assertEquals(10, ISqrt.isqrt(100)); } /** Returns the test suite for this test class. */ public static Test suite() { return new TestSuite(TestISqrt.class); } } ------------------------------------------ ------------------------------------------ COMPILATION AND OUTPUT (formatted) $ javac ISqrt.java TestISqrt.java $ java TestISqrt .F Time: 0.03 There was 1 failure: 1) testIsqrt(TestISqrt) junit.framework.AssertionFailedError: expected:<0> but was:<1> at TestISqrt.testIsqrt (TestISqrt.java:26) at sun.reflect.NativeMeth... at sun.reflect.NativeMeth... at sun.reflect.Delegating... at TestISqrt.main (TestISqrt.java:15) FAILURES!!! Tests run: 1, Failures: 1, Errors: 0 ------------------------------------------ Is this better? Why? 3. exercise for students ------------------------------------------ FOR YOU TO DO Write a JUnit test class for testing public class ForYou { /** Return the minimum of x and y. */ public static int min(int x, int y) { ... } } === By filling in the following: === import junit.framework.*; import junit.textui.*; /** Test ForYou. */ public class TestForYou extends TestCase { /** Test min. */ public void testMin() { } /* ... rest as before ... */ } ------------------------------------------ D. using the JUnit framework 1. definitions a. test case, success and failure ------------------------------------------ SOME TERMINOLOGY def: A *test case* for a method M is a pair (o, args) where o is not null and M can be sent to o, args is a tuple of arguments that can be passed to M def: A test case, (o, args), for M *succeeds* iff it o.M(args) behaves as expected. def: A test case, (o, args), for M *fails* iff it does not behave as expected. ------------------------------------------ Why should o not be null? If M has a bug that is revealed by a test case, does that test case for M succeed or fail? b. test code terms ------------------------------------------ PARTS OF TEST CODE def: The *test fixture* is the set of variables used in testing. def: The *test driver* is the class that runs the tests def: The *test oracle* for a test case is the code that decides success or failure for that test case. ------------------------------------------ What in the code we saw so far was the test driver? The oracle? What difference is there between JUnit testing and non JUnit testing in what we saw before? 2. basic usage ------------------------------------------ BASIC USAGE OF JUNIT (1) To test a type T: 1. Write a class like: import junit.framework.*; import junit.textui.*; /** Test of class T. */ public class TestT extends TestCase { /** Run the tests. */ public static void main(String[] args) { TestRunner.run(suite()); } /** Returns the test suite for this test class. */ public static Test suite() { return new TestSuite(TestT.class); } } ------------------------------------------ ------------------------------------------ BASIC USAGE OF JUNIT (2) 2. Compile T.java and TestT.java $ javac T.java TestT.java 3. Run the JUnit graphical user interface on TestT $ java junit.swingui.TestRunner TestT OR Run the text interface (good from makefiles) $ java TestT 4. Look at the failures and errors ------------------------------------------ 3. extended example from StickSync ------------------------------------------ package sticksync; import java.io.BufferedWriter; import java.io.File; import java.io.FileWriter; import java.io.IOException; import java.io.Writer; import java.util.Date; import junit.framework.TestCase; public class TestLocation extends TestCase { private File[] testFiles; private Location[] testLocs; public TestLocation(String name) { super(name); } public void testConstructors() { for (int i=0; i < testFiles.length; i++) { Location l1 = new Location(testFiles[i] .getPath()); assertEquals(l1.getPath(), testLocs[i].getPath()); } } public void testCopyTo() throws IOException { String orig0 = testLocs[0].contentsAsString(); String orig1 = testLocs[1].contentsAsString(); assertTrue(!orig0.equals(orig1)); Date d = testLocs[0].copyTo(testLocs[1]); assertEquals(orig0, testLocs[1].contentsAsString()); // line 76 below assertEquals(d,testLocs[0].modTime()); assertEquals(d,testLocs[1].modTime()); } // SETUP AND TEARDOWN protected void setUp() throws Exception { testFiles = configureTestFiles(2); testLocs = new Location[testFiles.length]; for (int i=0; i but was: at ...TestLocation.testCopyTo( TestLocation.java:76) at sun.reflect.NativeMethod... at sun.reflect.NativeMethod... at sun.reflect.Delegating... at ...TestLocation.main( TestLocation.java:38) FAILURES!!! Tests run: 5, Failures: 1, Errors: 0 ------------------------------------------ ------------------------------------------ class Location { /* ... */ /** Copies this location's file to the given location's file. */ public Date copyTo(Location loc) throws IOException { InputStream source = new BufferedInputStream( new FileInputStream(getFile())); OutputStream dest = new BufferedOutputStream( new FileOutputStream( loc.getFile())); byte[] buf = new byte[256]; int count = 0; while ((count = source.read(buf)) != -1) { dest.write(buf,0,count); } source.close(); dest.close(); getFile().setLastModified( loc.modTime().getTime()); return new Date(); } } ------------------------------------------ E. other aspects of JUnit 1. naming conventions ------------------------------------------ NAMING CONVENTIONS Test methods start with "test" e.g., testCopyTo, testIsqrt Test classes start with "Test" e.g., TestLocation ------------------------------------------ 2. test suites ------------------------------------------ TEST SUITES Organize tests into a larger test. Help with automation of testing. See the JUnit documentation for details. ------------------------------------------ II. Design by Contract with JML A. motivation 1. design by contract ------------------------------------------ DESIGN BY CONTRACT A way of recording: - details of method responsibilities - avoiding constantly checking arguments - assigning blame across interfaces Example: //@ requires y >= 0; //@ ensures (* \result is root of y *); Contracts in software: Obligations of client - passes positive integer Rights of client - gets square root approximation Rights of implementor - assumes argument is positive Obligations of implementor - must compute and return square root ------------------------------------------ ------------------------------------------ PRE AND POSTCONDITIONS def: a method's *precondition* says what must be true to call it. Example: //@ requires y >= 0; def: a method's *normal postcondition* says what is true when it returns (without throwing an exception) Example: //@ ensures \result * \result == y; def: an *exceptional postcondition* says what is true when a method throws an exception. /*@ signals (IllegalArgumentException e) @ e != null && y <= 0; @*/ ------------------------------------------ Did you see these in fully dressed use cases? Which is the obligation of the client? Of the implementor? Which is the rights of the client? Of the implementor? ------------------------------------------ RELATIONAL MODEL OF METHODS Think of a method as a relation: Inputs <--> Outputs precondition describes the domain postcondition describes the relationships ------------------------------------------ 2. documentation ------------------------------------------ CONTRACTS AS DOCUMENTATION For each method say: - what it requires (if anything) - what it ensures Example in JML: //@ requires y >= 0; /*@ ensures -y <= \result @ && \result <= y @ && \result * \result <= y @ && y < (Math.abs(\result) + 1) @ * (Math.abs(\result) + 1); @*/ public static int isqrt(int y) { ... } Contracts are: - more abstract than code - not necessarily constructive - often machine checkable (in JML) and so can help with debugging - machine checkable contracts can always be up-to-date In summary, good documentation ------------------------------------------ How is constructing a square root different than the above spec? ------------------------------------------ ABSTRACTION BY SPECIFICATION A contract can be satisfied in many ways: E.g., for square root: - linear search - binary search - Newton's method ... These will have varying non-functional properties - efficiency - memory usage So a contract abstracts from all these implementations ==> can change implementations later ------------------------------------------ 3. blame assignment, modularity ------------------------------------------ MORE ADVANTAGES OF CONTRACTS Blame assignment: Who is to blame if: - precondition doesn't hold? - postcondition doesn't hold? Avoids inefficient defensive checks: /*@ requires a != null @ && (* a is sorted *); @*/ int binarySearch(Thing [] a, Thing x) { ... } ------------------------------------------ ------------------------------------------ MODULARITY OF REASONING Typical OO code: ... source.close(); dest.close(); getFile() .setLastModified( loc.modTime().getTime()); return modTime(); How to understand this code? - read the code for all methods? - read the contracts for all methods? ------------------------------------------ What if some methods are recursive? What if there is polymorphism? ------------------------------------------ RULES FOR REASONING Client code - must work for every implementation that satisifies contract - can thus only use the contract (not the code!) - must establish precondition - gets to assume postcondition //@ assert 9 >= 0; int res = ISqrt.isqrt(9); //@ assert res * res == 9; Implementation code: - must satisfy contract - gets to assume precondition - must establish postcondition - but can do anything permitted by it ------------------------------------------ ------------------------------------------ CONTRACTS AND INTENT Code makes a poor contract, because can't separate: - what is intended (contract) - what is an implementation decision E.g., if floating point square root routine gives an approximation good to 3 decimal places, can that be changed in the next release? By contrast, contracts: - allow vendors to specify intent - allow vendors freedom to change details - tell clients what they can count on ------------------------------------------ What kinds of changes might vendors want to make that don't break existing contracts? B. what is JML? ------------------------------------------ JML Stands for "Java Modeling Language" Design by contract for Java Available from www.jmlspecs.org Uses Java 1.3 (and 1.4) ------------------------------------------ C. annotations ------------------------------------------ ANNOTATIONS JML specifications are contained in annotations, which are comments like: //@ ... or /*@ ... @ ... @*/ At-signs (@) on the beginnings of lines are ignored within annotations ------------------------------------------ What's the advantage of using annotations They look like comments to Java, and are ignored by javac D. simple examples 1. informal specs (organize specifications) ------------------------------------------ INFORMAL DESCRIPTIONS An informal description looks like: (* some text describing a property *) It is treated as a boolean by JML. Allows: - escape from formality - organization of English as contracts Example: public class ISqrt { //@ requires (* y is positive *); /*@ ensures (* \result is an @ int approximation to @ the square root of y *) @ && \result >= 0; @*/ public static int isqrt(int y) { ... } } ------------------------------------------ Note that informal descriptions can be combined with formal stuff, as in the postcondition above. ------------------------------------------ FOR YOU TO DO Write informal pre and postconditions for the other operations of this type: public class Person { private String name; private int weight; /*@ also @ ensures \result != null @ && (* \result is a display @ form of this person *); @*/ public String toString() { return "Person(\"" + name + "\"," + weight + ")"; } public int getWeight() { return weight; } public void addKgs(int kgs) { weight += kgs; } public Person(String n) { name = n; weight = 0; } } ------------------------------------------ Explain "also" ... My answer would be something like: public class Person { private String name; private int weight; /*@ also @ ensures \result != null @ && (* \result is a display @ form of this person *); @*/ public String toString() { return "Person(\"" + name + "\"," + weight + ")"; } /*@ ensures (* \result is @ this person's weight *); @*/ public int getWeight() { return weight; } /*@ requires kgs >= 0; @ ensures (* this person's weight @ is their old weight plus @ kgs *); @*/ public void addKgs(int kgs) { weight += kgs; } /*@ requires n != null; @ ensures (* This person has @ name n and weight 0 *); @*/ public Person(String n) { name = n; weight = 0; } } 2. formalization Q: Are the informal specifications longer than the code sometimes? Are the informal specifications longer than the code sometimes? ------------------------------------------ WRITING FORMAL SPECIFICATIONS IN JML Formal assertions are written as Java expressions, but: - Cannot have side effects - no use of =, ++, -- - can only call "pure" methods - Can use some extensions to Java: Syntax meaning ===================================== \result result of method call a ==> b a implies b b <== a b is implied by a a <==> b a iff b \old(E) value of E in pre-state ------------------------------------------ ------------------------------------------ public class Person { private /*@ spec_public non_null @*/ String name; private /*@ spec_public @*/ int weight; /*@ public invariant !name.equals("") @ && weight >= 0; @*/ //@ also //@ ensures \result != null; public String toString() { return "Person(\"" + name + "\"," + weight + ")"; } /*@ ensures \result == weight; @*/ public /*@ pure @*/ int getWeight() { return weight; } /*@ ensures kgs >= 0 @ && weight @ == \old(weight + kgs); @ signals (IllegalArgumentException e) @ kgs < 0; @*/ public void addKgs(int kgs) { weight += kgs; } /*@ requires n != null @ && !n.equals(""); @ ensures n.equals(name) @ && weight == 0; @*/ public Person(String n) { name = n; weight = 0; } } ------------------------------------------ ------------------------------------------ MEANING OF ENSURES AND SIGNALS |-----------| |------------| | | | | [ Normal | [ Exceptional| | (Return) | | (Throws) | [ O | [ O | | | | | | | | | [ | [ | | | | | |-----------| |------------| /------------/ /------------/ / ensures / /signals(...)/ / kgs >= 0 ... / kgs < 0; / /------------- /------------/ ------------------------------------------ 3. invariants ------------------------------------------ INVARIANTS Are properties that are always true of an object's state, when control is not inside the object's methods /*@ public invariant !name.equals("") @ && weight >= 0; @*/ Allow you to define: - acceptable states of an object - "consistency" of an object's state ------------------------------------------ 4. exercise for students ------------------------------------------ FOR YOU TO DO Formally specify the method (in Person) public void changeName(String newName) { name = newName; } Hint: watch out for the invariant! ------------------------------------------ E. basic tool usage 1. overview of tools ------------------------------------------ TOOLS FOR JML Runtime assertion checking compiler: jmlc Java interpreter with RAC support: jmlrac JUnit test oracle generator: jmlunit JUnit test runner with RAC support: jml-junit Documentation (HTML) generator: jmldoc Type checker: jml ------------------------------------------ 2. jmlc ------------------------------------------ The Runtime Assertion Checking Compiler jmlc Usage: $ jmlc Person.java produces Person.class $ jmlc -Q *.java produces *.class, quietly $ jmlc -d ../bin Person.java produces ../bin/Person.class ------------------------------------------ ------------------------------------------ RUNNING CODE COMPILED WITH JMLC Must have JML's jmlruntime.jar in Java's boot class path Automatic if you use script jmlrac $ jmlrac PersonMain ------------------------------------------ ------------------------------------------ A MAIN PROGRAM public class PersonMain { public static void main(String [] argv) { System.out.println(new Person(null)); System.out.println(new Person("")); } } ------------------------------------------ ------------------------------------------ EXAMPLE (formatted) $ jmlc -Q PersonMain.java Person.java $ jmlrac PersonMain org.jmlspecs.jmlrac.runtime .JMLEntryPreconditionError: By method "Person" of class "Person" for assertions specified at Person.java:34:22, when 'n' is null at Person.checkPre$$init$$Person( Person.java:855) at Person.(Person.java:44) at PersonMain.main(PersonMain.java:7) Exception in thread "main" ------------------------------------------ Any questions about the tools? F. more examples, from StickSync ------------------------------------------ STICKSYNC EXAMPLES (MORE REALISTIC CONTRACTS) class Location { /** The pathname of a file. */ private /*@ spec_public non_null @*/ String path; /** Caches the associated file. * Access only via getFile(). * This is null if no File is cached. * @see getFile() */ private transient File fileCache; /** Returns the associated file */ //@ ensures \result != null; private File getFile() { if (fileCache == null) { fileCache = new File(path); } return fileCache; } /** Initializes this Location. * @param fullpath the full pathname * to the file. */ //@ requires fullpath != null; //@ assignable path; //@ ensures path.equals(fullpath); public Location(String fullpath) { path = fullpath; } /** Is the file readable? */ /*@ ensures \result @ == getFile().canRead(); @*/ public /*@ pure @*/ boolean readable() { return getFile().canRead(); } /** Copies this location's file * to the given location's file. */ //@ requires loc != null; //@ ensures \result != null; /*@ ensures \result.equals(modTime()) @ && loc.modTime().equals(modTime()); @*/ public Date copyTo(Location loc) throws IOException { ... } } ------------------------------------------ Can you read these? To what extent do these specifications add to the javadocs? G. other things in JML 1. model fields and represents clauses ------------------------------------------ MODEL FIELDS, REPRESENTS CLAUSES What if you want to change a spec_public field's name? private /*@ spec_public non_null @*/ String path; to private /*@ non_null @*/ String fullPath; For specifications: - need to keep the old name public - but don't want two Strings So use a model variable: //@ public model non_null String path; and a represents clause //@ private represents path <- fullPath; ------------------------------------------ ------------------------------------------ MODEL VARIABLES Are specification-only variables. Like domain-level constructs. Given value only by represents clauses: path abstract, model ^ | represented by | fullPath concrete (real) ------------------------------------------ 2. web site for more info III. JML and JUnit based unit testing A. motivation 1. problem of writing test oracle code ------------------------------------------ WRITING TEST ORACLES Lots of test cases ==> don't want to write oracle for each Duplicates documentation, specifications ==> so hard to maintain Easy to not make them abstract ==> so depends on representation ------------------------------------------ B. idea ------------------------------------------ IDEA JML specifications can be checked at runtime. If the specifications are complete, then they can serve as test oracles: - precondition descibes what test cases are acceptable for testing a method - postcondition describes expected behavior So: 1. Compile code with jmlc 2. Test driver monitors assertion violations a. Entry precondition violations ==> reject test data b. Other violations ==> test fails (code has a bug) ------------------------------------------ What happens if the precondition isn't specified? What happens if the postcondtion isn't specified? C. simple example 1. using jmlunit tool ------------------------------------------ USING JMLC AND JMLUNIT TO DO TESTING Compile code with jmlc $ jmlc ISqrt.java produces ISqrt.class Generate JUnit test driver and test case files $ jmlunit ISqrt.java produces ISqrt_JML_Test.java and ISqrt_JML_TestCase.java ------------------------------------------ ------------------------------------------ WHAT'S IN THE _JML_Test*.java FILES // This file was generated by jmlunit ... public abstract class ISqrt_JML_Test extends junit.framework.TestCase { /* ... */ /** Initializes the test data variables. * This must be overridden * by subclasses. */ protected abstract void setUp(); protected void tearDown() {} /** Test if the override of setUp * has initialized all the fixture * variables in the fixture. */ public void test$FixtureInitialization() { /* ... */ } /** Test to see if the code * was compiled with jmlc. */ public void test$IsRACCompiled() { /* ... */ } ------------------------------------------ ------------------------------------------ /** Harness for testing the * isqrt method. */ public void testIsqrt() { final int[] y = this.vint; for (int i$0 = 0; y != null && i$0 < y.length; i$0++) { try { ISqrt.isqrt(y[i$0]); } catch (JMLEntryPreconditionError e$) { // ... note meaningless test input continue; } catch (JMLAssertionError e$) { // ... note test failure } catch (java.lang.Throwable e$) { // test success continue; } finally { tearDown(); setUp(); } } } ------------------------------------------ ------------------------------------------ /** Receiver objects for methods * to be tested. */ protected ISqrt[] receivers; /** Data of type int for use in testing. */ protected int[] vint; } ------------------------------------------ 2. adding test data ------------------------------------------ ADDING TEST DATA Edit test data into ISqrt_JML_TestCase public class ISqrt_JML_TestCase extends ISqrt_JML_Test { /* ... */ protected void setUp() { /* ... */ try { receivers = new ISqrt[] { new ISqrt(), }; vint = new int[] { 0, 1, 7, 8, 9, 10, -1, -5, 17, 20, 24, 99, 100, 101, }; } finally { /* ... */ } } ------------------------------------------ What test data would you use for testing Person's addKgs method? 3. running the test, output ------------------------------------------ RUNNING THE TESTS Compile the files: $ jmlc ISqrt.java $ javc ISqrt_JML_Test*.java Then run the tests either by: $ jml-junit ISqrt_JML_TestCase OR $ jmlrac ISqrt_JML_TestCase ------------------------------------------ ------------------------------------------ OUTPUT (formatted) jmlrac ISqrt_JML_TestCase ...F Time: 0.032 There was 1 failure: 1) testIsqrt(ISqrt_JML_TestCase) junit.framework.AssertionFailedError: Method 'isqrt' applied to Receiver class ISqrt Argument 'y' (vint[0]): 0 Caused org...JMLPostconditionError By method "isqrt" of class "ISqrt" for assertions specified at ISqrt.java:4:22, when 'y' is 0 '\result' is 1 at ISqrt_JML_Test.testIsqrt (ISqrt_JML_Test.java:87) at sun.reflect.NativeMethodAc... at sun.reflect.NativeMethodAc... at sun.reflect.DelegatingMeth... at ISqrt_JML_Test.run (ISqrt_JML_Test.java:22) at org....JMLTestRunner.run (JMLTestRunner.java) at ISqrt_JML_TestCase.main (ISqrt_JML_TestCase.java:17) FAILURES!!! Tests run: 3, Failures: 1, Errors: 0 JML test runs: 1/1 (meaningful/total) ------------------------------------------ D. conclusions ------------------------------------------ SUMMARY Unit Testing - should be done in layers - should be systematic, automated ==> repeatable, efficient (in human time) - JUnit helps do that for Java programs Design by Contract - documents assumptions and obligations - Specifications more abstract than code - Specifications that are checked: - will be up to date - can help in isolating bugs (blame assignment) JML/JUnit combination: - Use specifications as test oracles - Limitation: how complete the specification is. ------------------------------------------