Test  Exercise 2007
 

Deadline: 15/5 2007 12.00

Two standard options are available. If you have different wishes please discuss this with the instructor - it must be approved.

Objective

The objective is to evaluate the usability and effectiveness of two approaches to automated real-time testing: Manually assisted test purpose based test generation and full automatic randomized online testing. Each group is given a UppAal specification of the light controller, and an (assumed!) correct implementation in java, including two example mutants. The experiment proceeds in two phases: Preparation phase and evaluation phase. 

In the preparation phase, each group will independently (manually) generate a set of test cases and a set of mutated implementations based on the given specification and implementation. Each group runs their test suite on their own mutants, and possibly extend it as desired. The mutants must be non-conforming to the specification, ie. killable by some test case.   

In the evaluation phase, each group swaps mutants (java byte-code) and runs 

  1. their own test suite against the groups own and other groups mutants, collecting statistics with the results (which own/other mutants were killed). 

  2. applies  UppAal-TRON (randomized online testing) to the collected mutants (e.g 10 runs per mutant)., again collecting statistics about the percentage of runs killing the mutants, and preferable duration of the test runs.

Expectations. 

Each group is to hand in the following deliverables to be evaluated. 

The deliverables must be signed by each student who, by his signature, declares active and equal participation in preparation of the deliverables

Software

  1. Java sources for Light Controller is distributed with the TOV exercise package.  Requires > java 1.5:  /pack/j2sdk-1.5.0.01/bin/java)
  2. TRON. The windows version is included in the exercise package. If you wish to use Linux, please dewnload and install according to the Tron  webpage. Then read the instructions in the readme.html file in the exercise package to start tron.
  3. Du skal bruge denne øvelsespakke. Tron til windows er inkluderet (hvis du vil bruge Linux evt. homer.cs og marge.cs) så kan du hente tron fra Tron  websiden. For at aktivere tron skal du følge instruktionerne i README.html i øvelsespakken.

    1. Prøv at lave en eller flere mutanter (logisk eller timing som er synlig på black-box niveau iht. rtioco) af LightController.java (oversæt) og se om fejlen fanges.  
    2. Generer vha test-purposes og offline test generering et par sekvenser, og kod dem op som input sekvenser til tron som eksemplificeret i automaten (CoverageUser) i lightContr4.xml i øvelsespakken

Instructions (execution of offline generated tests):

We recommend executing your preset input sequences using TRON by modeling the test input sequence as a timed automaton and by replacing the environment with this automaton, and executing tron in eager mode. An example is provided in LightContr4.xml (Template: LightCov and Envy Closure. See system section). Start tron as described below, or try eager option:

Start tron: using  "SocketAdaptor" adaptor,  using test specification "lightContr4.xml", testing  a java lightcontroller-implementation running on host ip number (can also be either "localhost", a dns-name, or an ipnumber, -P options indicates that uppaal-TRON should select inputs at random times (within environment behavior:
 

Instructions (Real-time problems):

Here are some tips that may reduce real-time scheduling problems (alternatively use simulated time)

  1. use lightly loaded machines
  2. use local isolated, single user computers, and and perferable 2 cpu machine (or two machines interconnected via cross cable)
  3. use lightly loaded multi-cpu machine
  4. use Linux Kernel 2.6 (marge, not homer)
  5. Disable "Nagle's" algorithm by setting the socket option: setTcpNoDelay(true); This should already be done in all the applications and adaptors you will work with, including the "TestLightController" application.
  6. Do not X-tunnel the lightcontroller gui. If you cannot run the lightcontroller locally, try to remove GUI from Java Lightcontroller implementation by uncomment the code that create the graphics objects: run the light controller with -N option.
     

Brian