Case study: CS000016

Case Study CS000016:

The problem: Regression Testing. Create a series of tests that prove software is operating correctly.

The solution:

Write test code that exercises all aspects of the software
Run the test code and capture its output.
Compare the output with previous runs to assure no changes

Note: You can cut and paste these code fragments into the code pane of the Glee interpreter and experiment as you go along to see the actual operations live.

The Glee code:

$$======================================================= 'test'#op{ $$ The Tester $$======================================================= ''=> #csexact => #csglee; '{',l__,'}' #x #crccsws => x__x; x__x=r__=>s_a_m_e; :?(s_a_m_e){'passed:'l__$} :: {('!'%%60)(13 #asc)('failed:') (l__)('; got:') (x__x)('; want:')(r__)$} ;}; $$======================================================= '='%%40=>b$;'Calibrating the tests:'$;b$; $$======================================================= {$R__$r (2.._2) + __=>x; 0=>y;x test y;}; $$------------------------------------------------------- {$R__$r 1 2 3 + (6 5 4) __=>x; 0=>y;x test y;}; $$------------------------------------------------------- {$R__$r _2 0 3 - __ =>x; 0=>y;x test y;}; $$------------------------------------------------------- $$======================================================= b$;'A test run after calibration (and code change):'$;b$; $$======================================================= {$R__$r (2.._2) + __ =>x; 65278 =>y;x test y;}; $$------------------------------------------------------- {$R__$r 1 3 2 + (6 5 4) __ =>x; 458752 =>y;x test y;}; $$------------------------------------------------------- {$R__$r _2 0 3 - __ =>x; 50330880 =>y;x test y;}; $$=======================================================

The Output:
======================================== Calibrating the tests: ======================================== !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! failed: (2.._2) + ; got:65278; want:0 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! failed: 1 2 3 + (6 5 4) ; got:458752; want:0 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! failed: _2 0 3 - ; got:50330880; want:0 ======================================== A test run after calibration (and code change): ======================================== passed: (2.._2) + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! failed: 1 3 2 + (6 5 4) ; got:235408897; want:458752 passed: _2 0 3 -

The play-by-play:

A sample test case:

{$R__$r 1 3 2 + (6 5 4) __ =>x; 458752 =>y;x test y;};. Test cases are written as blocks { ... }
$R__$r. Set up to capture the code to be tested. Between the $R and $r is the delimiter __.
1 3 2 + (6 5 4) __ =>x;. The code to be tested 1 3 2 + (6 5 4) is found by scanning to the delimiter ... (6 5 4) __. The result is just a character string which is stored in a variable =>x;
0 =>y;. When we study the tester, we will see that it will execute our test, run #CRCUCWS on the result (Cyclical Redundency Test sensitive to Upper Case (UC) and White Space (WS)). This is the strictest test we can perform. The result is an integer. Starting out, we don't know what that integer will be so we just guess 0 =>y;. x test y;. Giving test our code to be executed and the expected CRC, the test program does all the work.

The test program:

$$======================================================= 'test'#op{ $$ The Tester $$======================================================= ''=> #csexact => #csglee; '{',l__,'}' #x #crccsws =r__ =>s_a_m_e; :?(s_a_m_e){'passed:'l__$} :: {('!'%%60)(10 #asc)('failed:') (l__)('; got:') (x__x)('; want:')(r__)$} ;};

'test'#op{ $$ The Tester. Our testing program is named test and we implement it as an operator #op{.
''=> #csexact => #csglee;. We need to assure that the operating environment does not change from test to test. We do this by setting it deliberately. In real life (actually this is real life for the Glee regression testing), this environment conditioning might be more involved.
'{',l__,'}' #x #crccsws =>x__x;. We build up a string to execute. First we assure insulation with the block opening brace '{'. The left argument l__ contains the code under test which we catenate on and close the block by catenating on the right brace.'{',l__,'}'. This we execute #x. The result is passed to our CRC generator and the resulting integer is stored ... ,'}' #x #crccsws =>x__x;
x__x = r__ =>s_a_m_e;. We compare the test CRC with the expected CRC in the right argument r__ using the = operator. It checks for complete identity between Glee objects. The result is saved in s_a_m_e.. This is a boolean which will be #true on success and #false on failure.
:?(s_a_m_e){'passed:'l__$}. If :? we have #true in s_a_m_e we display the test annotated as having passed.{'passed:'l__$}. Note: In real life I don't care about what passes so I display nothing here.
:: {('!'%%60)(10 #asc)('failed:') (l__)('; got:') (x__x)('; want:')(r__)$}
:: Else ('!'%%60)(13 10 #asc)('failed:') do some shrieking. Then (l__)('; got:') (x__x)('; want:')(r__)$} show what we got and what we want.

Calibrating the tests:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! failed: (2.._2) + ; got:65278; want:0
Initially we fail all tests. We put in 0 for the CRCs which should never be right. So we take each of the CRCs we get and paste them into our test cases. From then on our series of cases must produce these CRCs or we must know why not.
{$R__$r 1 3 2 + (6 5 4) __ =>x; 458752 =>y;x test y;};. I illustrate making a change 1 2 3 to 1 3 2 which shrieks
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! failed: 1 3 2 + (6 5 4) ; got:235408897; want:458752
We can correct the error or accept it by changing the expected CRC in the test case.

I use this technique to regression test the entire Glee website. It allows me to play fast-and-loose with my code changes and still maintain confidence that I have not created failure where I once enjoyed success.

This completes the example. To better understand these operators and other things you can do with them, consult the operator pages according to the type of data you see being operated on.