openssl/test

How to add recipes

==================



For any test that you want to perform, you write a script located in

test/recipes/, named {nn}-test_{name}.t, where {nn} is a two digit number and

{name} is a unique name of your choice.



Please note that if a test involves a new testing executable, you will need to

do some additions in test/build.info. Please refer to the section "Changes to 

test/build.info" below.





Naming conventions

=================



A test executable is named test/{name}test.c



A test recipe is named test/recipes/{nn}-test_{name}.t, where {nn} is a two

digit number and {name} is a unique name of your choice.



The number {nn} is (somewhat loosely) grouped as follows:



00-04  sanity, internal and essential API tests

05-09  individual symmetric cipher algorithms

10-14  math (bignum)

15-19  individual asymmetric cipher algorithms

20-24  openssl commands (some otherwise not tested)

25-29  certificate forms, generation and verification

30-35  engine and evp

60-79  APIs:

   60  X509 subsystem

   61  BIO subsystem

   65  CMP subsystem

   70  PACKET layer

80-89  "larger" protocols (CA, CMS, OCSP, SSL, TSA)

90-98  misc

99     most time consuming tests [such as test_fuzz]





A recipe that just runs a test executable

=========================================



A script that just runs a program looks like this:



    #! /usr/bin/perl



    use OpenSSL::Test::Simple;



    simple_test("test_{name}", "{name}test", "{name}");



{name} is the unique name you have chosen for your test.



The second argument to `simple_test' is the test executable, and `simple_test'

expects it to be located in test/



For documentation on OpenSSL::Test::Simple, do

`perldoc util/perl/OpenSSL/Test/Simple.pm'.





A recipe that runs a more complex test

======================================



For more complex tests, you will need to read up on Test::More and

OpenSSL::Test.  Test::More is normally preinstalled, do `man Test::More' for

documentation.  For OpenSSL::Test, do `perldoc util/perl/OpenSSL/Test.pm'.



A script to start from could be this:



    #! /usr/bin/perl



    use strict;

    use warnings;

    use OpenSSL::Test;



    setup("test_{name}");



    plan tests => 2;                # The number of tests being performed



    ok(test1, "test1");

    ok(test2, "test1");



    sub test1

    {

        # test feature 1

    }



    sub test2

    {

        # test feature 2

    }





Changes to test/build.info

==========================



Whenever a new test involves a new test executable you need to do the

following (at all times, replace {NAME} and {name} with the name of your

test):



* add {name} to the list of programs under PROGRAMS_NO_INST



* create a three line description of how to build the test, you will have

to modify the include paths and source files if you don't want to use the

basic test framework:



    SOURCE[{name}]={name}.c

    INCLUDE[{name}]=.. ../include ../apps/include

    DEPEND[{name}]=../libcrypto libtestutil.a



Generic form of C test executables

==================================



    #include "testutil.h"



    static int my_test(void)

    {

        int testresult = 0;                 /* Assume the test will fail    */

        int observed;



        observed = function();              /* Call the code under test     */

        if (!TEST_int_eq(observed, 2))      /* Check the result is correct  */

            goto end;                       /* Exit on failure - optional   */



        testresult = 1;                     /* Mark the test case a success */

    end:

        cleanup();                          /* Any cleanup you require      */

        return testresult;

    }



    int setup_tests(void)

    {

        ADD_TEST(my_test);                  /* Add each test separately     */

        return 1;                           /* Indicate success             */

    }



You should use the TEST_xxx macros provided by testutil.h to test all failure

conditions.  These macros produce an error message in a standard format if the

condition is not met (and nothing if the condition is met).  Additional

information can be presented with the TEST_info macro that takes a printf

format string and arguments.  TEST_error is useful for complicated conditions,

it also takes a printf format string and argument.  In all cases the TEST_xxx

macros are guaranteed to evaluate their arguments exactly once.  This means

that expressions with side effects are allowed as parameters.  Thus,



    if (!TEST_ptr(ptr = OPENSSL_malloc(..)))



works fine and can be used in place of:



    ptr = OPENSSL_malloc(..);

    if (!TEST_ptr(ptr))



The former produces a more meaningful message on failure than the latter.