p≡p engine
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

7.2 KiB

Building and Running the Tests for the p≡p Engine

Work in progress.

Caveat, before you begin

Right now, the engine tests only function on *nix-like systems (including MacOS).

(Conversion to Windows will require, at the very least, looking at some of the file-handling code. If you want to fix this, start by looking in Engine.cc in the test/src directory!)


In addition to the engine requirements, you will need:

  • cmake
  • python3
  • git (for getting the gtest-parallel repository, unless you grab the tarball from somewhere)

Building the prerequisites

The Engine test suite now requires (at least) two additional pieces to run:

  • googletest
  • gtest-parallel

How this proceeds depends on your platform and whether or not you use a packaged distribution.

These instructions do this with cmake. If you can manage it with bazel instead, more power to you ;)

Installing googletest

Packaged distributions

This is the currently preferred way to do this, because everyone was doing it anyway and who am I to judge?

Debian and Ubuntu (and derivatives)

Using the libgtest-dev is easy, but not straightforward. Unfortunately, the version of google test in Debian Buster is too old: it's version 1.8 and we require version 1.9. Version 1.9 is available in Debian Testing, but it is built with g++ 9.0, which is ABI incompatible with binaries built with g++ 8.0, which is in Debian stable. Specifically, gcc has changed the semantics of std::string with C++11 and using g++ 8.0 results in the errors like the following:

undefined reference to `std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()'

It's possible to install g++ 9.0 from testing to get the test suite working, but that breaks other things (at least for me -Neal). Instead, the easiest thing to do it to rebuild gtest for Debian Stable. This is straightforward:

$ sudo apt install build-essential cmake debhelper $ apt source -t testing libgtest-dev $ cd googletest- $ dpkg-buildpackage -us -uc ... $ sudo dpkg -i googletest_1.9.0.20190831-1_amd64.deb libgtest-dev_1.9.0.20190831-1_amd64.deb

That's it.

  1. Install package gtest from Macports

  2. Macports will build the libraries for you.

  3. In the next major section ("Building the Test Suite"), under "Makefile and local.conf", set GTEST_SRC_DIR to /opt/local/src/googletest in local.conf (see instructions below)

  4. Make sure /opt/local/lib is in your library path when compiling and linking the tests.

Downloading and compiling the source yourself

  1. Get the source, Fred. (Luke is tired of the source, I hear.)
git clone https://github.com/google/googletest.git
  1. Switch into the source directory and find the directory containing the src and include directories. Mark this directory for later. (For me, this is ./googletest/googletest)

  2. Edit CMakeLists.txt here to contain the following line at the top:


(If you don't, it won't compile, and I will shake my fist at you.)

  1. Execute, in this directory:
cmake CMakeLists.txt
  1. In the lib directory of your current directory are located the library files you'll use (lib/*.a). Copy or symlink them to the library location of your choice (make sure this is a directory that can be seen during the test build process - i.e. one that's in one of the library paths used in building. Mine are located in $HOME/lib.

  2. See Makefile and local.conf under "Building the test suite" below - In this scenario, I set GTEST_SRC_DIR as <clone_path>/googletest/googletest (i.e. the absolute path of where the src and include directories were above - for me, /Users/krista/googletest/googletest).

Installing gtest-parallel

Pick a source directory and put your gtest-parallel source there (e.g. via git clone https://github.com/google/gtest-parallel.git).

We'll deal more with this when preparing to compile the test suite.

Building the test suite

Makefile and local.conf

So local.conf in the top-level engine directory is where we stick all of the Makefile overrides. The test Makefile contains some defaults for relevant variables here, but if you need to override them, please either create or modify local.conf in the top-level engine directory as needed. The relevant variables are:

  • GTEST_SRC_DIR: This is the directory where you compiled googletest above (defaults to /usr/src/gtest)

  • GTEST_INC_DIR: This is where the include files for googletest are located (defaults to $(GTEST_SRC_DIR)/include)

  • GTEST_PL: This is the full path to the python file for gtest_parallel (default presumes you cloned it under src in your home directory, i.e. it is $(HOME)/src/gtest-parallel/gtest_parallel.py)


Presuming the above works, then from the top test directory, simply run make.

Running the test suite

To simply run the test suite and see what tests fail...

Do one of:

  1. make test OR

  2. python3 <path to gtest-parallel.py> ./EngineTests

To run individual test suites, especially for debugging purposes

Note that for some test suites, this will, if something goes dreadfully wrong, mean that one test's failure may pollute another test. This generally means you have found a dastardly bug in the engine, but it can also be a test issue.

Caveat lector.

  1. To run sequentially, in the same process:
./EngineTests --gtest_filter=TestSuiteName*

For example, for DeleteKeyTest:

./EngineTests DeleteKeyTest*
  1. To debug the same with lldb:
lldb ./EngineTests -- --gtest_filter=TestSuiteName*
  1. To debug with gdb:
gdb --args ./EngineTests --gtest_filter=TestSuiteName*

To run and/or debug individual test cases

  1. To run:
./EngineTests --gtest_filter=TestSuiteName.test_function_name

For example, for check_delete_single_pubkey in DeleteKeyTest:

./EngineTests DeleteKeyTest.check_delete_single_pubkey
  1. To debug the same with lldb:
lldb ./EngineTests -- --gtest_filter=TestSuiteName.test_function_name
  1. To debug with gdb:
gdb --args ./EngineTests --gtest_filter=TestSuiteName.test_function_name

When debugging a failing test, use '--gtest_break_on_failure' to have gtest automatically break into the debugger where the assertion fails.

Creating new tests

Script next on the agenda...

Known Problems

There are 5 (maybe 6) tests that currently fail which are the function of broken tests, not the engine. They are on the agenda to fix (expired keys inside imported mails, etc). You'll see something like this - please ignore for this particular second in time:

    170 ms: ./EngineTests SubkeyRatingEvalTest.check_subkey_rating_eval_no_es
     170 ms: ./EngineTests SubkeyRatingEvalTest.check_subkey_rating_eval_revoked_sign_no_alt
     253 ms: ./EngineTests SubkeyRatingEvalTest.check_subkey_rating_eval_weak_e_strong_ecc_se
    1189 ms: ./EngineTests KeyResetMessageTest.check_non_reset_receive_revoked
    1376 ms: ./EngineTests KeyResetMessageTest.check_reset_receive_revoked