p≡p engine fork for my own dirty testing of stuff
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

207 lines
6.6 KiB

  1. # Building and Running the Tests for the p≡p Engine
  2. Work in progress... Currently have the debian build/run instructions in.
  3. ## Caveat, before you begin
  4. Right now, the engine tests only function on \*nix-like systems (including
  5. MacOS).
  6. *(Conversion to Windows will require, at the very least, looking at some of the
  7. file-handling code. If you want to fix this, start by looking in Engine.cc
  8. in the test/src directory!)*
  9. ## Requirements
  10. In addition to the engine requirements, you will need:
  11. * `cmake`
  12. * `python3`
  13. * `git` (for getting the `gtest-parallel` repository, unless you grab the
  14. tarball from somewhere)
  15. ## Building the prerequisites
  16. The Engine test suite now requires (at least) two additional pieces to run:
  17. * `googletest`
  18. * `gtest-parallel`
  19. How this proceeds depends on your platform and whether or not you use a packaged
  20. distribution.
  21. These instructions do this with `cmake`. If you can manage it with `bazel`
  22. instead, more power to you ;)
  23. ### Installing `googletest`
  24. #### Packaged distributions
  25. This is the currently preferred way to do this, because everyone was doing it
  26. anyway and who am I to judge?
  27. ##### Debian and Ubuntu (and derivatives)
  28. Thanks to Erik Smistad for this starting point (condensed from [Getting Started
  29. with Google Test On
  30. Ubuntu](https://www.eriksmistad.no/getting-started-with-google-test-on-ubuntu/)):
  31. 1. Install the packages `cmake` and `libgtest-dev` from the repository. This
  32. will install the gtest source files to `/usr/src/gtest`. You'll still need to
  33. compile the code and link the library files to be able to use them.
  34. 2. Compile the source files:
  35. ```
  36. cd /usr/src/gtest
  37. sudo cmake CMakeLists.txt
  38. sudo make
  39. ```
  40. 3. Copy/symlink the libraries to the library location of your choice (here,
  41. it's `/usr/lib`, hence the `sudo`, but as long as it's in your library path,
  42. it shouldn't matter where you stick it):
  43. ```
  44. sudo cp *.a /usr/lib
  45. ```
  46. ##### MacOS
  47. I am totally guessing for now - this is a placeholder - that
  48. macports gtest install will do the same. Will need to find the directories this
  49. goes in. Guessing `/opt/local/src`.
  50. #### Downloading and compiling the source yourself
  51. 1. Get the source, Fred. (Luke is tired of the source, I hear.)
  52. ```
  53. git clone https://github.com/google/googletest.git
  54. ```
  55. 2. Switch into the source directory and find the directory
  56. containing the `src` and `include` directories. Mark this directory
  57. for later. (For me, this is `./googletest/googletest`)
  58. 3. Edit `CMakeLists.txt` here to contain the following line at the top:
  59. ```
  60. set (CMAKE_CXX_STANDARD 11)
  61. ```
  62. (If you don't, it won't compile, and I will shake my fist at you.)
  63. 4. Execute, in this directory:
  64. ```
  65. cmake CMakeLists.txt
  66. make
  67. ```
  68. 5. In the lib directory of your current directory are located the
  69. library files you'll use. Copy or symlink them to the library location
  70. of your choice (make sure this is a directory that can be seen during
  71. the test build process - i.e. one that's in one of the library paths
  72. used in building. Mine are located in `$HOME/lib`.
  73. 6. See `Makefile` and `local.conf` under "Building the test suite" -
  74. In this scenario, I set GTEST_SRC_DIR googletest/googletest under the
  75. directory I checked out googletest in (i.e. the absolute path of where
  76. the `src` and `include` directories were above - for me,
  77. `/Users/krista/googletest/googletest`).
  78. ### Installing `gtest-parallel`
  79. Pick a source directory and put your `gtest-parallel` source there
  80. (e.g. via `git clone https://github.com/google/gtest-parallel.git`).
  81. We'll deal more with this when preparing to compile the test suite.
  82. ## Building the test suite
  83. ### `Makefile` and `local.conf`
  84. So `local.conf` in the top-level engine directory is where we stick all of the
  85. Makefile overrides. The test Makefile contains some defaults for relevant
  86. variables here, but if you need to override them, please either create or modify
  87. `local.conf` in the top-level engine directory as needed. The relevant variables
  88. are:
  89. * `GTEST_SRC_DIR`: This is the directory where you compiled googletest above
  90. (defaults to `/usr/src/gtest`)
  91. * `GTEST_INC_DIR`: This is where the include files for googletest are located
  92. (defaults to `$(GTEST_SRC_DIR)/include`)
  93. * `GTEST_PL`: This is the full path to the *python file* for `gtest_parallel`
  94. (default presumes you cloned it under `src` in your home directory, i.e. it is
  95. `$(HOME)/src/gtest-parallel/gtest_parallel.py`)
  96. ### Building
  97. Presuming the above works, then from the top test directory, simply run make.
  98. ## Running the test suite
  99. ### To simply run the test suite and see what tests fail...
  100. Do one of:
  101. 1. `make test` OR
  102. 2. `python3 <path to gtest-parallel.py> ./EngineTests`
  103. ### To run individual test suites, especially for debugging purposes
  104. Note that for some test suites, this will, if something goes dreadfully wrong,
  105. mean that one test's failure may pollute another test. This generally means you
  106. have found a dastardly bug in the engine, but it can also be a test issue.
  107. *Caveat lector*.
  108. 1. To run sequentially, *in the same process*:
  109. ```
  110. ./EngineTests --gtest_filter=TestSuiteName*
  111. ```
  112. For example, for `DeleteKeyTest`:
  113. ```
  114. ./EngineTests DeleteKeyTest*
  115. ```
  116. 2. To debug the same with lldb:
  117. ```
  118. lldb ./EngineTests -- --gtest_filter=TestSuiteName*
  119. ```
  120. 3. To debug with gdb:
  121. ```
  122. gdb --args ./EngineTests --gtest_filter=TestSuiteName*
  123. ```
  124. ### To run and/or debug individual test cases
  125. 1. To run:
  126. ```
  127. ./EngineTests --gtest_filter=TestSuiteName.test_function_name
  128. ```
  129. For example, for `check_delete_single_pubkey` in `DeleteKeyTest`:
  130. ```
  131. ./EngineTests DeleteKeyTest.check_delete_single_pubkey
  132. ```
  133. 2. To debug the same with lldb:
  134. ```
  135. lldb ./EngineTests -- --gtest_filter=TestSuiteName.test_function_name
  136. ```
  137. 3. To debug with gdb:
  138. ```
  139. gdb --args ./EngineTests --gtest_filter=TestSuiteName.test_function_name
  140. ```
  141. # Creating new tests
  142. Script next on the agenda...
  143. # Known Problems
  144. There are 5 (maybe 6) tests that currently fail which are the function of
  145. broken tests, not the engine. They are on the agenda to fix (expired
  146. keys inside imported mails, etc). You'll see something like this - please
  147. ignore for this particular second in time:
  148. ```
  149. 170 ms: ./EngineTests SubkeyRatingEvalTest.check_subkey_rating_eval_no_es
  150. 170 ms: ./EngineTests SubkeyRatingEvalTest.check_subkey_rating_eval_revoked_sign_no_alt
  151. 253 ms: ./EngineTests SubkeyRatingEvalTest.check_subkey_rating_eval_weak_e_strong_ecc_se
  152. 1189 ms: ./EngineTests KeyResetMessageTest.check_non_reset_receive_revoked
  153. 1376 ms: ./EngineTests KeyResetMessageTest.check_reset_receive_revoked
  154. ```