p≡p engine fork for my own dirty testing of stuff
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

271 lines
8.7 KiB

2 years ago
2 years ago
2 years ago
  1. # Building and Running the Tests for the p≡p Engine
  2. Work in progress.
  3. ## Caveat, before you begin
  4. Right now, the engine tests only function on \*nix-like systems (including
  5. MacOS).
  6. *(Conversion to Windows will require, at the very least, looking at some of the
  7. file-handling code. If you want to fix this, start by looking in Engine.cc
  8. in the test/src directory!)*
  9. ## Requirements
  10. In addition to the engine requirements, you will need:
  11. * `cmake`
  12. * `python3`
  13. * `git` (for getting the `gtest-parallel` repository, unless you grab the
  14. tarball from somewhere)
  15. ## Building the prerequisites
  16. The Engine test suite now requires (at least) two additional pieces to run:
  17. * `googletest`
  18. * `gtest-parallel`
  19. How this proceeds depends on your platform and whether or not you use a packaged
  20. distribution.
  21. These instructions do this with `cmake`. If you can manage it with `bazel`
  22. instead, more power to you ;)
  23. ### Installing `googletest`
  24. #### Packaged distributions
  25. This is the currently preferred way to do this, because everyone was doing it
  26. anyway and who am I to judge?
  27. ##### Debian and Ubuntu (and derivatives)
  28. Thanks to Erik Smistad for this starting point (condensed from [Getting Started
  29. with Google Test On
  30. Ubuntu](https://www.eriksmistad.no/getting-started-with-google-test-on-ubuntu/)):
  31. 1. Install the packages `cmake` and `libgtest-dev` from the repository. This
  32. will install the gtest source files to `/usr/src/gtest`. You'll still need to
  33. compile the code and link the library files to be able to use them.
  34. 2. Compile the source files:
  35. ```
  36. cd /usr/src/gtest
  37. sudo cmake CMakeLists.txt
  38. sudo make
  39. ```
  40. 3. Copy/symlink the libraries to the library location of your choice (here,
  41. it's `/usr/lib`, hence the `sudo`, but as long as it's in your library path,
  42. it shouldn't matter where you stick it):
  43. ```
  44. sudo cp *.a /usr/lib
  45. ```
  46. ##### MacOS
  47. 1. Install package `gtest` from Macports
  48. 2. Macports will build the libraries for you.
  49. 3. In the next major section ("Building the Test Suite"), under
  50. "Makefile and local.conf", set `GTEST_SRC_DIR` to
  51. `/opt/local/src/googletest` in `local.conf` (see instructions below)
  52. 4. Make sure `/opt/local/lib` is in your library path when compiling and
  53. linking the tests.
  54. #### Downloading and compiling the source yourself
  55. 1. Get the source, Fred. (Luke is tired of the source, I hear.)
  56. ```
  57. mkdir -p ~/pep-src/googletest
  58. git clone https://github.com/google/googletest.git ~/pep-src/googletest
  59. ```
  60. 2. Switch into the source directory and find the subdirectory of
  61. `googletest` (the source distribution of googletest also contains a
  62. `googlemock` library which is not useful to us here)
  63. containing the `src` and `include` directories. Remember this
  64. directory's path. For me it is `~/pep-src/googletest/googletest` .
  65. 3. Edit the file `CMakeLists.txt` in that directory, adding the
  66. following line at the top:
  67. ```
  68. set (CMAKE_CXX_STANDARD 11)
  69. ```
  70. (If you don't, it won't compile, and I will shake my fist at you.)
  71. 4. Go to the googletest source directory (it should be the parent
  72. directory of the directory containing the file you edited) and build
  73. the library.
  74. ```
  75. cd ~/pep-src/googletest
  76. cmake CMakeLists.txt
  77. make
  78. ```
  79. 5. In the lib directory of your current directory are located the
  80. library files you'll use (`lib/*.a`). You may leave them there without
  81. installing them, if you accept running a more complex command line than
  82. ```
  83. make test
  84. ```
  85. later. If you want to install the library, read on.
  86. Copy or symlink them to the library
  87. location of your choice (make sure this is a directory that can be seen
  88. during the test build process - i.e. one that's in one of the library paths
  89. used in building. Mine are located in `$HOME/lib`.
  90. See `Makefile` and `local.conf` under "Building the test suite" below -
  91. In this scenario, I set `GTEST_SRC_DIR` as `<clone_path>/googletest/googletest`
  92. (i.e. the absolute path of where the `src` and `include` directories were
  93. above - for example, `/Users/krista/googletest/googletest`).
  94. ### Installing `gtest-parallel`
  95. Pick a source directory and put your `gtest-parallel` source there
  96. (e.g. via `git clone https://github.com/google/gtest-parallel.git`):
  97. ```
  98. mkdir -p ~/pep-src/gtest-parallel
  99. git clone https://github.com/google/gtest-parallel.git ~/pep-src/gtest-parallel
  100. ```
  101. This library is written in Python and does not require any actual build.
  102. We'll deal more with this when preparing to compile the test suite.
  103. ## Building the test suite
  104. ### `Makefile` and `local.conf`
  105. So `local.conf` in the top-level engine directory is where we stick all of the
  106. Makefile overrides. The test Makefile contains some defaults for relevant
  107. variables here, but if you need to override them, please either create or modify
  108. `local.conf` in the top-level engine directory as needed. The relevant variables
  109. are:
  110. * `GTEST_SRC_DIR`: This is the directory where you compiled googletest above
  111. (defaults to `/usr/src/gtest`)
  112. * `GTEST_INC_DIR`: This is where the include files for googletest are located
  113. (defaults to `$(GTEST_SRC_DIR)/include`)
  114. * `GTEST_PL`: This is the full path to the *python file* for `gtest_parallel.py`
  115. (default presumes you cloned it under `src` in your home directory, i.e. it is
  116. `$(HOME)/src/gtest-parallel/gtest_parallel.py`)
  117. The sample `local.conf` included in `../doc/build-debian.md` contains correct
  118. definitions for these variables, assuming the user installed from sources under
  119. `~/pep-src` as described here.
  120. ### Building
  121. Presuming the above works and you installed every library, then from the top test
  122. directory, simply run ``make``.
  123. Libraries which have not been installed into the `lib` subdirectory of some
  124. standard prefix will not be found automatically. But this problem is easy to
  125. circumbent by setting environment libraries (on system using the GNU
  126. linked-loader `LD_LIBRARY_PATH` for dynamic libraries, `LIBRARY_PATH` for static
  127. libraries).
  128. For example, this should be sufficient to run the test suite under a debian
  129. system following the instruction at `../doc/build-debian.md`:
  130. ```
  131. cd ~/pep-src/pep-engine
  132. LIBRARY_PATH=$LIBRARY_PATH:$HOME/pep-src/googletest/lib LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/pep-src/pep-engine/src:$HOME/pep-src/sequoia/target/release:$HOME/pep-src/libetpan/build/lib make test
  133. ```
  134. Such variable definitions may be prepended to the command lines below for running
  135. individual tests or the entire test suite.
  136. In case of test failures remember to build the database, on which the test suite
  137. depends, and to copy it into the system directory:
  138. ```
  139. make dbinstall
  140. ```
  141. The database does not need to be rebuilt and reinstalled for every test suite
  142. run: running the `dbinstall` target once suffices.
  143. ## Running the test suite
  144. ### To simply run the test suite and see what tests fail...
  145. Do one of:
  146. 1. `make test` OR
  147. 2. `python3 <path to gtest-parallel.py> ./EngineTests`
  148. ### To run individual test suites, especially for debugging purposes
  149. Note that for some test suites, this will, if something goes dreadfully wrong,
  150. mean that one test's failure may pollute another test. This generally means you
  151. have found a dastardly bug in the engine, but it can also be a test issue.
  152. *Caveat lector*.
  153. 1. To run sequentially, *in the same process*:
  154. ```
  155. ./EngineTests --gtest_filter=TestSuiteName*
  156. ```
  157. For example, for `DeleteKeyTest`:
  158. ```
  159. ./EngineTests DeleteKeyTest*
  160. ```
  161. 2. To debug the same with lldb:
  162. ```
  163. lldb ./EngineTests -- --gtest_filter=TestSuiteName*
  164. ```
  165. 3. To debug with gdb:
  166. ```
  167. gdb --args ./EngineTests --gtest_filter=TestSuiteName*
  168. ```
  169. ### To run and/or debug individual test cases
  170. 1. To run:
  171. ```
  172. ./EngineTests --gtest_filter=TestSuiteName.test_function_name
  173. ```
  174. For example, for `check_delete_single_pubkey` in `DeleteKeyTest`:
  175. ```
  176. ./EngineTests DeleteKeyTest.check_delete_single_pubkey
  177. ```
  178. 2. To debug the same with lldb:
  179. ```
  180. lldb ./EngineTests -- --gtest_filter=TestSuiteName.test_function_name
  181. ```
  182. 3. To debug with gdb:
  183. ```
  184. gdb --args ./EngineTests --gtest_filter=TestSuiteName.test_function_name
  185. ```
  186. N.B. The gtest_filter can be globbed and will run all matching tests; if you
  187. want to run every test in a test suite, be sure to use TestSuiteName*.
  188. (Different shells will require different quoting styles for this - YMMV.)
  189. When debugging a failing test, use '--gtest_break_on_failure' to have
  190. gtest automatically break into the debugger where the assertion fails.
  191. ### Output
  192. Compile tests with -DDEBUG_OUTPUT to (possibly) see output to cout. May only work
  193. in the tests run directly from ./EngineTests (with or without filter)
  194. # Creating new tests
  195. Script next on the agenda...
  196. # Known Problems
  197. The normal run of the tests in parallel eats output. Try running the individual test case as above if you need to see a test case's output.