A local copy of OpenSSL from GitHub
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

170 lines
6.1 KiB

  1. Using OpenSSL Tests
  2. ===================
  3. After a successful build, and before installing, the libraries should be tested.
  4. Run:
  5. $ make test # Unix
  6. $ mms test ! OpenVMS
  7. $ nmake test # Windows
  8. **Warning:** you MUST run the tests from an unprivileged account
  9. (or disable your privileges temporarily if your platform allows it).
  10. If some tests fail, take a look at the section Test Failures below.
  11. Test Failures
  12. -------------
  13. If some tests fail, look at the output. There may be reasons for the failure
  14. that isn't a problem in OpenSSL itself (like an OS malfunction or a Perl issue).
  15. You may want increased verbosity, that can be accomplished like this:
  16. Full verbosity, showing full output of all successful and failed test cases
  17. (`make` macro `VERBOSE` or `V`):
  18. $ make V=1 test # Unix
  19. $ mms /macro=(V=1) test ! OpenVMS
  20. $ nmake V=1 test # Windows
  21. Verbosity on failed (sub-)tests only
  22. (`VERBOSE_FAILURE` or `VF` or `REPORT_FAILURES`):
  23. $ make test VF=1
  24. Verbosity on failed (sub-)tests, in addition progress on succeeded (sub-)tests
  25. (`VERBOSE_FAILURE_PROGRESS` or `VFP` or `REPORT_FAILURES_PROGRESS`):
  26. $ make test VFP=1
  27. If you want to run just one or a few specific tests, you can use
  28. the make variable TESTS to specify them, like this:
  29. $ make TESTS='test_rsa test_dsa' test # Unix
  30. $ mms/macro="TESTS=test_rsa test_dsa" test ! OpenVMS
  31. $ nmake TESTS='test_rsa test_dsa' test # Windows
  32. And of course, you can combine (Unix examples shown):
  33. $ make test TESTS='test_rsa test_dsa' VF=1
  34. $ make test TESTS="test_cmp_*" VFP=1
  35. You can find the list of available tests like this:
  36. $ make list-tests # Unix
  37. $ mms list-tests ! OpenVMS
  38. $ nmake list-tests # Windows
  39. Have a look at the manual for the perl module Test::Harness to
  40. see what other HARNESS_* variables there are.
  41. To report a bug please open an issue on GitHub, at
  42. <https://github.com/openssl/openssl/issues>.
  43. For more details on how the `make` variables `TESTS` can be used,
  44. see section Running Selected Tests below.
  45. Running Selected Tests
  46. ----------------------
  47. The `make` variable `TESTS` supports a versatile set of space separated tokens
  48. with which you can specify a set of tests to be performed. With a "current
  49. set of tests" in mind, initially being empty, here are the possible tokens:
  50. alltests The current set of tests becomes the whole set of available
  51. tests (as listed when you do 'make list-tests' or similar).
  52. xxx Adds the test 'xxx' to the current set of tests.
  53. -xxx Removes 'xxx' from the current set of tests. If this is the
  54. first token in the list, the current set of tests is first
  55. assigned the whole set of available tests, effectively making
  56. this token equivalent to TESTS="alltests -xxx".
  57. nn Adds the test group 'nn' (which is a number) to the current
  58. set of tests.
  59. -nn Removes the test group 'nn' from the current set of tests.
  60. If this is the first token in the list, the current set of
  61. tests is first assigned the whole set of available tests,
  62. effectively making this token equivalent to
  63. TESTS="alltests -xxx".
  64. Also, all tokens except for "alltests" may have wildcards, such as *.
  65. (on Unix and Windows, BSD style wildcards are supported, while on VMS,
  66. it's VMS style wildcards)
  67. ### Examples
  68. Run all tests except for the fuzz tests:
  69. $ make TESTS='-test_fuzz*' test
  70. or, if you want to be explicit:
  71. $ make TESTS='alltests -test_fuzz*' test
  72. Run all tests that have a name starting with "test_ssl" but not those
  73. starting with "test_ssl_":
  74. $ make TESTS='test_ssl* -test_ssl_*' test
  75. Run only test group 10:
  76. $ make TESTS='10' test
  77. Run all tests except the slow group (group 99):
  78. $ make TESTS='-99' test
  79. Run all tests in test groups 80 to 99 except for tests in group 90:
  80. $ make TESTS='[89]? -90' test
  81. To run specific fuzz tests you can use for instance:
  82. $ make test TESTS='test_fuzz_cmp test_fuzz_cms'
  83. To stochastically verify that the algorithm that produces uniformly distributed
  84. random numbers is operating correctly (with a false positive rate of 0.01%):
  85. $ ./util/wrap.sh test/bntest -stochastic
  86. Running Tests in Parallel
  87. -------------------------
  88. By default the test harness will execute the selected tests sequentially.
  89. Depending on the platform characteristics, running more than one test job in
  90. parallel may speed up test execution.
  91. This can be requested by setting the `HARNESS_JOBS` environment variable to a
  92. positive integer value. This specifies the maximum number of test jobs to run in
  93. parallel.
  94. Depending on the Perl version different strategies could be adopted to select
  95. which test recipes can be run in parallel. In recent versions of Perl, unless
  96. specified otherwise, any task can be run in parallel. Consult the documentation
  97. for `TAP::Harness` to know more.
  98. To run up to four tests in parallel at any given time:
  99. $ make HARNESS_JOBS=4 test
  100. Randomisation of Test Ordering
  101. ------------------------------
  102. By default, the test harness will execute tests in the order they were added.
  103. By setting the `OPENSSL_TEST_RAND_ORDER` environment variable to zero, the
  104. test ordering will be randomised. If a randomly ordered test fails, the
  105. seed value used will be reported. Setting the `OPENSSL_TEST_RAND_ORDER`
  106. environment variable to this value will rerun the tests in the same
  107. order. This assures repeatability of randomly ordered test runs.
  108. This repeatability is independent of the operating system, processor or
  109. platform used.
  110. To randomise the test ordering:
  111. $ make OPENSSL_TEST_RAND_ORDER=0 test
  112. To run the tests using the order defined by the random seed `42`:
  113. $ make OPENSSL_TEST_RAND_ORDER=42 test