tls_process_{client,server}_certificate(): allow verify_callback return > 1

Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/13937)

CHANGES.md

@@ -292,6 +292,15 @@ breaking changes, and mappings for the large list of deprecated functions.
 
  * Deprecated the obsolete X9.31 RSA key generation related functions.
 
+ * While a callback function set via `SSL_CTX_set_cert_verify_callback()`
+   is not allowed to return a value > 1, this is no more taken as failure.
+
+   *Viktor Dukhovni and David von Oheimb*
+
+ * Deprecated the obsolete X9.31 RSA key generation related functions
+   BN_X931_generate_Xpq(), BN_X931_derive_prime_ex(), and
+   BN_X931_generate_prime_ex().
+
    *Tomáš Mráz*
 
  * The default key generation method for the regular 2-prime RSA keys was

doc/man3/SSL_CTX_set_cert_verify_callback.pod

@@ -32,11 +32,11 @@ By setting I<callback> to NULL, the default behaviour is restored.
 
 I<callback> should return 1 to indicate verification success
 and 0 to indicate verification failure.
-In server mode, a return value other than 1 leads to handshake failure.
+In server mode, a return value of 0 leads to handshake failure.
 In client mode, the behaviour is as follows.
-A return value greater than 1 leads to handshake failure.
-Other values are ignored if the verification mode is B<SSL_VERIFY_NONE>.
-On return value 0 the handshake will fail.
+All values, including 0, are ignored
+if the verification mode is B<SSL_VERIFY_NONE>.
+Otherwise, when the return value is 0, the handshake will fail.
 
 In client mode I<callback> may also return -1,
 typically on failure verifying the server certificate.

ssl/statem/statem_clnt.c

@@ -1884,10 +1884,6 @@ WORK_STATE tls_post_process_server_certificate(SSL *s, WORK_STATE wst)
         return WORK_ERROR;
     }
     ERR_clear_error();          /* but we keep s->verify_result */
-    if (i > 1) {
-        SSLfatal(s, SSL_AD_HANDSHAKE_FAILURE, i);
-        return WORK_ERROR;
-    }
 
     /*
      * Inconsistency alert: cert_chain does include the peer's certificate,

ssl/statem/statem_srvr.c

@@ -3524,10 +3524,6 @@ MSG_PROCESS_RETURN tls_process_client_certificate(SSL *s, PACKET *pkt)
                      SSL_R_CERTIFICATE_VERIFY_FAILED);
             goto err;
         }
-        if (i > 1) {
-            SSLfatal(s, SSL_AD_HANDSHAKE_FAILURE, i);
-            goto err;
-        }
         pkey = X509_get0_pubkey(sk_X509_value(sk, 0));
         if (pkey == NULL) {
             SSLfatal(s, SSL_AD_HANDSHAKE_FAILURE,