check i2d_ASN1_TYPE return value

add a length check to the return value of function i2d_ASN1_TYPE. Return an error instead of trying to malloc a negative number.

CLA: trivial

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/14828)

crypto/asn1/a_strex.c

@@ -280,6 +280,8 @@ static int do_dump(unsigned long lflags, char_io *io_ch, void *arg,
     t.type = str->type;
     t.value.ptr = (char *)str;
     der_len = i2d_ASN1_TYPE(&t, NULL);
+    if (der_len <= 0)
+        return -1;
     if ((der_buf = OPENSSL_malloc(der_len)) == NULL) {
         ERR_raise(ERR_LIB_ASN1, ERR_R_MALLOC_FAILURE);
         return -1;