Excessive time spent checking DSA keys and parameters

Vulnerability Description

Issue summary: Checking excessively long DSA keys or parameters may be very slow. Impact summary: Applications that use the functions EVP_PKEY_param_check() or EVP_PKEY_public_check() to check a DSA public key or DSA parameters may experience long delays. Where the key or parameters that are being checked have been obtained from an untrusted source this may lead to a Denial of Service. The functions EVP_PKEY_param_check() or EVP_PKEY_public_check() perform various checks on DSA parameters. Some of those computations take a long time if the modulus (`p` parameter) is too large. Trying to use a very large modulus is slow and OpenSSL will not allow using public keys with a modulus which is over 10,000 bits in length for signature verification. However the key and parameter check functions do not limit the modulus size when performing the checks. An application that calls EVP_PKEY_param_check() or EVP_PKEY_public_check() and supplies a key or parameters obtained from an untrusted source could be vulnerable to a Denial of Service attack. These functions are not called by OpenSSL itself on untrusted DSA keys so only applications that directly call these functions may be vulnerable. Also vulnerable are the OpenSSL pkey and pkeyparam command line applications when using the `-check` option. The OpenSSL SSL/TLS implementation is not affected by this issue. The OpenSSL 3.0 and 3.1 FIPS providers are affected by this issue.

References

• https://www.openssl.org/news/secadv/20240516.txt
• https://github.com/openssl/openssl/commit/3559e868e58005d15c6013a0c1fd832e51c73397
• https://github.com/openssl/openssl/commit/9c39b3858091c152f52513c066ff2c5a47969f0d
• https://github.com/openssl/openssl/commit/da343d0605c826ef197aceedc67e8e04f065f740
• https://github.com/openssl/openssl/commit/53ea06486d296b890d565fb971b2764fcd826e7e