[Bug]: RFC 5246 violation: Invalid TLS Handshake Message Ordering Is Accepted

[GSoJC234]

Contact Details

thkighie1224@postech.ac.kr

Version

v5.8.4-stable

Description

Hello wolfSSL team,

During protocol conformance testing, I identified several cases where wolfSSL accepts invalid TLS 1.2 handshake message orderings, in violation of RFC 5246.

Specifically, certain mandatory handshake messages are either omitted or reordered. Although the handshake does not continue successfully once these messages reach their corresponding processing routines, the implementation still allows the invalid messages to enter those routines. According to the specification, such messages must be rejected at the protocol state machine level with a fatal unexpected_message alert and must not be dispatched to any message-specific processing logic.

Below are three distinct cases that demonstrate the issue.

Case 1: Missing Server Certificate Message

Used ciphersuite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
(This key exchange method requires server authentication via certificates.)

Expected (Valid) Ordering

→ ClientHello
→ ServerHello
→ Certificate
→ ServerKeyExchange
→ …

Observed (Invalid) Ordering Accepted by wolfSSL

→ ClientHello
→ ServerHello
→ ServerKeyExchange
→ …

RFC Reference — Section 7.4.2 (Server Certificate)

The server MUST send a Certificate message whenever the agreed-upon key exchange method uses certificates for authentication (this includes all key exchange methods defined in this document except DH_anon). This message will always immediately follow the ServerHello message.

Case 2: Client Certificate Message Omitted After CertificateRequest

Expected (Valid) Ordering

→ ClientHello
→ ServerHello
→ Certificate (Server)
→ ServerKeyExchange
→ CertificateRequest
→ ServerHelloDone
→ Certificate (Client)
→ ClientKeyExchange
→ …

Observed (Invalid) Ordering Accepted by wolfSSL

→ ClientHello
→ ServerHello
→ Certificate (Server)
→ ServerKeyExchange
→ CertificateRequest
→ ServerHelloDone
→ ClientKeyExchange
→ …

RFC Reference — Section 7.4.6 (Client Certificate)

This is the first message the client can send after receiving a ServerHelloDone message.
This message is only sent if the server requests a certificate.
If no suitable certificate is available, the client MUST send a Certificate message containing no certificates.

Case 3: ClientKeyExchange Message Skipped

Expected (Valid) Ordering

ClientHello
→ ServerHello
→ Certificate (Server)
→ ServerKeyExchange
→ CertificateRequest
→ ServerHelloDone
→ Certificate (Client)
→ ClientKeyExchange
→ CertificateVerify
→ …

Observed (Invalid) Ordering Accepted by wolfSSL

ClientHello
→ ServerHello
→ Certificate (Server)
→ ServerKeyExchange
→ CertificateRequest
→ ServerHelloDone
→ Certificate (Client)
→ CertificateVerify
→ …

RFC Reference — Section 7.4.7 (ClientKeyExchange)

This message is always sent by the client.
It MUST immediately follow the client Certificate message, if it is sent.

During my investigation, I did not identify any direct attack vectors or exploitable vulnerabilities resulting from this behavior.
As such, I am reporting this as a protocol conformance bug rather than a security vulnerability, and am intentionally disclosing it via the public issue tracker instead of a private security report.

Thank you for your time and for maintaining wolfSSL.

Best regards,
Jaehun Lee

Reproduction steps

Attached Files

The following files are attached to this issue for reproduction:

• first.py
• second.py
• third.py
• etc.zip (includes ca-cert.pem, server-ecc-cert.pem, server-ecc-key.pem, user-settings.h)

Common Build Configuration

The following build configuration was used for all test cases:

./configure --enable-debug --enable-usersettings
make
make install

First case

1. python3 first.py
2. examples/client/client -p 4444 -v 3 -l TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

Second case

1. examples/server/server -p 4444 -v 3 -l TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 -A ca-cert.pem -c server-ecc-cert.pem -k server-ecc-key.pem
2. python3 second.py

Third case

1. examples/server/server -p 4444 -v 3 -l TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 -A ca-cert.pem -c server-ecc-cert.pem -k server-ecc-key.pem
2. python3 third.py

Relevant log output