bugfix: sslhandshake lost verify error when handshake completed immediately

[tzssangglass]

sister PR: openresty/lua-resty-core#533

Summary

When ngx_ssl_handshake(c) completes synchronously (NGX_OK), the C handler may still record a verify error in u->error_ret. The old code only checked rc == NGX_ERROR, so the error was silently lost. The primary fix is on the C side: also check u->error_ret. The Lua side then no longer needs its special early-return path, which is removed.

The bug

ngx_ssl_handshake(c) → NGX_OK        // TLS protocol handshake done
    |
    v
SSL_get_verify_result() → 21         // but cert verification failed!
    |
    v
u->error_ret = "unable to verify..." // error recorded in cosocket state
    |
    v
if (rc == NGX_ERROR)                 // rc is NGX_OK, check passes
    return NGX_ERROR;                // ERROR LOST — returns NGX_OK/FFI_OK

The C FFI returned FFI_OK even though u->error_ret was set. In Lua, the FFI_OK path returned true early without entering the error-handling loop, so callers saw sslhandshake() -> true followed by a misleading sock:send() -> "closed".

The fix

C side (lua-nginx-module): the primary fix. One line — also check u->error_ret:

- if (rc == NGX_ERROR) {
+ if (rc == NGX_ERROR || u->error_ret != NULL) {

Now the C FFI returns FFI_ERROR whenever a verify error was recorded, including the synchronous-handshake case.

Lua side (lua-resty-core): simplification that follows from the C fix. Since FFI_ERROR is now surfaced on the first call (not just on async resume), the special FFI_OK early-return path is removed:

-if rc == FFI_OK then
-    if reused_session == false then
-        return true
-    end
-    rc = C...get_sslhandshake_result(...)
-end
-
 while true do
     if rc == FFI_ERROR then
+        rc = C...get_sslhandshake_result(...)
         ...

And get_sslhandshake_result() is added in the FFI_ERROR branch so errmsg/openssl_error_code are populated even when the error comes from the first call (not a yield/resume).

I hereby granted the copyright of the changes in this pull request
to the authors of this lua-nginx-module project.