SSLContext ALPN/NPN callbacks keep a stale VALUE after compaction (master-only regression from rb_gc_mark_movable)

[jeremy]

Summary

On master, SSLContext was changed from pinned to movable, but the compaction
callback only fixes up one of the four places its raw VALUE is stored. The ALPN and NPN
callbacks keep the pre-move address, so once the GC compacts a context that has already
completed a handshake, ssl_alpn_select_cb reads a stale VALUE — ALPN negotiation fails,
and it can SEGV.

This is not in any release: 3.3.0, 3.3.1, 4.0.0 and 4.0.2 all pin with rb_gc_mark and
are unaffected (verified by execution — the context never moves). It is a master-only
regression.

Cause

ext/openssl/ossl_ssl.c:56-60 on master marks the context as movable:

static void
ossl_sslctx_mark(void *ptr)
{
    SSL_CTX *ctx = ptr;
    rb_gc_mark_movable((VALUE)SSL_CTX_get_ex_data(ctx, ossl_sslctx_ex_ptr_idx));
}

and ossl_ssl.c:68-77 updates only the ex_data slot:

static void
ossl_sslctx_compact(void *ptr)
{
    SSL_CTX *ctx = ptr;
    VALUE self = (VALUE)SSL_CTX_get_ex_data(ctx, ossl_sslctx_ex_ptr_idx);
    if (self) {
        (void)SSL_CTX_set_ex_data(ctx, ossl_sslctx_ex_ptr_idx,
                                  (void *)rb_gc_location(self));
    }
}

But ossl_sslctx_setup stashes the same raw VALUE in three more places that the compact
callback never touches:

ossl_ssl.c:810   SSL_CTX_set_next_protos_advertised_cb(ctx, ssl_npn_advertise_cb, (void *)self);
ossl_ssl.c:814   SSL_CTX_set_next_proto_select_cb(ctx, ssl_npn_select_cb, (void *) self);
ossl_ssl.c:830   SSL_CTX_set_alpn_select_cb(ctx, ssl_alpn_select_cb, (void *) self);

and ossl_ssl.c:610-616 casts it straight back:

ssl_alpn_select_cb(SSL *ssl, const unsigned char **out, unsigned char *outlen,
                   const unsigned char *in, unsigned int inlen, void *arg)
{
    VALUE sslctx_obj, cb;

    sslctx_obj = (VALUE) arg;
    cb = rb_attr_get(sslctx_obj, id_i_alpn_select_cb);

What makes this reachable rather than theoretical is that ossl_sslctx_setup is lazy and
one-shot — ossl_ssl.c:712 early-returns when the context is frozen, and ossl_ssl.c:834
freezes it at the end:

if(OBJ_FROZEN(self)) return Qnil;
...
rb_obj_freeze(self);

So the callbacks are registered exactly once, at the first handshake, and never re-registered
with a refreshed address. Any long-lived server context that has served at least one
connection and then experiences compaction is exposed.

Reproduction

require "openssl"
require "socket"

key  = OpenSSL::PKey::RSA.new(2048)
cert = OpenSSL::X509::Certificate.new
cert.version = 2
cert.serial  = 1
cert.subject = cert.issuer = OpenSSL::X509::Name.parse("/CN=localhost")
cert.public_key = key.public_key
cert.not_before = Time.now - 60
cert.not_after  = Time.now + 3600
cert.sign(key, OpenSSL::Digest::SHA256.new)

ctx = OpenSSL::SSL::SSLContext.new
ctx.cert = cert
ctx.key  = key
ctx.alpn_select_cb = ->(protocols) { protocols.include?("h2") ? "h2" : protocols.first }

# Parked off-stack: a local is conservatively pinned, which masks the bug.
$holder = [ctx]

srv     = TCPServer.new("127.0.0.1", 0)
port    = srv.addr[1]
$ssl_srv = OpenSSL::SSL::SSLServer.new(srv, $holder[0])

def handshake(port)
  t = Thread.new { s = $ssl_srv.accept; s.close rescue nil }
  cctx = OpenSSL::SSL::SSLContext.new
  cctx.verify_mode = OpenSSL::SSL::VERIFY_NONE
  cctx.alpn_protocols = ["h2", "http/1.1"]
  sock = OpenSSL::SSL::SSLSocket.new(TCPSocket.new("127.0.0.1", port), cctx)
  sock.connect
  proto = sock.alpn_protocol
  sock.close
  t.join(5)
  proto
end

# Required: this is what actually runs ossl_sslctx_setup and freezes the context,
# registering the ALPN callback with the pre-move address.
p handshake(port)                 # => "h2"

GC.verify_compaction_references(expand_heap: true, toward: :empty)

p handshake(port)                 # => raises

Actual, on master (9796ee8):

"h2"
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error:
  tlsv1 alert no application protocol (SSL alert number 120)

with the server side raising NoMethodError: undefined method 'call' for nil — i.e.
rb_attr_get on the stale address returned no callback. Fails 3/3; with the
verify_compaction_references line removed it returns "h2" both times, which is the
control.

Under a full-handshake load it SEGVs outright rather than raising:

[BUG] Segmentation fault at 0x00000001035a4fc0
openssl.bundle(ssl_alpn_select_cb+0x3c)
openssl.bundle(tls_handle_alpn)
openssl.bundle(tls_post_process_client_hello)

Dropping the alpn_select_cb and instead observing the object address directly shows the
move that the ALPN arg misses:

master 9796ee8 : SSLContext 0x124ca67d0 -> 0x124ee1748  MOVED=true   -> handshake fails
openssl 4.0.2  : SSLContext 0x...       -> 0x...(same)  MOVED=false  -> handshake ok
openssl 4.0.0  : MOVED=false -> ok
openssl 3.3.1  : MOVED=false -> ok
openssl 3.3.0  : MOVED=false -> ok

The releases are safe precisely because rb_gc_mark pins, so the three un-updated copies
never go stale.

Suggested fix

Update the other three stashes in ossl_sslctx_compact alongside the ex_data slot:

static void
ossl_sslctx_compact(void *ptr)
{
    SSL_CTX *ctx = ptr;
    VALUE self = (VALUE)SSL_CTX_get_ex_data(ctx, ossl_sslctx_ex_ptr_idx);
    if (self) {
        VALUE updated = rb_gc_location(self);
        (void)SSL_CTX_set_ex_data(ctx, ossl_sslctx_ex_ptr_idx, (void *)updated);
        /* these hold the same VALUE and must move with it */
        if (SSL_CTX_get_alpn_select_cb(ctx) /* or track whether it was set */)
            SSL_CTX_set_alpn_select_cb(ctx, ssl_alpn_select_cb, (void *)updated);
        ...
    }
}

Since the callback arg is always the context itself, a tidier alternative is to stop passing
self as the arg entirely and recover it inside each callback from the SSL_CTX's ex_data —
which the compact callback already maintains — e.g. SSL_get_SSL_CTX(ssl) then
SSL_CTX_get_ex_data(..., ossl_sslctx_ex_ptr_idx). That leaves exactly one stored copy to
keep in sync instead of four.

The two NPN sites (:810, :814) have identical shape. I could not exercise them because
NPN is not negotiated at TLS 1.3, so those are by code reading only — but they need the
same treatment.

Environment

• ruby/openssl master @ 9796ee8, built from source
• ruby 4.0.6 (2026-07-14 revision 03b6d3f889) +PRISM [arm64-darwin23]
• OpenSSL 3.5.5
• Unaffected: openssl gem 3.3.0, 3.3.1, 4.0.0, 4.0.2 (all verified by execution)