Raise StackError instead of crashing on deeply nested recursive extensions

[Watson1978]

Summary

A deeply nested payload that uses a recursive extension type (a type registered with recursive: true) can crash the Ruby VM with a SIGSEGV instead of raising MessagePack::StackError.

read_raw_body_begin() ignored the return value of _msgpack_unpacker_stack_push() on the recursive-extension path:

/* ext/msgpack/unpacker.c */
_msgpack_unpacker_stack_push(uk, STACK_TYPE_RECURSIVE, 1, Qnil);
int raised;
obj = protected_proc_call(proc, 1, &uk->self, &raised);

Recursive extensions re-enter msgpack_unpacker_read() through the user proc, consuming a full C stack frame per nesting level. Once the unpacker stack is already at MSGPACK_UNPACKER_STACK_CAPACITY, the failed push is silently discarded and the recursion continues unbounded, exhausting the C stack and crashing the process — whereas every other container type (array/map/raw) correctly propagates PRIMITIVE_STACK_TOO_DEEP and raises StackError.

Impact

This is reachable from MessagePack.unpack / Factory#load on attacker-controlled bytes whenever a recursive extension type is registered. msgpack is a core dependency of Fluentd and similar log/data pipelines, where the unpacker routinely consumes untrusted input, so an unbounded-recursion crash is a denial-of-service risk rather than a clean, rescuable error.

Reproduction

require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem "msgpack"
end

recursive_type = Struct.new(:payload)
factory = MessagePack::Factory.new
factory.register_type(0x01, recursive_type,
  packer:   ->(obj, packer)  { packer.write(obj.payload) },
  unpacker: ->(unpacker)     { recursive_type.new(unpacker.read) },
  recursive: true,
)

obj = 42
1000.times { obj = recursive_type.new(obj) }
payload = factory.dump(obj)

factory.load(payload)   # => SIGSEGV before the fix; MessagePack::StackError after

🤖 Generated with Claude Code

[Watson1978]

Thanks