CVE-2026-50161: libre WebSocket Integer Overflow Vulnerability

CVE-2026-50161 Overview

CVE-2026-50161 is an integer overflow vulnerability [CWE-190] in the libre real-time communications library, affecting versions prior to 4.8.1. The flaw resides in the websock_decode() function in src/websock/websock.c, which improperly validates masked WebSocket frames using 64-bit extended length encoding. The expression 4 + hdr->len can wrap when hdr->len approaches UINT64_MAX, bypassing the mbuf_get_left() bounds check. The subsequent XOR unmasking loop then writes beyond the heap buffer, enabling attacker-controlled heap corruption or denial of service.

Critical Impact
Remote unauthenticated attackers can trigger heap corruption or denial of service against any application using websock_accept() or websock_accept_proto() to serve WebSocket connections.

Affected Products

libre (baresip/re) versions prior to 4.8.1
Applications using websock_accept() to implement WebSocket servers
Applications using websock_accept_proto() with protocol negotiation

Discovery Timeline

2026-08-18 - CVE-2026-50161 published to NVD
2026-08-19 - Last updated in NVD database

Technical Details for CVE-2026-50161

Vulnerability Analysis

The vulnerability affects the WebSocket frame decoder in the libre library, a generic C library for real-time communications with asynchronous I/O support. When processing a masked WebSocket frame that uses the 64-bit extended length encoding, the decoder reads a 64-bit length value into hdr->len and then attempts to verify that enough bytes remain in the buffer to cover the 4-byte masking key plus the payload.

The check mbuf_get_left(mb) < (4 + hdr->len) performs unsigned 64-bit arithmetic. When hdr->len is set to a value near UINT64_MAX, the addition wraps to a small number, allowing the check to pass with an undersized buffer. The decoder then proceeds to the XOR unmasking loop, which iterates over the attacker-declared length and writes outside the allocated heap buffer.

Root Cause

The root cause is missing integer overflow validation before performing the size comparison. The 64-bit extended length field of the WebSocket protocol permits payload lengths up to UINT64_MAX, but the code did not guard against arithmetic wrap when adding the fixed 4-byte mask key size.

Attack Vector

An unauthenticated remote attacker completes the HTTP WebSocket upgrade handshake against a vulnerable server, then transmits a crafted masked frame declaring a 64-bit length near UINT64_MAX. The bounds check passes due to the wrap, and the unmasking loop corrupts adjacent heap memory. Successful exploitation grants attacker-controlled writes into the heap, which can lead to code execution or crash the process.

// Patch from src/websock/websock.c
// Source: https://github.com/baresip/re/commit/718b92615c7963670d26c1a2b246968b58d782e8

	if (hdr->mask) {

-		if (mbuf_get_left(mb) < (4 + hdr->len))
+		if (hdr->len > SIZE_MAX - 4 ||
+		    mbuf_get_left(mb) < (4 + hdr->len))
 			return ENODATA;

 		hdr->mkey[0] = mbuf_read_u8(mb);

The patch adds an explicit check that hdr->len does not exceed SIZE_MAX - 4 before performing the addition, preventing the arithmetic wrap.

Detection Methods for CVE-2026-50161

Indicators of Compromise

Unexpected process crashes or segmentation faults in services linking libre shortly after a WebSocket upgrade
WebSocket frames with extended 64-bit length fields declaring values near UINT64_MAX (0xFFFFFFFFFFFFFFFF)
Heap corruption traces in core dumps referencing websock_decode in the call stack
Anomalous outbound connections from baresip or other libre-based processes following inbound WebSocket traffic

Detection Strategies

Deploy WebSocket-aware network inspection to flag frames with 64-bit length values above realistic payload thresholds
Enable AddressSanitizer or heap canaries in development and staging builds to catch out-of-bounds writes triggered by fuzzing
Inventory all deployed binaries statically linking libre and cross-reference against version 4.8.1

Monitoring Recommendations

Monitor process telemetry for crashes in WebSocket server components and correlate with recent network sessions
Log the 2-byte and 8-byte extended length headers of incoming WebSocket frames at the reverse-proxy layer
Alert on repeated ENODATA or abnormal disconnect patterns from clients performing HTTP upgrade handshakes

How to Mitigate CVE-2026-50161

Immediate Actions Required

Upgrade libre to version 4.8.1 or later on every host that embeds or dynamically links the library
Rebuild and redistribute downstream applications such as baresip that statically link libre
Restrict inbound access to WebSocket endpoints to trusted networks until patches are deployed

Patch Information

The fix is available in GitHub Release v4.8.1 and was merged via Pull Request #1584. Technical details are documented in GHSA-hvxv-v2gp-v93h and the upstream commit.

Workarounds

Place a WebSocket-validating reverse proxy in front of vulnerable services to reject frames with implausibly large 64-bit length fields
Disable WebSocket support in affected applications until the patched library is deployed
Enforce network-layer access controls limiting who can reach the WebSocket upgrade endpoint

bash

# Verify installed libre version and rebuild against 4.8.1
git clone https://github.com/baresip/re.git
cd re
git checkout v4.8.1
cmake -B build
cmake --build build
sudo cmake --install build