Pony MessagePack¶

Pure Pony implementation of the MessagePack serialization format.

Four public APIs are available:

MessagePackEncoder — Stateless encoding methods. Compact methods (uint, int, str, bin, array, map, ext, timestamp) automatically select the smallest wire format. Format-specific methods (uint_8, uint_32, fixstr, str_8, etc.) are available for explicit control.
MessagePackDecoder — Stateless decoding methods that work with buffered.Reader. Compact methods (uint, int, str, array, map) accept any wire format within a format family. Format-specific methods are available when the caller knows the exact wire format. Assumes all data is available; not suitable for streaming.
MessagePackZeroCopyDecoder — Zero-copy decoding methods that work with ZeroCopyReader. Same API as MessagePackDecoder but returns String val and Array[U8] val views into the reader's buffer instead of copying. Use this when you need to avoid allocation overhead for large string and binary payloads.
MessagePackStreamingDecoder — A streaming-safe decoder that peeks before consuming bytes. Returns NotEnoughData when more bytes are needed, with zero bytes consumed. Uses zero-copy decoding internally.

Encoding and Decoding Scalar Values¶

Compact methods pick the smallest wire format automatically:

use "buffered"
use "msgpack"

// Encode
let w: Writer ref = Writer
MessagePackEncoder.nil(w)
MessagePackEncoder.bool(w, true)
MessagePackEncoder.uint(w, 42)       // positive_fixint (1 byte)
MessagePackEncoder.str(w, "hello")?  // fixstr (6 bytes)

// Transfer encoded bytes to a reader
let r: Reader ref = Reader
for bs in w.done().values() do
  r.append(bs)
end

// Decode — compact methods accept any format in the family
MessagePackDecoder.nil(r)?
let b = MessagePackDecoder.bool(r)?
let n = MessagePackDecoder.uint(r)?
let s = MessagePackDecoder.str(r)?

Format-specific methods (uint_32, fixstr, str_8, etc.) are available when you need explicit control over the wire format.

Encoding and Decoding Arrays¶

Array and map methods only write or read headers containing the element count. The caller must encode or decode each element individually.

// Encode a 3-element array of U32
let w: Writer ref = Writer
MessagePackEncoder.array(w, 3)  // picks fixarray
MessagePackEncoder.uint(w, 1)
MessagePackEncoder.uint(w, 2)
MessagePackEncoder.uint(w, 3)

// Decode
let r: Reader ref = Reader
for bs in w.done().values() do
  r.append(bs)
end

let count = MessagePackDecoder.array(r)?
var i: U32 = 0
while i < count do
  let v = MessagePackDecoder.uint(r)?
  // use v
  i = i + 1
end

Encoding and Decoding Maps¶

Map entries are key-value pairs. The header specifies the number of pairs. Keys and values are encoded alternately.

// Encode a 2-entry map: "a" => 1, "b" => 2
let w: Writer ref = Writer
MessagePackEncoder.map(w, 2)       // picks fixmap
MessagePackEncoder.str(w, "a")?
MessagePackEncoder.uint(w, 1)
MessagePackEncoder.str(w, "b")?
MessagePackEncoder.uint(w, 2)

// Decode
let r: Reader ref = Reader
for bs in w.done().values() do
  r.append(bs)
end

let count = MessagePackDecoder.map(r)?
var i: U32 = 0
while i < count do
  let key = MessagePackDecoder.str(r)?
  let value = MessagePackDecoder.uint(r)?
  // use key and value
  i = i + 1
end

Streaming Decoder¶

MessagePackStreamingDecoder is safe for incremental data. It returns MessagePackArray or MessagePackMap header objects for containers; the caller then reads the elements.

let sd = MessagePackStreamingDecoder
sd.append(data)

match sd.next()
| None => None // decoded nil
| let v: U32 => None // decoded a U32
| let a: MessagePackArray =>
  // read a.size elements
  var i: U32 = 0
  while i < a.size do
    match sd.next()
    | let v: U32 => None // use v
    end
    i = i + 1
  end
| let m: MessagePackMap =>
  // read m.size key-value pairs
  var i: U32 = 0
  while i < m.size do
    match sd.next()
    | let key: String val => None
    end
    match sd.next()
    | let value: U32 => None
    end
    i = i + 1
  end
| NotEnoughData => None // need more bytes
| LimitExceeded => None // value too large
| InvalidData => None // stream is corrupt
end

Zero-Copy Decoding¶

ZeroCopyReader + MessagePackZeroCopyDecoder provide zero-copy decoding as an alternative to buffered.Reader + MessagePackDecoder. When string or binary data falls within a single chunk, the decoder returns a shared view into the reader's buffer — no allocation or copy. When data spans chunk boundaries, it falls back to copying.

MessagePackStreamingDecoder uses zero-copy decoding internally. Callers get the benefit automatically with no code changes.

For low-level decoding where you want zero-copy directly:

use "buffered"
use "msgpack"

let w: Writer ref = Writer
MessagePackEncoder.str(w, "hello")?

let r = ZeroCopyReader
for bs in w.done().values() do
  r.append(bs)
end

let s = MessagePackZeroCopyDecoder.str(r)?

Tradeoff: ZeroCopyReader is specific to this package and is not interoperable with APIs that expect buffered.Reader. Decoded values may hold references to the reader's internal chunks, pinning them in memory until the decoded value is discarded. For most use cases (decode, process, discard) this is not a concern.

Decoding Untrusted Data¶

MessagePack headers declare the size of variable-length values (strings, byte arrays, extensions) and the element count of containers (arrays, maps). A malicious or malformed payload can claim sizes far larger than the actual data — for example, a 5-byte message whose header declares a 2 GB string. Without limits, a decoder may attempt to allocate memory proportional to the declared size, enabling denial-of-service attacks.

MessagePackDecoder provides no protection. It trusts the declared sizes and will attempt to read whatever the header says. Use it only when you control both ends of the connection or have already validated the payload.

MessagePackStreamingDecoder enforces size limits. By default, conservative limits are applied: 1 MB for str/bin/ext data and 131,072 for array/map element counts. When a value exceeds its limit, next() returns LimitExceeded with zero bytes consumed.

For network-facing code that decodes data from untrusted sources, use MessagePackStreamingDecoder:

// Default limits — safe for most applications:
let sd = MessagePackStreamingDecoder

// Tighter limits for a constrained protocol:
let limits = MessagePackDecodeLimits(
  where max_str_len' = 4096,
        max_array_len' = 100)
let sd = MessagePackStreamingDecoder(limits)

// No limits (trusted data only):
let sd = MessagePackStreamingDecoder(
  MessagePackDecodeLimits.unlimited())

See MessagePackDecodeLimits for the full set of configurable limits.

UTF-8 Validation¶

The MessagePack spec defines the str format family as UTF-8 strings. By default, this library does not validate UTF-8 — str values are treated as raw byte sequences for backward compatibility and performance. Opt-in validation is available at every layer:

Validating method variants — Append _utf8 to any str encoding or decoding method. These validate UTF-8 and error on invalid byte sequences:

// Encode — errors if bytes are not valid UTF-8
MessagePackEncoder.str_utf8(w, value)?

// Decode — errors if decoded bytes are not valid UTF-8
let s = MessagePackDecoder.str_utf8(r)?
let s = MessagePackZeroCopyDecoder.str_utf8(r)?

Streaming decoder — Pass validate_utf8' = true to the constructor. Invalid str values return InvalidUtf8 instead of the decoded string:

let sd = MessagePackStreamingDecoder(
  where validate_utf8' = true)
match sd.next()
| let s: String val => // valid UTF-8
| InvalidUtf8 => // invalid UTF-8, can continue decoding
end

Decode then validate — Use MessagePackValidateUTF8 for caller-side validation when you need access to the raw bytes on failure:

let s = MessagePackDecoder.str(r)?
if not MessagePackValidateUTF8(s) then
  // s still available — log, reject, or use as raw bytes
end