Compressed, static-typed binary buffers in HTML5 / Node.js

• 🚀 Designed for real-time HTML5 games (via geckos.io or socket.io)
• 🗜️ Lossless and lossy compression, up to ~50% smaller than FlatBuffers or Protocol Buffers
• ✨ Out-of-the-box boolean packing, 16-bit floats, 8-bit scalars, and more
• 🚦 Compile-time safety & runtime validation

tinybuf is safe for use with property mangling & code minification like terser

FlatBuffers and Protocol Buffers are heavy, cross-platform libraries that have limited encodings, and depend on clumsy external tooling. tinybuf is optimized for speed, performance, and ✨ developer productivity. See comparison table for more.

Easily encode to and from binary formats

import { encoder, Type } from 'tinybuf';

// Define format:
const PlayerMessage = encoder({
  id: Type.UInt,
  health: Type.UInt8,
  position: {
    x: Type.Float32,
    y: Type.Float32
  }
});

// Encode:
const bytes = PlayerMessage.encode(myPlayer);

Decoding many formats:

import { decoder } from 'tinybuf';

// Create a decoder:
const myDecoder = decoder()
  .on(PlayerMessage, data => handlePlayerMessage(data))
  .on(OtherMessage, data => handleOtherMessage(data));

// Trigger handler (or throw UnhandledBinaryDecodeError):
myDecoder.processBuffer(bytes);

tinybuf provides the ability to quickly encode and decode strongly-typed message formats.

The core concepts are:

1. encoder: Flexible, static-typed binary encoding formats
2. Types: 25+ built-in encoding formats
3. decoder: A parser for processing multiple binary buffer formats

For more information on additional pre/post-processing rules, check out Validation and Transforms.

# npm
npm install tinybuf -D

# yarn
yarn add tinybuf --dev

Create an encoding format like so:

import { encoder, Type } from 'tinybuf';

// Define your format:
const GameWorldData = encoder({
  time: Type.UInt,
  players: [{
    id: Type.UInt,
    isJumping: Type.Boolean,
    position: {
      x: Type.Float,
      y: Type.Float
    }
  }]
});

Then call encode() to turn it into binary (as ArrayBuffer).

// Encode:
const bytes = GameWorldData.encode({
  time: 123,
  players: [
    {
       id: 44,
       isJumping: true,  
       position: {
         x: 110.57345,
         y: -93.5366
       }
    }
  ]
});

bytes.byteLength
// 14

And you can also decode it directly from the encoding type.

// Decode:
const data = GameWorldData.decode(bytes);

The encoder will automatically infer the types for encode() and decode() from the schema provided (see the Types section below).

For example, the type T for GameWorldData.decode(...): T would be inferred as:

{
  timeRemaining: number,
  players: {
    id: string,
    health: number,
    isJumping: boolean,
    position?: {
      x: number,
      y: number
    }
  }[]
}

You can also use the Decoded<T> helper type to get inferred types in any custom method/handler:

import { Decoded } from 'tinybuf';

function updateGameWorld(data: Decoded<typeof GameWorldData>) {
  // e.g. Access `data.players[0].position?.x`
}

Serialize data as a number of lossless (and lossy!) data types

^*Int is a variable-length integer ("varint") which encodes <±64 = 1 byte, <±8,192 = 2 bytes, <±268,435,456 = 4 bytes, otherwise = 8 bytes.

^#UInt is a variable-length unsigned integer ("varuint") which encodes <128 = 1 byte, <16,384 = 2 bytes, <536,870,912 = 4 bytes, otherwise = 8 bytes.

^†Length of payload bytes as a UInt. Typically 1 byte, but could be 2-8 bytes for very large payloads.

^¶2-bit overhead: 6 booleans per byte (i.e. 9 booleans would require 2 bytes).

By default, each encoder encodes a 2-byte identifier based on the shape of the data.

You can explicitly set Id in the encoder(Id, definition) to any 2-byte string or unsigned integer (or disable entirely by passing null).

Handle multiple binary formats at once using a decoder:

import { decoder } from 'tinybuf';

const myDecoder = decoder()
  .on(MyFormatA, data => onMessageA(data))
  .on(MyFormatB, data => onMessageB(data));

// Trigger handler (or throw UnhandledBinaryDecodeError)
myDecoder.processBuffer(binary);

Note: Cannot be used with formats where Id was disabled.

You can manually read message identifers from incoming buffers with the static function BinaryCoder.peekIntId(...) (or BinaryCoder.peekStrId(...)):

import { BinaryCoder } from 'tinybuf';

if (BinaryCoder.peekStrId(incomingBinary) === MyMessageFormat.Id) {
  // Do something special.
}

By default Id is based on a hash code of the encoding format. So the following two messages would have identical Ids:

const Person = encoder({
  firstName: Type.String,
  lastName: Type.String
});

const FavoriteColor = encoder({
  fullName: Type.String,
  color: Type.String
});

NameCoder.Id === ColorCoder.Id
  // true

If two identical formats with different handlers is a requirement, you can explicitly set unique identifiers.

const Person = encoder(1, {
  firstName: Type.String,
  lastName: Type.String
});

const FavoriteColor = encoder(2, {
  fullName: Type.String,
  color: Type.String
});

Identifiers can either be a 2-byte string (e.g. 'AB'), an unsigned integer (0 -> 65,535).

The great thing about binary encoders is that data is implicitly type-validated, however, you can also add custom validation rules using setValidation():

const UserMessage = encoder({
  uuid: Type.String,
  name: Optional(Type.String),
  // ...
})
.setValidation({
  uuid: (x) => {
    if (!isValidUUIDv4(x)) {
      throw new Error('Invalid UUIDv4: ' + x);
    }
  }
});

You can also apply additional encode/decode transforms.

Here is an example where we're stripping out all whitespace:

const PositionMessage = encoder({ name: Type.String })
  .setTransforms({ name: a => a.replace(/\s+/g, '') });

let binary = PositionMessage.encode({ name: 'Hello  There' })
let data = PositionMessage.decode(binary);

data.name
  // "HelloThere"

Unlike validation, transforms are applied asymmetrically.

The transform function is only applied on encode(), but you can provide two transform functions.

Here is an example which cuts the number of bytes required from 10 to 5:

const PercentMessage = encoder(null, { value: Type.String })
  .setTransforms({
    value: [
      (before) => before.replace(/\$|USD/g, '').trim(),
      (after) => '$' + after + ' USD'
    ]
  });

let binary = PercentMessage.encode({ value: ' $45.53 USD' })
let data = PercentMessage.decode(binary);

binary.byteLength
  // 5

data.value
  // "$45.53 USD"

Choosing for real-time HTML5 / Node.js applications and games.

Here are some use cases stacked uup.

^†Based on the Reference data formats and schemas

^*When using protobufjs

{
  "players": [
    {
      "id": 123,
      "position": {
        "x": 1.0,
        "y": 2.0,
        "z": 3.0
      },
      "velocity": {
        "x": 1.0,
        "y": 2.0,
        "z": 3.0
      },
      "health": 1.00
    },
    {
      "id": 456,
      "position": {
        "x": 1.0,
        "y": 2.0,
        "z": 3.0
      },
      "velocity": {
        "x": 1.0,
        "y": 2.0,
        "y": 3.0
      },
      "health": 0.50
    }
  ]
}

const ExampleMessage = encoder({
  players: [
    {
      id: Type.UInt,
      position: {
        x: Type.Float16,
        y: Type.Float16,
        z: Type.Float16
      },
      velocity: {
        x: Type.Float16,
        y: Type.Float16,
        y: Type.Float16
      },
      health: Type.UScalar
    },
  ],
});

// ExampleMessage.fbs

namespace ExampleNamespace;

table Vec3 {
  x: float;
  y: float;
  z: float;
}

table Player {
  id: uint;
  position: Vec3;
  velocity: Vec3;
  health: float;
}

table ExampleMessage {
  players: [Player];
}

root_type ExampleMessage;

syntax = "proto3";

package example;

message Vec3 {
  float x = 1;
  float y = 2;
  float z = 3;
}

message Player {
  uint32 id = 1;
  Vec3 position = 2;
  Vec3 velocity = 3;
  float health = 4;
}

message ExampleMessage {
  repeated Player players = 1;
}

See docs/ENCODING.md for an overview on how most formats are encoded (including the dynamically sized integer types).

Developed from a hard-fork of Guilherme Souza's js-binary.