Synchronous Tokenizer
A simple synchronous string tokenizer using Regex.
Installation
yarn add tokenize-sync
Usage
The package ships a single tokenize
function that takes an (string) input and a (Object) ruleMap that maps (string) token names to Regexes.
const ruleMap = identifier: /^[a-z-]+$/i number: /^\d+$/ whitespace: /^\s+$/ const input = 'test 12 foobar3' const tokens = tokens === type: 'identifier' value: 'test' start: 0 end: 4 type: 'whitespace' value: ' ' start: 4 end: 5 type: 'number' value: '12' start: 5 end: 7 type: 'whitespace' value: ' ' start: 7 end: 9 type: 'identifier' value: 'foobar' start: 9 end: 15 type: 'number' value: '3' start: 15 end: 16
License
tokenize-sync is licensed under the MIT License.
Documentation is licensed under Creative Common License.
Created with ♥ by @rofrischmann.