❤
npm
Sign UpSign In

hx-tokenizer

1.0.0 • Public • Published

tokenizer Build Status npm version XO code style

Split text into tokens.

Install

$ npm install --save hx-tokenizer

Usage

const tokenizer = require("hx-tokenizer");
const tokens = tokenizer.tokenize("Lorem ipsum, something.");
// tokens == ["Lorem", "ipsum", ",", "something", "."]
const text = tokenizer.join(tokens);
// text == "Lorem ipsum, something."

API

tokenizer.tokenize(text)

tokenizer.join(tokens)

License

MIT © Hexacta

Readme

Keywords

Package Sidebar

Install

npm i hx-tokenizer

Repository

github.com/hexacta/tokenizer

Homepage

github.com/hexacta/tokenizer#readme

Weekly Downloads

1

Version

1.0.0

License

MIT

Last publish

Collaborators

  • pomber
Try on RunKit
Report malware