Search results
146 packages found
Time a JavaScript tokenizer
Split text into tokens.
A CLI tool for converting JSX files into standard JS.
A CLI command for converting JSX files into standard JS.
Lexical analyzer built in Javascript
- lex
- lexer
- lexical
- analysis
- scan
- scanner
- scanning
- token
- tokenize
- tokenizer
- flex
- language support
- built in
- javascript
A fast tokenizer/lexer core that is well tested.
Parsing and tokenizing attributes string
Tokenize JavaScript
Generate the Abstract Syntax Tree (AST) of a Bash command.
Create a string representation of an object or primative
[![Build Status](https://travis-ci.org/kmsheng/tibetan-tokenize.svg?branch=master)](https://travis-ci.org/kmsheng/tibetan-tokenize)
POS Tagger and lemmatizer for javascript
Tokenizes an HTML string, extracting plain text while ignoring HTML tags
Splits a JSON string into an annotated list of tokens
Extracts plain text from Markdown strings
A simple, Twitter-aware tokenizer.
- tokenise
- tokenize
- tokenising
- tokenizing
- tokeniser
- tokenizer
- token
- NLP
- language
- text
- strings
- stanford
- dlatk
Module for fast css selector tokenization from string
Tokenizer for tokenizing sentences, for BERT or other NLP preprocessing.