Search results
460 packages found
This project is about to implement a simple javascript parser and visualize the ast
Wix Restaurants credit-cards tokenizer
A powerful and flexible text search library for JavaScript that enables you to build a simple text search engine.
- search
- text-search
- query
- query-parser
- parser
- tokenizer
- interpreter
- AST
- abstract-syntax-tree
- boolean-search
- evaluation
- normalization
- factory
- search-engine
- View more
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 / Claude Instant / Claude 2
- BPE
- encoder
- decoder
- tokenizer
- GPT
- GPT-2
- GPT-3
- GPT-3.5
- GPT-4
- NLP
- Natural Language Processing
- Text Generation
- OpenAI
- Machine Learning
Tokenizer inspired by Laravel's Sanctum
A class Tokenizer to convert text documents into sequences of tokens
Isomorphic utilities for GPT-3 tokenization and prompt building.
TS tokenizer for Mistral-based LLMs
Tokenizes a string that represents a regular expression.
Fastly VCL tokenizer
JavaScript implementation of Japanese morphological analyzer
A React Native supported JavaScript implementation of Japanese morphological analyzer
Simple, but powerful lexical scanner that is a more minimal implementation of X-Scanner
- y-scanner
- yscanner
- x-scanner
- xscanner
- stringscanner
- scanner
- string
- text
- textscanner
- lex
- lexer
- lexical
- parse
- parser
- View more
Simple synchronous string tokenizer using Regex
Simple algorithm to tokenize Chinese texts into words using CC-CEDICT.
A CLI command for converting JSX files into standard JS.
Input field with tagging/token/chip capabilities written in raw JavaScript
An expression tokenizer, parser and evaluator.
A streaming JSON tokenizer
A tokenizer for Google-like search queries