Search results

@orama/tokenizers

Additional tokenizers for Orama

• full-text search
• search
• fuzzy search
• typo-tolerant search
• full-text
• vector search
• stemming
• tokenizers
• mandarin
• chinese

micheleriva

published 2.0.18 • 16 days agopublished 2.0.18 16 days ago

fib-tokenizers

cpp tokenizer module for fibjs.

• tokenizers
• fib-tokenizers

xicilion

published 1.2.0 • 4 months agopublished 1.2.0 4 months ago

tokengeex

This repository holds the code for the TokenGeeX Rust crate and Python package. TokenGeeX is a tokenizer for [CodeGeeX](https://github.com/THUDM/Codegeex2) aimed at code and Chinese. It is based on [UnigramLM (Taku Kudo 2018)](https://arxiv.org/abs/1804.1

• NLP
• tokenizers

luojiesi

published 0.6.2 • 2 months agopublished 0.6.2 2 months ago