docproc
A document processing pipeline mostly used with search-index
docProc = readableStream
DocProc is a pumpify chain of
transform streams that turns Plain Old JSON Objects into a format that
can be indexed by search-index
.
Each processed document must have the following fields:
id
- document idvector
- vector, used for rankingstored
- the document that will be cachedraw
- the unadulterated documentnormalised
- the "cleaned up" document.tokenised
- the tokenised document.
So
id: 'one' text: 'the first doc'
becomes
id: 'one' normalised: id: 'one' text: 'the first doc' raw: id: 'one' text: 'the first doc' stored: id: 'one' text: 'the first doc' tokenised: id: 'one' text: 'the' 'first' 'doc' vector: id: one: 1 '*': 1 text: doc: 1 first: 1 the: 1 '*': 1 '*': one: 1 '*': 1 doc: 1 first: 1 the: 1
...after being passeds through docProc.
You can also compose document processing pipelines by reusing the stages provided, or by creating new ones using the node.js transform stream specification:
docProc
API
.defaultPipeline(options)
A function that returns a writable stream that contains a sensible default document processing pipeline
.customPipeline(pipeline)
A function that takes in an Array of pipeline stages where every stage is a transform stream and returns a writable stream.
CalculateTermFrequency
A transform stream that calculates term frequency.
CreateCompositeVector
A transform stream that calculates the composite vector- used for searching accross all fields.
CreateSortVectors
A transform stream that creates sort vectors.
CreateStoredDocument
A transform stream that defines the parts of each document that are to be cached in the index itself.
FieldedSearch
A transform stream that determines which fields can be searched on individually. In order to make indexes smaller, you can index fields that can be searched on.
IngestDoc
A transform stream that takes an unprocessed document and converts it
into a structure that can be processed by search-index
.
LowCase
A transform stream that converts text to lower case.
NormaliseFields
A transform stream that converts non-string fields into Strings.
RemoveStopWords
A transform stream that removes stopwords
Spy
A transform stream that will do nothing other than print out the state
of the document to console.log
. Use this when developing and
debugging.
Tokeniser
A transform stream that splits fields down into their individual linguistic tokens
Options
See: https://github.com/fergiemcdowall/search-index/blob/master/doc/API.md#options-and-settings