ElasticSearch 8 Node.js ODM library
Basic information
- This library maps ES documents and functions to JS classes
Alias
-
ODM models are represented via ES aliases
-
Calling functions like
createIndex()
will create an underlying index and also sets an alias into it -
Most of the communication is accomplished via aliases, real indices are used only when necessary
- This is to simplify some processes like reindexing/cloning/migrating from source (old) index into the new one
- Thanks to this, we can create new index, migrate data from the original one, than switch the aliases and delete the original one.
- And the index are still in human-readable format
- This is to simplify some processes like reindexing/cloning/migrating from source (old) index into the new one
-
Alias consist from two parts:
- tenant (AKA prefix) - defaults to
*
- name - main index name
- tenant (AKA prefix) - defaults to
-
Final alias looks like this:
tenant_name
Work with aliases
-
Alias name is specified when model is created, like this -
createClass('myIndex')
-
Tenant can be specified using
createClass('myIndex', void 0, 'myTenant')
- or later by using
.in('myTenant')
- or later by using
-
Function
in('tenant')
(andimmediateRefresh(bool)
as well) creates new ODM model with updated values, original ODM remains unchanged -
example:
-
createClass('myIndex').in('default')
->default_myIndex
-
createClass('myIndex')
->*_myIndex
-
createClass('myIndex', void 0, 'tenant')
->tenant_myIndex
-
createClass('myIndex', void 0, 'tenant').in('changedTenant')
->changedTenant_myIndex
-
-
When creating a new instance (or performing some operations), the alias cannot contain any wildcard
- it means you must fully specify the tenant
Underlying index
-
Underlying index is in a very similar format, just also contains unique identifier:
tenant_name-id
-
This library is made to use aliases, but technically speaking it should be also working with ES indices directly (or even with mixture of indices and aliases)
- It is necessary to avoid any conflicts between indices / aliases
- Newly created indices will always be created as aliases with underlying indices
Usage
const { createClass, BulkArray, BaseModel, JointModel, esClient, setClient, esErrors, setLoggerConfig, setLoggerUidFunction } = require('odm');
-
createClass
function creates new class- Each class contains several properties
-
schema
,alias
,_tenant
,_name
and_immediateRefresh
-
- Class static functions profit from those properties
- e.g.
static search()
uses them to get alias - You may write your own static function, which will use those properties
- Or rewrite existing functions
- Which may influence another functions
- Or redefine class properties
- Or rewrite existing functions
- e.g.
- Instances have access to those properties / functions
myInstance.constructor.alias
- And they use them during ES operations
- eq. to validate data, ...
- Whenever we need new / independent class, we must either create a new one or clone an existing one
- Each class contains several properties
- Additionally, library provides
BulkArray
- Class inherited from
Array
- Contains methods which uses ES Bulk API
-
save
,delete
andreload
- Plus some additional methods to dynamically mark and remove ODM instance - see in code
-
- Class inherited from
- Exported
BaseModel
should be used mainly forinstanceof
checks- Do not create instances from it / do not change its parameters (unless you really know what you are doing)
-
JointModel
is a special class used to "record" searches from multiple distinct ODM classes and then perform single search with different queries above multiple indices -
setClient
replaces ES client singleton- Should be called once at application startup
- New client is then used - even in already created classes
-
esErrors
- Errors from underlying ES library -
setLoggerConfig
- Replaces default logger configuration -
setLoggerUidFunction
- Pass custom function to be used to generate ID, which is passed to generated logs
Class usage
Create new class
- To create new class, call
createClass(name, ?schema, ?tenant)
:-
name
- essential, it must be specified -
schema
- may be undefined or Joi object, it is there, so we can do additional validations -
tenant
- Serves as part of the index (prefix)- This is required and cannot contain underscores (
_
) - By default, it is
*
- You can rewrite it later by using
.in('newTenant')
- It will create inherited class
const SpecificTenantClass = MyClass.in('newTenant');
- You must specify it before creating new instance
- When searching for records, found records will have correctly set tenant
- You can rewrite it later by using
- This is required and cannot contain underscores (
-
Modify / clone existing class
-
Class can be modified / cloned using:
-
in(tenant)
- Returns class clone with tenant changed to given value
-
immediateRefresh(bool)
- Returns class clone with immediate refresh set to given value
- By default, new classes (and its instances) does perform
refresh: true
with all write operations -> You can use this to disable it
- By default, new classes (and its instances) does perform
- Returns class clone with immediate refresh set to given value
-
clone(?changes)
- Clones class
-
changes
is optional object to set cloned class properties - This method is internally used by
in()
andimmediateRefresh()
- You should not need to use it
-
-
Cloning class means:
- New inherited class is created
- Changes made afterwards to cloned class are NOT transferred to original one
Instances
- Instances are made from prepared class
- Manually:
- You prepare class and then you can call
new MyClass(?data, ?_id, ?_version, ?_highlight, ?_primary_term, ?_seq_no, ?_score, ?_sort)
-
data
is optional object whose properties are placed to instance -
_id
is optional ES _id -
_version
is optional ES _version -
_highlight
is optional ES highlight -
_primary_term
is optional ES _primary_term -
_seq_no
is optional ES _seq_no -
_score
is optional ES _score -
_sort
is optional ES sort
-
- You prepare class and then you can call
- From static functions:
- When you call functions like
findAll()
,search()
, ...- Instance is made from class and correct data is loaded from ES
- When you call functions like
- Manually:
- Instance contains only ES data and methods to save / reload / validate / ...
- All ES properties, search functions, ... are saved in class
- NOTE:
_id
,_version
,_highlight
,_primary_term
,_seq_no
,_score
and_sort
are not enumerable
BulkArray
- For ES Bulk API, you can use
BulkArray
- Class inherited from
Array
- All static search functions in BaseModel class returns
BulkArray
of instances instead ofArray
- Provides bulk functions:
-
async save(?useVersion)
- Saves all items to ES
- Not existing ids are generated and pushed to instances
- Returns ES response
-
useVersion
- Checks if version match- uses sequence numbers internally, if not presented, it will fetch them and checks version automatically
- Saves all items to ES
-
async delete(?useVersion)
- Deletes all items from ES
- Returns ES response
-
useVersion
- Checks if version match - uses sequence numbers internally, if not presented, it will fetch them and checks version automatically
-
async reload()
- Reloads all instances in bulk array
-
Examples:
class Counter extends createClass('counter', Joi.object()) {
async customFunction() {
return 666;
}
static customStatic() {
return new this({ a: 4 }, `myId`);
}
static async search(...args) {
// Rewritting static function -> affects search, find and findAll
const results = await super.search(...args);
if (results.length <= 0) {
throw Error(`Nothing`);
} else {
return results;
}
}
}
...
const MyCounter = Counter.in('default'); //Index is 'default_counter'
const allCounters = await MyCounter.findAll(); //BulkArray with all counters in index
await allCounters.delete(); //Everything deleted from ES
const instanceCounter = new MyCounter({ counter: 15 }, 'myId'); //Prepared new instance
await instanceCounter.save(); //New instance saved
Functions / Methods
Internal getters
-
static get alias
- Returns class full alias, usable for ES queries
- It consists of tenant and name
Class level API
-
static async search(body, ?from, ?size, ?additional)
- Performs ES search
- Returns
BulkArray
of instances - Used by another static functions
- Redefining this function will affect their behavior
- User must specify
body
-
alias
is already defined in the class
-
-
from
andsize
are optional- Returns all results if
from
/size
are not specified, no matter how many there are
- Returns all results if
-
additional
is an optional object with additional data
-
static async *bulkIterator(body, ?source, ?bulkSize, ?additional)
- Async generator for iterating over bulks of documents
-
body
,source
andadditional
arguments are the same as in thesearch
function -
bulkSize
is optional custom size of a bulk
-
static async *itemIterator(body, ?source, ?bulkSize, ?additional)
- Async generator for iterating over documents
- Arguments are the same as in the
bulkIterator
generator
-
static async findAll()
- Finds all entries in ES, matching class
alias
- Uses
this.search()
- Returns
BulkArray
- Returns
- Finds all entries in ES, matching class
-
static async find(ids)
- Performs ES 'search' query
- Always returns
BulkArray
of instances - Uses
this.search()
-
static async get(ids)
- Performs ES 'get'
- If 'ids' is strings, returns single instance
- Else if 'ids' is an array of strings, returns
BulkArray
of instances
-
static async head(ids)
- Performs ES 'mget' without '_source' field
- Throws when any of the IDs is not found
- If 'ids' is strings, returns single object
- Else if 'ids' is an array of strings, returns array of objects
-
static async delete(ids, ?version)
- Performs ES 'delete'
- Uses bulk API
- Class alias must be fully specified
- Returns ES response
- If
version
is specified, only one id is allowed
-
static async exists(ids)
- Performs ES 'exists'
- If 'ids' is strings, returns single boolean
- Else if 'ids' is array of strings, returns array of booleans
- true if exists, else otherwise
-
static async update(ids, body)
- Performs ES 'update'
- Uses bulk API
- Returns ES response
-
static async count(?body)
- Performs ES 'count'
- Returns number
-
static async updateByQuery(body)
- Performs ES 'update_by_query'
- Returns ES response
-
static async deleteByQuery(body)
- Performs ES 'delete_by_query'
- Returns ES response
Indices API
-
static async createIndex(?body, ?setAlias = true)
- Creates index given by current class
-
body
is optional settings -
setAlias
is used when we do not want automatically set an alias to newly create index
-
static async getIndex()
- Returns ES index of ODM alias
- Returns string
-
static async aliasIndex(index)
- Puts alias of current ODM onto selected
index
- Puts alias of current ODM onto selected
-
static async deleteAlias()
- Deletes ODM alias from ES, underlying index remains unchanged
-
static async aliasExists()
- Checks ODM alias existence
- Returns boolean
-
static async indexExists()
- Checks ODM index existence
- Returns boolean
-
static async deleteIndex()
- Deletes alias and index given by current class
-
static async getMapping()
- Gets mapping of index given by current class
-
static async putMapping(mapping)
- Puts mapping to index given by current class
-
static async getSettings(?includeDefaults = false)
- Gets settings of index given by current class
-
static async putSettings(settings)
- Puts settings to index given by current class
-
static async reindex(destinationModel, ?script)
- Reindex from current model class to selected one
- Destination may be chosen by ODM or by alias/index string
-
script
specifies optional painless script to be used
-
static async cloneIndex(?settings)
- Creates clone of ODM index
- Current index has to be manually blocked for write (be made read-only)
-
settings
is optional settings to be used for newly created index - Returns newly created index
-
static async refresh()
- Refreshed current index / indices
-
static async openPIT()
- Opens Point in Time in given index
-
static async closePIT(id)
- Closes given Point in Time
Instance level API
-
async save(?useVersion)
- saves or re-saves instance
- it uses specified
_id
or generates new one if not specified- it uses ES 'index' function
-
useVersion
- checks if version matches,_id
and_version
must be specified- sequence number will be fetched automatically in not presented
-
async reload()
- Reloads instance data from ES
- Uses
get
andnew this.constructor()
internally
-
async delete(?useVersion)
- Deletes an instance from ES
-
_id
must be specified and entry must exist in ES -
useVersion
- checks if version matches,_version
must be specified- sequence number will be fetched automatically in not presented
-
clone(?_preserveAttributes)
- Returns clone of current instance
- Deep copy
- Cloned instance is created from the same class
-
preserveAttributes
(defaults true) can be used to preserve attributes (_id
,_version
, ...)
- Returns clone of current instance
-
async validate()
- Validates instance using Joi
- Throws in case of error / incorrect data
Class copy
-
static clone(?changes)
- Creates class copy
- In fact, it creates new class which inherits from the current one
-
changes
is an optional object with properties to be set
-
static in(newTenant)
- Clones class using
clone()
- Sets given tenant
- Clones class using
-
static immediateRefresh(newRefresh)
- Clones class using
clone()
- Sets given refresh mode
- Clones class using
Other
-
static _parseIndex(index)
- Parses given index (or even alias) and returns its parts
-
static __checkIfFullySpecified(functionName)
- Checks if current class is fully specified, meaning it doesn't contain any wildcard in alias
- Throws otherwise
-
static __getConstructor(searchResult, constructorCache)
- Returns the correct class (constructor) for given search result
-
static _alterSearch(body)
- Alters the search function query (and also query of other functions that use the query)
- By default, just passes the body without any change, but can be rewritten
-
static _unpackData(source)
- Alters the fetched ES documents before the instance is created from them
- By default, just passes the source without any change, but can be rewritten
-
async _packData(cache)
- Alters the instance data before it is saved to ES
- By default, just passes the instance data without any change, but can be rewritten
-
static async _afterSearch(instances, ?cache)
- Special function called for each record founded by search / find / findAll / get
- By default, it is empty, but you can overwrite it in your code
-
static async _getBulkSize()
- Returns optimal size of bulk for given model
- Used in
search
function for pagination