Oplog Transform
Free your oplog
Why?
Often people want a duplicate of their data in another data store. Doing imports on an interval works but need to be scheduled. MongoDB replica sets have a tailable oplog that provides push like updates about changes happening to the records in the database. By tapping into this we can publish changes to other datastores too, keeping them up to date in real time.
oplog-transform
is simply a module that provides an interface for insert
, update
, and remove
hooks for you to fill in with how to modify the target datastore. You can then pass in oplog records and oplog-transform
will use the hooks accordingly.
Usage
Here is an example of initializing oplog-transform
using an in-memory JSON object as our datastore.
var store = {} Transform = transform = { store_id = doc } { } { delete store_id }
You can write to the transform using the transform
function
transform
Or you can use the streaming interface to pipe oplog docs
tailedOplog //or transform
Hooks
insert
{ /* insert doc */ }
The insert hook must be a function that accepts 3 parameters, the doc _id
, the doc
, and a callback to fire once you've inserted the document
update
{ var retrievedDoc = /* fetch doc from your secondary store */ }
The update hook must be a function that accepts 2 parameters, the doc _id
, a done
callback. Once you've fetched the document from your target datastore call the done
callback with the fetched document and a callback to receive an error, the updated document, and a final callback to fire once you've updated the record in your target datastore
remove
{ /* delete doc */ }
The remove hook must be a function that accepts 2 parameters, the doc _id
and a callback to fire once you've removed the doc.
Made with ⚡️ by @taterbase