Castor Load

Traverse a directory to build a MongoDB collection with the found files. Then it enables to keep directory and collection synchronised.

Contributors

• Nicolas Thouvenin
• Yannick Schurter

Installation

With npm do:

$ npm install castor-load

Tests

Use mocha to run the tests.

$ npm install mocha
$ mocha test

API Documentation

Constructor Loader(String directory, [Object options])

Create an new object to synchronise directory with MongoDB collection

Options

• connexionURI - string - URL to connect to MongoDB (see documentation, if not specified, it can look up the environment variable "MONGO_URL" ; default : 'mongodb://localhost:27017/test/'
• ignore - array - List of paths to ignore (either minimatch patterns or Regex) : default : empty
• include - array - List of node types to handle (directory and/or files) : default : ['files']
• collectionName - string - MongoDB collection name : default : automatic
• concurrency - number - Define how many files/documents can be processed in parallel : default : 1
• maxFileSize - string - Maximum size of files, beyond which they will be rejected : default : 128mb
• delay - number - Delay of file processing when the stack is full (milliseconds) : default : 1000
• writeConcern - number/string - Write concern level used for insertions. (see documentation)
• watch - boolean - enable tree watching after the initial synchronization. default: true
• dateConfig - date - (Optional) arbitrary date appended to files metadata. Files whose dateConfig has changed will be resynchronized, regardless of their modification date.
• strictCompare - boolean - always check file content when a change is detected, redardless of its modification date. Should be used when files can get multiple changes in a short period of time. default : false
• modifier - function(baseDoc) - a function to modify the base document of a file upon its creation.

var options = {
    "connexionURI" : "mongodb://localhost:27017/test/",
    "ignore" : [ "**/.*", "*~", "*.sw?", "*.old", "*.bak", "**/node_modules"]
};
var fr = new Loader(__dirname, options);
 

Loader.use([String pattern,] Function middleware)

Add a middleware to be executed on either all files or those matching the given pattern. The middleware is given the document associated with the file, and a callback that can be can be called in two ways :

• if the file matches a single document, then it should be called once with a potential error and the final document.
• if the file must be exploded in mulitple subdocuments, then it should be called multiple times with either a subdocument or an error, and one last time without any argument when all subdocuments have been submitted.

var fr = new Loader(__dirname);
 
/**
 * Just modify the documents associated with .txt files
 */
fr.use('**/*.txt', function (doc, submit) {
  doc.name = doc.basename.toUpperCase();
  submit(null, doc);
});
 
/**
 * Explode the documents of .csv files into multiple subdocuments
 */
fr.use('**/*.csv', function (doc, submit) {
  require('fs').readFile(doc.location, function (err, content) {
    content.split('\n').forEach(function (line) {
      var clonedDoc = {};
      for (var p in doc) { clonedDoc[p] = doc[p]; } // clone the initial document
      clonedDoc.content = line; // add the current line as content
      submit(clonedDoc); // submit the subdocument
    });
    submit(); // call when all subdocuments have been submitted
  });
});

Loader.sync(Function callback)

Start synchronization between the directory and the MongoDB collection. callback will be called after a complete analysis. Its argument is the number of files/directories that were either cancelled or (re)synchronized with the database.

var fr = new Loader(__dirname);
fr.sync(function(processed) {
  console.log('Synchronization done, %d files were either cancelled or checked', processed);
});

Loader.submit(String filename, Function callback)

Submit manually a file to the synchronization system callback will be called after the file analysis. Its arguments are two, an error (if exists) and a object representing the filename in the database.

Loader.syncr.connect(Function callback)

Open a MongoDB connection, or use the existing one. The callback returns a potential error object and a handle to the working collection. Use this if you want to perform some actions on the collection before you start synchronizing.

var fr = new Loader(__dirname);
fr.syncr.connect(function(err, collection) {
  collection.ensureIndex({ 'filename': 1 }, function (err) {
    console.log('Added an index on filename, now starting synchronization');
    fr.sync();
  });
});

Events