nosqplimport
A command-line tool and Node.js library that allows CSV/TSV data to be imported into a variety of NoSQL databases
Installing
Command-line tool
Install with npm:
> npm install -g nosqlimport
You can preview the type of JSON documents nosqlimport will create by piping a file into it:
> cat test.tsv | nosqlimport or > cat test.csv | nosqlimport --delimiter ','
To import data into a NoSQL database, you'll probably want one of the NoSQL plugins:
- CouchDB/Cloudant - nosqlimport-couchdb
- MongoDB - nosqlimport-mongodb
- Elasticsearch - nosqlimport-elasticsearch
> npm install -g nosqlimport-couchdb or install everything > npm install -g nosqlimport-couchdb nosqlimport-mongodb nosqlimport-elasticsearch
Then import data like so:
> cat mydatafile.tsv | nosqlimport --nosql couchdb --url http://localhost:5984 --db mydbnosqlimport Using couchdb nosqlimport writer +0msnosqlimport CouchDB URL: http://localhost:5984 +388msnosqlimport CouchDB Database: mydb +2msnosqlimport
Environment variables can be used to save typing:
> export NOSQL_URL=http://localhost:5984> export NOSQL_DATABASE=mydb> cat mydatafile.tsv | nosqlimport --nosql couchdb
Using within your own application
Use npm install the library
> npm install --save nosqlimport
You'll probably want one of the NoSQL plugins too e.g.
> npm install --save nosqlimport-couchdb
And import data in your code:
var rs = fs; var opts = url: 'http://localhost:5984' nosql: 'couchdb' db: 'mydb' nosqlimport;
Nomenclature
Different NoSQL databases have different names for things. CouchDB stores documents in "databases". MongoDB in "collections" that live in databases. Elasticseach in "types" that live in "indexes".
When using nosqlimport with CouchDB/Cloudant the url
parameter defines the URL of the CouchDB instance and the database
parameter means the database to be written to e.g.:
cat test.txt | nosqlimport --nosql couchdb --url https://myusername:mypassword@myhost.cloudant.com --db mydb
When using nosqlimport with MongoDB, the url
parameter defines the URL of the MongoDB instance including the database and the database
parameter means the MongoDB collection to writer to e.g.:
cat test.txt | nosqlimport --nosql mongodb --url mongodb://localhost:27017/mydatabase --database mycollection
When using nosqlimport with Elasticsearch, the url
parameter defines the URL of the Elastic instance including the index and the database
parameter means the Elasticsearch type to writer to e.g.:
cat test.txt | nosqlimport --nosql elasticsearch --urlhttp://localhost:9200/mydatabase --database mycollection
Transform functions
You can supply your own JavaScript function into the document processing stream for casting types, filtering documents or any other purpose. Your JavaScript function must be synchronous, taking a single doc
parameter which it must return e.g
module { // your code codes here return doc;};
Save your JavaScript to a file and then run nosqlimport
passing in the -t
parameter:
cat test.csv | nosqlimport -t './mytransform.js' -n 'couchdb'
Transform - Casting
module { docprice = ; return doc;};
Transform - Filtering
module { if docinstock === 'true' docinstock = true; return doc; else // nothing is written to the database return {} };
Command-line parameters
- -n --nosql [db type] - the type of NoSQL database to write to (default stdout)
- -t --transform [filename] - the filename containing the JavaScript transform function (default none)
- -d --delimiter [character] - the delimiter character to use (default '\t')
- --db --database [db name] - the name of the database to write to
- -u --url [url] - the url of the database to write to
Environment variables
- NOSQL_URL - same as --url
- NOSQL_DATABASE - same as --database
- DEBUG - set to 'nosqlimport' to output debug messages
Function reference
importStream(rs, ws, opts, callback)
Parameters
- rs - the read stream to read data from
- ws - the write stream to send data to
- opts - an object containing configuration options
- callback - calls back with (err, data) on completion
importFile(filename, ws, opts, callback)
Parameters
- rs - the filename to read data from
- ws - the write stream to send data to
- opts - an object containing configuration options
- callback - calls back with (err, data) on completion
previewFile(filename, callback)
Parameters
- filename - the name of the file to preview
- callback - calls back with (err, data, delimiter)
previewFile(filename, callback)
Parameters
- filename - the name of the file to preview
- callback - calls back with (err, data, delimiter)
previewURL(url, callback)
Parameters
- url - the url to preview
- callback - calls back with (err, data, delimiter)
previewStream(rs, callback)
Parameters
- rs - the read stream
- callback - calls back with (err, data, delimiter)
Origins of nosqlimport
This project is a refactor of my couchimport importer script for CouchDB/Cloudant. This codebase is simpler, more modular and makes more sense.
Contributing
The projected is released under the Apache-2.0 license so forks, issues and pull requests are very welcome.