s3-log-fetcher
Fetch logs from S3.
Features:
- incredibly fast
- log lines are pumped out as they come in
- incredibly fast
- uses {node,unix} streams
- incredibly fast
Installation
$ npm install s3-log-fetcher
Usage
const logFetcher = const rs = rs
cli:
Usage: s3-log-fetcher [options] Options: -h, --help Output usage information -v, --version Output version number -f, --from Start date (required) -u, --until End date (required) -k, --key S3 access key, defaults to env.S3_ACCESS_KEY (required) -s, --secret S3 secret key, defaults to env.S3_SECRET_KEY (required) -b, --bucket S3 bucket name (required) Examples: $ s3-log-fetcher -f 2015-10-03 -u 2015-10-03 \ -b testbucket # Fetch logs from S3 for 2015-10-03 $ s3-log-fetcher -f 2015-10-03 -u 2015-10-13 \ -b testbucket # Fetch logs from S3 for 2015-10-03 until 2015-10-13 Docs: https://github.com/TabDigital/node-s3-log-fetcherBugs: https://github.com/TabDigital/node-s3-log-fetcher/issues
S3 directory structure
S3-log-fetcher
expects the S3 bucket to follow a format. All files should
either be prefixed by a date, or live in a directory that is prefixed with the
date, this package don't care. For example:
. [ bucket: sydney-my-app-test ] - 2015-10-11-sydney-ec2-01.log - 2015-10-11-sydney-ec2-04.log - 2015-10-11 - sydney-ec2-02.log - sydney-ec2-03.log - 2015-10-12 - sydney-ec2-01.log - sydney-ec2-02.log - 2015-10-12-sydney-ec2-03.log - 2015-10-12-sydney-ec2-04.log
API
readStream = logFetcher.createReadStream(opts)
Create a readable stream for a start and end date. Takes the following opts:
- from (required): Start date.
- until (required): End date.
- key (required): S3 access key. Defaults to
process.env.S3_ACCESS_KEY
. - secret (required): S3 secret key. Defaults to
process.env.S3_SECRET_KEY
. - bucket (required): S3 bucket name.