Scarlet Task
A task queue module for node.js. You can set several children-queue for one task queue.
Why named Scarlet? ๛ก(ー̀ωー́ก)
At first, I wrote this module is for searching one song in 萌否收音機. And last I found that song named <the Embodiment of Scarlet Devil>.
For rembembering this and for my favorite Flandre Scarlet, I named this module Scarlet Task
.
Usage
For one situation, once you want to crawl one website. If you use primitive node.js
, it will like you're DDOSing that website.
So you need a task queue to help you. It will process tasks in queue one by one.
What's more, you can set that one queue has several children-queue to work concurrently.
And you can use it at any other situation that suitable.
Installation
$ npm install --save scarlet-task
If you're using it in browser, you may need to install events
as well.
$ npm install --save events
Tutorials
Require the module at first and instantiate an object.
var { Scarlet } = require("scarlet-task");
var taskQueue = new TaskQueue(10);
The parameter for
constructor
means number of children-queue. Pass no parameter for default 1 children-queue.
Define a processor
function for one task. In fact, you can pass an anonymous function.
function processor(taskObject) {
// get task object
var task = taskObject.task;
// Do something...
// blahblah...
taskObject.done(); // You can call `taskQueue.taskDone(taskObject);` either
console.log(taskQueue.numberOfProcessed());
};
Notice: In the
processor
function, you should calltaskObject.done()
ortaskQueue.taskDone(taskObject)
when you think this task is done. And then thetaskQueue
will process next task. The parametertaskObject
is a parameter thattaskQueue
passed to you.
You can push task(s) at anytime.
The task object can be any type - string, number, json, etc.
var task = "it may be a url, or an object that process can do something with this task object.";
taskQueue.push(task, processor);
See more reference at test/touhou.js
.
What's more, if you want to see the queue status for debuging, you can pass
true
totaskDone
andpush
.
eg.
taskObject.done(true); // or `taskQueue.taskDone(taskObject, true);`
taskQueue.push(task, processor, true);
You can reset the number of processed tasks as well:
taskQueue.resetNumberOfProcessed();
And you can set an after-finish function so that Scarlet will call it after a certain number of tasks finished.
taskQueue.afterFinish(20, done, false);
// this will call done() after 20 tasks done without loop (means only once unless you reset number of processed).
taskQueue.clearAfterFinish();
// You can clear after finish processor
See more reference at
examples/hackernews.js
.
Migrate From v1.x to v2.x
scarlet-task
v1.x exports Scarlet
directly:
const Scarlet = require('scarlet-task');
While v2.x exports like this:
const { Scarlet } = require('scarlet-task');
Just replace the requirement.
Contribute
You're welcome to make pull requests!
「雖然我覺得不怎麼可能有人會關注我」