Attach agents to endpoints. Regulate job workflows. Fetch data, file, graph, and json stores. This module documented IAW jsdoc.

From a remote agent node (a @totemorg/agent-deprived node), get/revise the agent registration template, or get/revise a agent service template from your local @totemorg/totem to register agents with the agent cloud.

From a native agent nodes (a @totemorg/agent-ready node), agents directly attach themselves to the agent cloud using the agent attach template.

Requires: module:http, module:https, module:os, module:vm, module:enums

• agent
  • static
    • .attachAgents(opts, agents, cb) ⇒
      • ~getGraph(req, res)
      • ~getJson(req, res)
      • ~getFile(req, res)
      • ~getAgent(spec, agents)
    • .regulateJob(spec, ctx, cb)
    • .fetchData(ref, data, ack)
    • .LexisNexis()
  • inner
    • ~quickLinks
    • ~maxRetry
    • ~certs

Attach agents to the agent cloud, where each agent(req,res) specifies its endpoint as follows:

NAME: agent                     // handle notebook/data requests
"/AREA": agent                  // handle file requests
".TYPE": agent                  // handle conversion requests 
"/": agent 	                    // default handler for file requests 
".": agent	                    // default handler for type conversions
NAME_MACHINE: agent             // handle requests via a MACHINE = py|cv|m|jx|db| ... 
_IGNORE: agent                  // ignored
"#EVENT": agent                 // handle notebook EVENT = charge|run requests
GET|PUT|POST|DELETE: agent      // handle CRUD data requests 

The supplied agents hash is automatically extended with "/json", "/graph", and "/file" reader agents.

When a request arrives to a agent server , attachAgents first

• attempts to resolve the request to the master agent, then (if SQL-equipped)
• attempts to resolve the request to a remote agent, then
• attempts to resolve the request to a notebook agent, then
• defaults to a CRUD agent based on the request type.

The following options opts can be provided when attaching agents:

• cores number of servers (workers) attached to the agents.

• port number that servers listen on.

• CORS flag to support Cross-Origin-Scripting.

• multipart flag to permit multipart files in request bodies.

• never = file cacheing exemption hash { AREA: 1, PATH: 1, ... }.

• mimes mime hash sets the mime header for requested TYPE.

• https options hash when creating a HTTPS server.

• map = area remapping hash {AREA: "/MappedPath", ...}.

• access(req) method to validate client request req.

• proxy(cert,headers) method to adjust cert from http headers.

• init(server,agents) method to initialize the agent server after listener attached.

• busy() method to block DOS attacks. Machines* require additional modules:

• py,cv,m requires @totemorg/agent

• jx requires @totemorg/man

• db requires @totemorg/enums

to support installed python, opencv, matlab, mathjx, mysql db, and neo4j db.

Each agent(req,res) is provided a res response method accepting a:

string
object
array
error
function

and a req request containing:

cookie: "...."		// client cookie string
agent: "..."		// client browser info
ipAddress: "..."	// client ip address
referer: "http://site"		// url during a cross-site request
method: "GET|PUT|POST|DELETE" 		// http request method
now: date			// date stamp when request started
post: "..."			// raw body text
url	: "/query"		// requested url path
reqSocket: socket	// socket to retrieve client cert, post etc
resSocket: socket	// method to create socket to accept response
cert: {...} 		// full client cert

path: "/[area/...]name.type"	// full node path
area: "name"		// file area being requested
table: "name"		// name of dataset/table being requested
type: "type" 		// type descriptor 

query: {...} 		// raw keys from url
where: {...} 		// sql-ized query keys from url
body: {...}			// body keys from request 
flags: {...} 		// flag keys from url
index: {...}		// sql-ized index keys from url

To attach your nodejs-ready machine to the agent cloud:

curl "http://totem/attach" | node [- ?|option=value|list|file.js] 

where option overrides the default port,cores,agents,etc.

Kind: static method of agent
Returns: macs Context of machines required by the agents

• .attachAgents(opts, agents, cb) ⇒
  • ~getGraph(req, res)
  • ~getJson(req, res)
  • ~getFile(req, res)
  • ~getAgent(spec, agents)

Graph reader agent returns neo4j graph at the requested path = /AGENT/GRAPH?QUERY

Kind: inner method of attachAgents

Json reader Agent returns json store at the requested path = /AGENT/TABLE/STORE$EXPRESSION.

Kind: inner method of attachAgents

File reader agent returns file at the requested path = /AREA/FILE

Kind: inner method of attachAgents

Return agent from the agents hash with the given {type,area,method,table} spec.

Kind: inner method of attachAgents

Regulate callback to cb(spec,ctx) on events defined by an event spec:

spec = "STEP / ONSTEPS / OFFSTEPS / START / END" || [spec, ...]

where each subspec is of the form

subspec = NUMBER | NUMBER [s|m|h|d|y] | dd-mm-yr

The context ctx can define:

ctx.queue = "task queue" to share job status
ctx.batch = NUMBER of records returned in callback cb(t,recs)
ctx.client = "client name" to chage this request
ctx.source = "sql table" record source
ctx.fields = "field, ..." source fields
ctx.priority = NUMBER of seconds to monitor signoff-status if required by client

Kind: static method of agent

Fetch data from a specified ref url

PROTOCOL://HOST/FILE ? QUERY 

where PROTOCOL is

• http(s) = http (https) protocol
• curl(s) = curl (curls uses certs/fetch.pfx to authenticate)
• wget(s) = wget (wgets uses certs/fetch.pfx to authenticate)
• mask = http access via rotated proxies
• lexnex = Lexis-Nexis oauth access to documents
• mysql = mysql database
• neo4j = neo4j database
• csv = csv file
• stream = stream file
• json = json file
• jpg = image file
• list = list file
• null = null file
• file = file/folder
• SITEREF = shortcut name

File paths may also contain wild-* cards. The callback name (GET || PUT || POST || DELETE || JSON) determines the request method. Optional QUERY keys:

batch 	= NUMBER of records to fetch 
offset	= NUMBER of records to offset fetch
select	="FIELD, ..." to fetch
keys	= [...] header keys in csv source
comma	= "delim" for csv source
newline = "delim" for csv source

Kind: static method of agent

Example

fetchData( ref, text => { // get request
})

Example

fetchData( ref, function json(ctx) {    // get request with json ctx
})

Example

fetchData( ref, [ ... ], function post(stat) { 	// post request with data hash list
})

Example

fetchData( ref, { ... }, function put(stat) { 	// put request with data hash
})

Example

fetchData( ref, null, function deete(stat) {	// delete request 
})

Start N Lexis-Nexis scrapes residing at the endpoint endpt + random counter when R is non-zero.

Kind: static method of agent

Fetch quick SITEREFs

    https://www.programmableweb.com/search/military

    ACLED
    https://www.programmableweb.com/api/acled-rest-api-v25
    ACCT teliy40602@plexfirm.com / ACLEDsnivel1
    API https://developer.acleddata.com/rehd/cms/views/acled_api/documents/API-User-Guide.pdf
    SITE https://developer.acleddata.com/
    The Armed Conflict Location & Event Data Project (ACLED) is a real-time data and and crisis analysis and mapping project on political violence and protest across the world. ACLED's mission is to produce dis-aggregated, locally informed data and analysis globally in real time. An ACLED REST API enables users to retrieve data about actors, actor type, country, region and get data in JSON, XML, CSV or text. Filter data by date, area, pagination, etc.

    Animetrics FIMS
    https://www.programmableweb.com/api/animetrics-fims-cloud-rest-api
    http://animetrics.com/fims-cloud
    Aimed at the law enforcement, security and military intelligence industries, Animetrics' FaceR Identity Management Solution (FIMS) allows organizations to bring mobile security and video surveillance facial-biometrics applications into the field for use in real time. FIMS Cloud is used to centralize and access a user's cloud based photographic stores using facial recognition. FIMS Cloud organizes, searches and centralizes access to photographic stores using 1:many web service based verification engine. Access to the service is provided via a RESTful API. Public documentation is not available.

    Navlost WXT Weather Tesseract
    https://www.programmableweb.com/api/navlost-wxt-weather-tesseract
    The WXT Weather Service provides atmospheric weather information through a REST architecture, HTTP requests, and JSON formats. It integrates METAR/TAF information, sun, and moon calculations, targeting aviation and energy applications. Advanced features include: -Upper atmosphere information (e.g., research, aviation, rocketry, military) -Automated, push-type notification of arbitrary weather-related events (alert service) -Calculation of arbitrary results derived from weather forecast information via a server-side scripting language. The default response type is application/json, although other formats may be supported. At the present time, there is partial support for comma-separated value (CSV) responses.
    https://wxt.navlost.eu/api/v1/
    https://wxt.navlost.eu/doc/api/

Kind: inner constant of agent

Fetch wget/curl maxRetry

Kind: inner constant of agent

Fetch certs

Kind: inner constant of agent