PouchDB LRU Cache
An LRU (least recently used) cache designed for storing binary data in PouchDB. Runs in modern browsers and Node.js.
Example
var db = 'my_cache';db; // store 5 MB maximum // store a blob or a bufferdblru;
Motivation
In mobile and offline-ready webapps, you often want to have a small store of binary data that you can guarantee won't grow out of control. For instance, this could be used for caching images so that you don't have to re-load them every time.
This is entirely possible in PouchDB, but implementing it correctly requires some subtle knowledge of how PouchDB deduplicates attachments and how CouchDB compaction works. Hence this plugin.
Why PouchDB? Because it's the most efficient and well-tested way to store binary data with cross-browser support. This plugin works in IE 10+, Windows Phone 8, Firefox, Firefox OS, Chrome, Safari, iOS, Android, and Node.js.
Since Blobs in the browser are kinda tricky to work with, you may also want to look into blob-util.
Usage
To get the plugin, download it from the dist
files above or from Bower:
bower install pouchdb-lru-cache
Then include it after pouchdb.js
in your HTML page:
Or to use it in Node.js or Browserify, just npm install
it:
npm install pouchdb-lru-cache
And then attach it to the PouchDB
object:
var PouchDB = ;PouchDB;
API
All API calls are on a db
object created using new PouchDB('myName')
. For best performance, you should use a separate DB for this LRU plugin and not call any non-LRU methods on the db
.
The API is largely a blatant ripoff homage to node-lru-cache.
Overview
db.initLru([maxSize])
db.lru.put(key, blob, type)
db.lru.get(key)
db.lru.peek(key)
db.lru.del(key)
db.lru.has(key)
db.lru.info()
db.initLru([maxSize])
Sets up the LRU plugin. You must call this before you can do any of the other API calls. It will create a magical db.lru
object, which you will need for the other stuff.
Arguments:
maxSize
: maximum number of bytes to store, total, in the LRU cache. Use0
(or unspecified) for unlimited storage.
Example:
db; // store 5 MB maximumdb; // no limit
This is a synchronous method and does not return a Promise.
Note: see an important caveat below about the true size on disk.
db.lru.put(key, blob, type)
Store a binary Blob in the database. Returns a Promise that will resolve with success if the attachment was successfully stored.
Arguments:
key
: a String to use to identify the blob (e.g. a URL).blob
: an HTML5 Blob, Node Buffer, or a base64-encoded string.type
: the content-type, e.g.'text/plain'
,'image/png'
,'image/jpeg'
, etc. Yes, this is redundant in the case of an HTML5 Blob, but we require it because NodeBuffer
s and base64-encoded strings do not have an inherent type.
Example:
With HTML5 Blobs:
var blob = 'some text data' type: 'text/plain';dblru;
With base64 strings:
var base64 = ;dblru;
db.lru.get(key)
Get a Promise for the binary data associated with the given String key
. The data is always returned in HTML5 Blob format (or a Buffer in Node). If the data is not present (either because it got evicted, or because it never existed), then you'll get an error with status 404.
Arguments:
key
: a String to use to identify the blob (e.g. a URL).
Example:
dblru;
db.lru.peek(key)
Same as get()
, but doesn't update the recent-ness of the blob.
Arguments:
key
: the String that identifies the blob.
db.lru.del(key)
Deletes the blob associated with the key, or does nothing if the blob doesn't exist or is already deleted.
Returns a Promise that resolves when the deletion is complete.
Arguments:
key
: the String that identifies the blob.
db.lru.has(key)
Returns a Promise for true
if the blob is in the store, or false
if it was deleted, evicted, or never existed.
Arguments:
key
: the String that identifies the blob.
Example:
dblru;
db.lru.info()
Returns a Promise for some basic information about what's stored in the LRU cache.
Example:
dblru;
The info
object might look like this:
"items": "foo.png": "length": 68 "digest": "md5-l4wb7knXrV/BpNgQmbE+GA==" "lastUsed": 1413557267330 "bar.png": "length": 68 "digest": "md5-l4wb7knXrV/BpNgQmbE+GA==" "lastUsed": 1413557267330 "baz.png": "length": 67 "digest": "md5-7cVkAmmX4suBnAFSJ4A2Wg==" "lastUsed": 1413557267332 "numUniqueItems": 2 "numEvicted": 0 "totalLength": 135
Return values:
items
: map of keys to the stored attachments, represented as their bytelength
, MD5digest
, andlastUsed
timestamp.numUniqueItems
: number of stored items after deduplication.numEvicted
: number of unique items evicted from the store. If the same attachment is evicted more than once, the number will not increment.totalLength
: total bytelength
of all unique items
Notice that the LRU cache takes into consideration the fact that attachments are deduped based on digest in PouchDB.
Implementation details
PouchDB 3.1.0+ only
This plugin only works with PouchDB 3.1.0+. Before that, attachments were not compacted.
maxSize
is an estimate
The maxSize
specified in initLru()
refers to the byte length of the binary attachments as interpreted by PouchDB (i.e. blob.size
). The underlying storage engine may take up more actual space on disk than the byte length, depending on the browser and adapter. PouchDB's FAQ page has some details.
In IndexedDB, modern browsers (Chrome 43+, Firefox, and IE 10+) will store Blobs directly to disk, so there should be a 1-to-1 correspondence between blob.size
and the space taken up on disk. In Chrome <43 and Android <5.0, it will be signficantly more due to PouchDB's base64 workaround.
In WebSQL, you can't really predict how much space a BLOB will consume; see this thread for details. In my own tests, it seemed to vary up to 50% overhead, proably due to WebSQL coercion of blobs to binary strings.
Furthermore, the maxSize
does not account for the metadata that needs to be stored in order to describe the attachments, so you should give yourself a reasonable buffer when you choose a maxSize
.
Browsers also have storage limits, so be aware of them.
CouchDB
This plugin also works on CouchDB, but YMMV. In particular, CouchDB doesn't dedup attachments based on digest (instead it uses doc ID + attachment name), so the assumptions this plugin makes about the true underlying size may be wrong.
Auto-compaction
You can use a PouchDB with auto_compaction
enabled, but it's not necessary, because this plugin already does the compaction for you.
Perf tricks
If you look at the code, you'll also see that I store data in a _local
doc. This is a special class of document that doesn't retain its full version history, so it avoids the metadata growing unreasonably large over time. Local docs are also faster than regular docs for normal gets/puts.
Replication
This plugin is not really designed for databases you want to replicate. In particular, the use of a _local
document makes it un-replicatable.
Building
npm install
npm run build
The plugin is now located at dist/pouchdb.lru-cache.js
and dist/pouchdb.lru-cache.min.js
and is ready for distribution.
Testing
In Node
This will run the tests in Node using LevelDB:
npm test
You can also check for 100% code coverage using:
npm run coverage
If you have mocha installed globally you can run single test with:
TEST_DB=local mocha --reporter spec --grep search_phrase
The TEST_DB
environment variable specifies the database that PouchDB should use (see package.json
).
In the browser
Run npm run dev
and then point your favorite browser to http://127.0.0.1:8001/test/index.html.
The query param ?grep=mysearch
will search for tests matching mysearch
.
Automated browser tests
You can run e.g.
CLIENT=selenium:firefox npm test
CLIENT=selenium:phantomjs npm test
This will run the tests automatically and the process will exit with a 0 or a 1 when it's done. Firefox uses IndexedDB, and PhantomJS uses WebSQL.
License
Licensed under the Apache License, v2.0.
Copyright, 2015, Squarespace, Inc.