< ![CDATA[
Update: Don't panic if you see a request to store data appear in your browser (you'll see this if you use Chrome or Opera). It's part of the 'hands-on' look at client side storage in this post (via the jsFiddles included). The FileSystem API example stores sample data and then displays it on screen. :-)
Ever wondered what your options are when it comes to storing data on the client in your web application? Browser support, API features, storage size - like so many DOM features, it can be difficult to know what's available, what browsers support it, and which option is the best fit for your needs. We'll take a look at the basics of each major type of client-side storage in this post, look at some of the pros and cons, and discuss some guidelines to keep in mind as you bring these tools to bear in your applications.
For a long time, the web limped along with one primary form of client-side storage: cookies. Browsers typically limited cookies to a size of 4KB, allowing 20 per domain. Given today's standards, that's some tiny storage space - and it comes at the (potentially high) price of the cookies being sent with each HTTP request.
Thankfully, browsers didn't stop there:
sessionStorage
& localStorage
. The name localStorage
has become nearly synonymous in usage with "Web Storage" - and you will often hear it used instead. Most implementations give you between 2MB and 5MB of key/value storage (with exceptions, of course).Of the storage options we'll cover, Web Storage is the most widely supported option available to you as of today. The Web Storage spec gives us sessionStorage
and localStorage
– which share the same API (both implement the Storage
prototype). sessionStorage
is used to store data for the duration of the current browser session. It will persist between page reloads, but will be cleared when the browser is closed. Data stored in localStorage
is persistent. It will only be cleared when the user specifically clears it via browser tools, or when your code removes items previously stored, etc.
Are there any real advantages of using sessionStorage
over localStorage
? In my opinion, no. The typical use-case for sessionStorage
would be when you need to persist data in a form, for example, because the user might accidentally refresh the page. The problem, though, is that the user might also accidentally close the browser, too. If you were relying on sessionStorage
in case of a refresh, and the user closes the browser instead…Oops. localStorage
handles both issues well. Bottom line: you will probably find localStorage
to be the best option of the two 99.9% of the time (insert anti-well-actually-troll disclaimer here :-)).
Let's look at an example.
Now - the JavaScript:
// cache our selectorsvar $name = $("#fullName");var $loc = $("#location");var store ={// store our input values as members of an object that gets// serialized, since local storage stores *strings* saveToStorage:function(){ localStorage.setItem("example", JSON.stringify(this.getInputValues()));},// a bit naive, but we parse the JSON into an object// and if we get something, we set our input values loadFromStorage:function(){var store = JSON.parse(localStorage.getItem("example"));if(store){ $name.val(store.fullName); $loc.val(store.location);}}, clearFields:function(){ $name.val(""); $loc.val("");}, clearStorage:function(){ localStorage.clear();}, getInputValues:function(){return{ fullName: $name.val(), location: $loc.val()}}};// hook up event handlers $("#setStorage").on("click", $.proxy(store.saveToStorage, store)); $("#loadStorage").on("click", $.proxy(store.loadFromStorage, store)); $("#clearFields").on("click", store.clearFields); $("#clearStorage").on("click", store.clearStorage);
You can see from the above code that the localStorage
API is quite simple. In our saveToStorage
method we're calling localStorage.setItem
and passing the key and value as arguments – saving our serialized object under the key "example" (we have to serialize our object because localStorage
only stores string values.) Then, in our loadFromStorage
method, we call localStorage.getItem
, passing the key of the item we want. Of course, we get a string back, which we need to parse into an object. Further down, you'll notice we're calling localStorage.clear
, which will remove all localStorage
data for this origin. It's possible to remove specific keys, however, by calling localStorage.removeItem(key)
.
localStorage
stores strings. This means that you will need to serialize your objects before storing them if you intend to persist something other than a string.localStorage
size limit (though I believe Chrome allows this for apps downloaded via the Chrome Web Store).storage
event if a different window changes local storage (IE 10 will fire the event for same-window changes, but Chrome and Safari do not, for example). So don't expect the event to be fired for changes within the same window.Overall - localStorage
can be a reliable workhorse - with older browser support often made possible via polyfills. I've used amplify.store for API normalization and fallbacks in many apps, for example. The good news is that current support looks good:
(taken from http://caniuse.com/#feat=namevalue-storage)
The bad news is – as with ANY of these client-side storage options – you should never assume that the data will be truly persistent (after all, the user can nuke it from orbit if they desire), and you should be careful as to the kinds of data you store (it's not hard for any user to view stored data via browser tools).
The conventional wisdom is that localStorage
works well for smaller amounts of data (especially when there's no heavy searching & filtering). When you need something more, that will most likely be IndexedDB (assuming you have browser support). Like localStorage
, IndexedDB allows you to store data persistently on the client, and it follows the same-origin policy. Here are a few defining characteristics of IndexedDB:
read-only
, read/write
, and versionchange
)I put together another jsFiddle to demonstrate some of IndexedDB's features. In our fiddle, we have a storageContainer
object that very lightly wraps IndexedDB. I've intentionally avoided bringing a UI framework into this example (though I did pull in an event emitter and message bus). I think it's important to see IndexedDB's API - once you get the hang of it, it starts to make sense. However, it can be awkward even after you've gotten used to it.
Many API calls return a "request" (an IDBRequest object) - which typically has an onsuccess
and onerror
member which you can assign a handler to (and some requests have additional handler hooks besides these two). It might feel like you're stuck half way between pure-event emitting and promises. I personally prefer event-emitting style APIs, so I brought in a message bus (postal.js) to act as the communications bridge between the hand-rolled view models and our storageContainer
instance. Our storageContainer
instance listens for a couple of different messages and reacts appropriately when one arrives. If any of this is unfamiliar to you, don't worry. The code in the fiddle itself is focused only on the storageContainer
.
You can take the example for a test drive here if your browser supports IndexedDB:
Let's break down some of the snippets a piece at a time:
The init
function of our storageContainer
is as follows:
var storageContainer ={//other members... init:function(){// open our DB and create object stores if it's the// first time this DB has been created on this clientvarself=this;var request = indexedDB.open("BandPrefs", version); request.onerror = jimIsAnIdiot; request.onsuccess =function(e){self.db = e.target.result;if(firstTime){self.loadSeedData();}if(!firstTime &&self.db.objectStoreNames.contains("prefs")){self.loadExistingPrefs();}};// We can only create Object stores in a versionchange transaction.// We know that if this gets called, it's the first time// that the code has run on this client request.onupgradeneeded =function(e){var prefStore; firstTime =true;self.db = e.target.result; prefStore =self.db.createObjectStore("prefs",{ keyPath:"_id"}); prefStore.createIndex("fullName","fullName",{ unique:false});};}};
One of the first steps we take in init
is the call to indexedDB.open
. The first argument is the name of the database, and the second is the version. This is one of the confusing aspects of the API if you're new to IndexedDB in general. Due to the transactional nature of IndexedDB, you can only modify the schema (e.g. - create object stores and/or indexes) in a version upgrade transaction. If the database has never been created before on the client, OR if you provide a version number higher than what exists, then you get the chance to modify the schema as part of the version upgrade by utilizing the onupgradeneeded
hook.
You can see that inside our onupgradeneeded
handler, we're creating an object store named 'prefs' (for storing band/artist preferences). The key for the prefs object store will be the _id
member of the object passed into the store. We're also creating a non-unique index on the prefs object store called "fullName" and the key of the index will be the fullName
property of the pref object. Having this index will make it easy to retrieve a list of band preferences using the name of the person who liked the band - rather than needing to know the key of each one. It's also important to note that onupgradeneeded
will fire before onsuccess
.
The storePref
method on our storageContainer
looks as follows:
var storageContainer ={// stores one band preference record storePref:function(pref){varself=this;var transaction =self.db.transaction(["prefs"],"readwrite"); transaction.oncomplete =function(event){self.emit("transaction.oncomplete");}; transaction.onerror = jimIsAnIdiot;//self-deprecation FTWvar prefs = transaction.objectStore("prefs");var addRequest = prefs.add(pref); addRequest.onsuccess =function(event){self.emit("cursor.onsuccess", pref);};}// other members, etc.};
As a user enters their name and a band/artist they like, the view model for the form publishes a message that our storageContainer
listens for. When that message arrives, the storePref
method is invoked. Again, you'll notice we're dealing with transactions. We start a transaction via self.db.transaction
. The first argument is an array of object store names that this transaction will involve. We only have one - "prefs". The second argument is the transaction mode. We're writing data, so we use readwrite
. Notice that our transaction request has oncomplete
and onerror
hooks, but the add
request also has onsuccess
(and an onerror
that I'm not using in this example). Our new pref object gets added to the store when we call prefs.add(pref)
- that's all it takes.
Our storageContainer
supports two ways to retrieve data - everything, or all the preferences for a given name. The two methods involved are loadExistingPrefs
and filterPrefs
:
var storageContainer ={// looks for existing band preferences and emits an// event containing the list if any exist. loadExistingPrefs:function(){var prefs =[],self=this;self.db.transaction(["prefs"]).objectStore("prefs").openCursor().onsuccess = getIterator(self, prefs,"prefs.exist");},// filters using an index based on the name of the person// and emits an event containing the results filterPrefs:function(name){var prefs =[],self=this;self.db.transaction(["prefs"]).objectStore("prefs").index("fullName").openCursor(IDBKeyRange.only(name)).onsuccess = getIterator(self, prefs,"prefs.filtered");}// other members, etc.};
For the most part, what these two methods do is nearly identical. We start a transaction (with self.db.transaction(["prefs"])
). Then we indicate which object store we're going to work with (via .objectStore("prefs")
). This is where the two diverge. On each one, we're opening a cursor that will allow us to iterate over results. However, the filterPrefs
method (which allows us to find the band/artist preferences for a specific person) passes IDBKeyRange.only(name)
to the cursor. This tells IndexedDB that we only want the items in this index that have the specified key. (The kinds of key constraints possible are very helpful - see this for more detail.)
Both methods are using the same function to generate an onsuccess
handler. (The onsuccess
handler will be called for each record the cursor returns). The function returned from getIterator
will be our onsuccess
handler:
var getIterator =function(container, prefs, topic){returnfunction(event){var cursor =event.target.result;if(cursor){ prefs.push({ fullName: cursor.value.fullName, bandName: cursor.value.bandName }); cursor.continue();}else{ container.emit(topic, prefs);}};};
As long as our cursor is truthy, we continue compiling our results (by adding them to the prefs
array). Once we've iterated over all the rows, the cursor will be falsy. At this point, we emit an event with the results we've compiled.
This example only shows you a few features of IndexedDB - but it shows enough to demonstrate not only the powerful capabilities, but (IMO) the need to wrap and abstract the API away from your application logic. The limited functionality in our storageContainer
still has a decent amount of boilerplate code. Abstracting it into a separate infrastructure-focused component will help keep your application from buckling under the weight of confusing & noisy boilerplate.
Browser support for IndexedDB is improving - but it's still very new. Safari still doesn't have support, nor does <= IE 9.
(taken from http://caniuse.com/#feat=indexeddb)
It's possible to enable IndexedDB on browsers that don't support it if they support Web SQL by using the IndexedDB Polyfill.
Size limits for IndexedDB vary by browser:
It's still early in the game for browser support of writing to the client file system. Currently native support exists only in Chrome 27+, Opera 15+ and Blackberry 10. Burke Holland mentioned idb.filesystem.js, an IndexedDB-based polyfill that emulates FileSystem API support browsers that don't currently have it but do support IndexedDB. We'll focus on the native API in our example, but you could easily adapt it to use the polyfill if you wanted to have a fun evening project.
If you've worked with file system APIs in any language, then the FileSystem API won't seem foreign. In this jsFiddle example, I've adapted most of the earlier IndexedDB example to use FileSystem instead.
You can take the example for a test drive here if you're in Chrome or Opera:
Our FileSystem-based storageContainer
has an init method that sets up initial access to the client file system. Here's the relevant snippet:
varmodule= $.extend({ fs:undefined, init:function(){varself=this;// we request 5MB of storage, though we won't use it all here navigator.webkitPersistentStorage.requestQuota(5*1024*1024,function(grantedBytes){// we request a handle to the file system window.webkitRequestFileSystem( PERSISTENT, grantedBytes,// the fs argument in this callback is our filesystem reffunction(fs){self.fs = fs;self.loadExistingPrefs(function(contents){if(contents.length){ console.log("We already have content we'll use");self.emit("prefs.exist", contents);}else{ console.log("We need to load seed data....");self.loadSeedData(function(){self.loadExistingPrefs();});}});}, errorHandler);}, errorHandler );}// other members, etc.},Monologue.prototype);
First, we ask the user for permission to store persistent data (via navigator.webkitPersistentStorage.requestQuota
). It's worth noting that most examples you'll find on the web today will use window.webkitStorageInfo.requestQuota
for this. However, that has recently been deprecated in favor of the one used above. The first argument is the size we're requesting (5MB). Assuming the user is OK with this, our success handler (the second argument) will be invoked, with the size the user approved passed in as the callback argument.
Once we have the user's permission and the size they approved, we call window.webkitRequestFileSystem
to request access to the file system. The first argument is the type of storage: PERSISTENT
or TEMPORARY
. TEMPORARY
storage can be cleaned by the browser at any time, where PERSISTENT
will only be cleared if the user explicitly does so. The third argument to webkitRequestFileSystem
is the success callback. You can see that it receives a handle to the FileSystem via the fs
argument. Inside our callback we're storing a reference to fs
on our storageContainer
and then we attempt to load any existing data for our example app. If no data exists, we populate the file system with our seed data and then load it up.
Our storageContainer
has a loadExistingPrefs
method that will load any existing band/artist preferences from a data.json
file, if it exists:
varmodule= $.extend({ loadExistingPrefs:function(cb){this.fs.root.getFile("data.json",{ create:true},function(fileEntry){ fileEntry.file(function(file){var reader =newFileReader(); reader.onloadend =function(e){var contents = JSON.parse(this.result ||"[]");if(cb){ cb(contents);}}; reader.readAsText(file);}, errorHandler);}, errorHandler);}// other members, etc.},Monologue.prototype);
By calling this.fs.root.getFile
, we're telling the file system that we want to load the data.json
file (first argument) and create it if it doesnt already exist ({ create: true }
, second argument). The third argument is the success callback that's invoked once the file is opened. Inside our callback we use the fileEntry
instance passed to us to get a handle to the file and read it. We hook up an onloadend
handler (which will fire once the file read has completed), in which we parse the JSON contents into the contents
variable. The code that called loadExistingPrefs
should have passed in a callback for us to invoke once we have the parsed contents - so we pass the contents
variable to that callback. Finally, we kick the read off by invoked reader.readAsText(file)
.
Overall reading isn't terribly complicated, but it could be a bit confusing if you're brand new to the asynchronous nature of JavaScript (and passing continuation callbacks around).
Our storageContainer
instance contains a storePref
method, in which we append a new band/artist preference from the user to the existing file and save it.
varmodule= $.extend({ storePref:function(pref){var currentContents,self=this;self.loadExistingPrefs(function(contents){ contents.push(pref);self.fs.root.getFile("data.json",{ create:false}, getWriter( contents,function(){self.emit("prefs.exist", contents);}), errorHandler );});}// other members, etc.},Monologue.prototype);
The first thing we do is load the existing prefs into memory. (Technically, I could have held onto the already-parsed contents from when we loaded the preferences earlier, but I wanted to show these actions together.) Once we have the existing preferences loaded, they get passed to our callback (as the contents
argument). We push the new band/artist preference into the existing contents
array. Then we call self.fs.root.getFile
to open the data.json
file (first argument) for writing. I've specified that we're not creating the file, since it should already exist. The third argument to getFile
is our success callback. We're calling getWriter
, which returns a properly configured success callback:
var getWriter =function(contents, cb){returnfunction(fileEntry){// Create a FileWriter object for data.json. fileEntry.createWriter(function(fileWriter){ fileWriter.onwriteend =function(e){ console.log("Write completed.");if(cb){ cb();}}; fileWriter.onerror =function(e){ alert("Write failed: "+ e.toString());};// Create a new Blob and write it to data.json.var blob =newBlob([JSON.stringify(contents)],{ type:"text/plain"}); fileWriter.write(blob);}, errorHandler);}};
Our getWriter
function takes the contents we want to save and a callback, and returns a function that handles writing the data. Inside it you can see that we're using the fileEntry handle passed in (getFile
passes this argument into the success callback) to create a writer. Once we have the fileWriter
instance (yet another level deep in nested callbacks, sigh), we hook up a onwriteend
and onerror
handler. Then we stored our serialized contents
array in a BLOG and write it to our file. It's important to note that I'm overwriting the file in this case, not appending. You can append, though. If you simply wanted to add something to the end of the file, then you could include fileWriter.seek(fileWriter.length);
before you call fileWriter.write(blob)
.
In my role as a Developer Advocate for Icenium, I definitely run into the need to store data in a mobile device's file system when building hybrid mobile apps. Icenium, like PhoneGap, uses Apache Cordova, which already supports file system access. The good news is the Apache Cordova File API is based on the W3C File API spec - which means nearly everything you saw in the above examples will be relevant. The main exception is that you won't need to use webkit-prefixed methods (e.g. - use window.requestFileSystem
instead of window.webkitRequestFileSystem
). In fact, it's quite common to see developers normalizing the webkit-prefixed methods in web applications like this:
window.requestFileSystem = window.requestFileSystem || window.webkitRequestFileSystem;
Current browser support:
(taken from http://caniuse.com/#feat=filesystem)
However, you might improve on the above support if you use the idb.filesystem.js polyfill mentioned earlier.
As I mentioned earlier, Web SQL has been deprecated, so we're not going to spend much time on it. However, there are quite a number of applications in the wild using this feature, so you may run into one.
The jsFiddle example for this storage option adapts our earlier localStorage
example to use Web SQL instead:
At the top of the fiddle's code, you see this:
var db = openDatabase("yaysql","1.0","SQL in the web - my worst nightmare",1*1024*1024); db.transaction(function(tx){ tx.executeSql("create table if not exists "+"example(name string, location string)",[],function(){ console.log("table created...maybe :-)");});});
Thankfully, openDatabase
will create one automatically if it doesn't already exist. The first argument is the name of the database, second is the version, third is the description of the database and the last argument is the estimated size.
Next I start a transaction by calling db.transaction
. The callback receives a handle to the transaction via the tx
argument. From here we call executeSql
to create our "example" table if it doesn't already exist. The first argument to executeSql
is the command text. The second argument is an optional array of parameters that can be mapped to placeholders in the command text (we'll see more of this in a moment). The third argument is the callback to be invoked after the command has executed.
Our saveToDB
method looks as follows:
var store ={ saveToDB:function(){var vals =this.getInputValues(); db.transaction(function(tx){ tx.executeSql("INSERT INTO example (name, location) VALUES (?, ?)",[vals.fullName, vals.location],function(){ console.log("Saved");});});}// more members, etc.};
To save the data, we start a new transaction. Inside the transaction callback, we call executeSql
again, this time passing an INSERT command. Note the (?, ?)
placeholder for the value to be inserted. The items we pass in the second argument array will be mapped, in order, to the ?
placeholders. Our third argument, like before, is a callback to be invoked once the command has executed.
Let's look at the loadFromDB
method:
var store ={ loadFromDB:function(cb){ db.transaction(function(tx){ tx.executeSql("SELECT name, location FROM example",[],function(tx, results){if(!results.rows.length){ alert("Nothing is stored in the DB currently.");return;}// we're only using the first row...example cheating FTW cb(results.rows.item(0).name, results.rows.item(0).location);});});}// more members, etc.};
I'm cheating a bit in this example, since I'm only concerned with the first row returned from our query - but this looks a lot like the other commands we've run, with the exception of the fact that we're actually making use of the arguments passed to our third argument callback. The second argument to that callback, results
contains the records returned from our query. If we have rows, we grab the name and location from the first one and pass them into the cb
callback, which was passed by the code that invoked loadFromDB
. You can, of course, use more substantial SQL commands if necessary (specifying a WHERE clause, for example).
Just like the other commands, we start with a transaction, and pass in a DELETE command as our command text, removing all the rows currently in the "example" table.
var store ={ clearDB:function(){ db.transaction(function(tx){ tx.executeSql("DELETE FROM example",[],function(){ console.log("Deleted data from DB");});});}// more members, etc.};
(taken from http://caniuse.com/#feat=sql-storage)
Thus far in my experience with client-side storage, two recurring themes have popped up:
I've linked to (below) a few of the source articles I've referred to a lot in my own development, but I'm also very interested in hearing about your experiences using any of these storage options in your apps. Oscar Godson mentioned the need to sync settings between clients via a server if you're using client-side storage for user settings. I've worked on a couple of apps that sync'd settings, as well as apps that intentionally avoided it. This made me curious - if you've built apps using client-side storage, how have you approached synchronization? If you have more information on size limits (or other features) you feel I should have mentioned or focused on - what would that be?
Finally - I intentionally did not cover using cookies as a client side storage option, though you certainly <airquote>can</airquote>. Aside from (mostly marginalized) stubborn ancient hold-outs, the web has matured well beyond being stuck with 4KB mini-key-value stores. However, if the need arises, there are many libraries out there to make dealing with cookies less painful - the most popular, perhaps, being jquery-cookie.