What api/database can I use in javascript client side webapps that reads the data from an online stored file and optionally caches locally

Question

I have a small html/javascript only webapplication that runs client-side only. There is no backend (side from the files hosted on a webserver of course). My app does need to lookup a few fields though, based on an integer index. The data is readonly. There is no need for the user to edit it (I publish a new version once every so often). The data is only a single "table" and each row contains only a few columns. But the data can potentially contain quite a few rows. So a simple json file with an array of objects may get problematic to load into memory. I was looking at websql and https://github.com/orbitaloop/WebSqlSync but websql isn't supported by firefox, so no option for me.

So is there any javascript database/api that I could use load a database file from a webserver and use on the client for the data lookup? It could potentially just cache the file as long as the webserver reports it as unchanged. Basically something like localstorage or an sqlite database or something.

So requirements are:

• no server side backend/software needed other than regular webserver file hosting
• potentially local caching in the browser
• only read-only access needed
• works with javascript in most modern browsers ("recent" versions of edge/FF/chrome)

Answer 1

After some searching around I settled on

1. IndexedDB API for a browser based database
2. The idb library because it adds promises/async supports to IndexedDB
3. The javascript fetch API to download the json files from the webserver.

The nice part of this approach is that with the fetch api, you can check the Last-Modified header from a response, which, if configured properly, is how the webserver tells you when the file was last changed. Using this, I can check if the data in IndexedDB is older than the Last-Modified date and if not, just don't load the json file. That gives a smart one-way sync from json files to an indexedDB where I can query the data.

In IndexedDB I can store javascript "objects" (hashmaps) with all relevant document data, and it can index by key, but you can also create custom indices for other fields of your object, which can be quite convenient. The json files are just arrays of those same objects, so the objects can just be loaded one by one into the database.

This is a sample syncdb function which combines the two and a function which uses an index to fetch data:

async function syncdb(table) {
  db = await openDB(table, 2, {
    upgrade(db) {
      console.log(`upgrading ${table}`);
      try {
        db.deleteObjectStore('systems', {keyPath: 'id'});
      } catch {};
      try {
        db.deleteObjectStore('config');
      }  catch {};
      const store = db.createObjectStore('systems');
      store.createIndex('coords', ['x','y','z'], {multiEntry: false}); 
      const config_store = db.createObjectStore('config');
    },
  });
  console.log('Fetching json data');
  const res = await fetch(table+".json")
  let last_modified = new Date(res.headers.get("Last-Modified"));
  
  const import_date = await db.get('config', 'importDate');
  console.log("Last modified:", last_modified, "import_date:", import_date);
  if(!import_date || new Date(import_date) < last_modified) {
    console.log('importing data');
    await db.clear('systems');
    let json = await res.json();
    const tx = db.transaction('systems', 'readwrite');
    for(let i=0;i<json.length;i++) {
      const system = json[i];
      tx.store.add(system, i);
    }
    await tx.done;
    console.log("Finished adding system data");
    await db.put('config',  new Date(), 'importDate');
  }
  
}

async function getSystemByCoordinates(x, y, z) {
  return db.getFromIndex('systems', 'coords', [x, y, z]);
}