3

I have a lookup table in MongoDB - about 1 million records - JSON dump is just over 200MB.

I'd like to get rid of the requirement for MongoDB and use something embedded in my webapp (running on Tomcat 8).

Ideally I would be able to deploy a file or set of files with, or alongside, the webapp, and the embedded library would read it from there. The webapp does not need to be able to make updates.

I would be perfectly happy having a separate standalone utility application to generate this file(set), e.g. reading the JSON dump from mongo. I also don't mind if I have to write this utility myself (presumably embedding the same DB library).

I see lots of options for embedded databases in Java, but I haven't played with any of them. Everything I've read about has support for updates, transactions, etc. which I don't need, but if they don't get in the way and don't make life more difficult for me I don't really care I guess.

Finally, ease of use and simplicity/elegance of client code is more important to me than performance (within reason).

Is there any particular candidate best suited to this scenario?

Improve this question
1

2 Answers 2

Reset to default
2

Map

Can you spare about 200 Meg's of memory in your JVM? If so, no need for any database engine or fancy library. Just load your data into memory in a Java Map implementation.

You can't get any more ease-of-use than that.

Performance should be excellent when working all in memory, though you will need to consider which implementation is best suited. Consider the implementations bundled with Java as well as third-party libraries such as Google Guava implementations and utilities. Note especially the bunch of immutable implementations in Google Guava. Those immutable classes reinforce to the programmers that the data is read-only (self-documenting), and protects against any inadvertent attempt to modify.

If the JSON is about 200 megs, my guess is that is about how much memory you will need for an in-memory structure.

For read-only usage, you don't even need to worry about getting a thread-safe implementation.

With 64-bit Java and operating systems, you can allocate a lot of memory to Java nowadays.

Don't over-engineer a solution.

Serialization

You might be able to speed up the starting of your app by serializing the Map object to storage for re-hydration on startup. I do not know which is best, but I'd consider:

  • Simple XML Serialization project.
    I've used this with great success on a couple projects, as it truly is simple and reliable. But I don't know whether or not it is a performant approach to serializing 200 megs of data.
  • Java Serialization (built-in)
    Probably performant. But beware of versioning and other issues.
  • JAXB (Java API for XML Binding)
    See Tutorial by Oracle.
Improve this answer
4
  • Good point - I probably should have specified in the question that I didn't want it in-memory as we're trying to keep the footprint small, but it's still worth considering.
    – CupawnTae
    Jul 23, 2015 at 7:43
  • Buying memory chips is among the cheapest, quickest, and easiest ways to solve a ton of programming and admin problems. If the hardware is under your control, I world strongly suggest considering this route. If not under your control, such as distributing to unknown end users, then that is a different story. Do edit your Question to clarify.
    – Basil Bourque
    Jul 23, 2015 at 7:53
  • AWS instances, under our control, so should be an option. I was just thinking when I started working with Java, my PC probably had 4MB RAM and a 200MB file probably wouldn't have fit on my hard disk, not to mind in-heap. Sometimes have to stop and remind myself how far things have moved on. Anyway, we're investigating and we're likely to go with this solution, in which case the question is ok as it stands and I'll accept this answer (still interested in other approaches if they exist though). If in-memory turns out not to be feasible, I'll edit the question to include the constraint. Thanks.
    – CupawnTae
    Jul 23, 2015 at 8:52
  • So. Java HashMap took >500MB to store the 200MB of data (parsed into objects). Guava ImmutableMap didn't make a noticeable difference. Adding String.intern() to the setters in my data object brought that down to about 200MB. And then we realized we could strip a few fields from the data and got it down to 180MB, which will work for our case. Thanks for the suggestion.
    – CupawnTae
    Jul 23, 2015 at 15:20
1

MapDB could be also an option as it is easy to embed and to use and combines the features of a map with serialization (and with good performance).

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.