2

Context:

  • This is a Jersey application
  • Run on a Tomcat server
  • Connects to a MySQL database using Hibernate
  • All on Netbeans

I have an application that has Item objects. Each Item has a unique Number id and as many as 100 metrics associated with it. Each metric has it's own unique Number id and a single field associated with it, a counter. I'm looking for a software recommendation that would be conducive to two things:

  • very frequent, large, batch, atomic incrementations of an Item's metrics given a list of Number's representing the metric ids.
  • very frequent, large, batch reads of all an Item's metrics

Ideally, the read would result in a Map<Number,Number> where an individual Map corresponds to an item, the keys are the metric ids, and the values are the metrics' corresponding counters.

I've come up with a couple implementations myself, but they don't seem well suited for both tasks.

Implementation 1:

A MySQL table exists in the form:

CREATE TABLE `metric` (
 `item_id` int(10) unsigned NOT NULL,
 `metric_id` int(10) unsigned NOT NULL,
 `counter` int(10) unsigned NOT NULL DEFAULT '0',
 PRIMARY KEY (`item_id`,`metric_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

I create a Metric Hibernate entity and add a HashMap<Number,Metric> field to the Item entity. This means everything would be done in MySQL. Reads would be performed by using the Item's id -- fast. Incrementations would be done probably with a SQL statement like so:

UPDATE metric SET counter=counter+1 WHERE item_id=<Item's id>
AND metric_id=<List of metric ids>

*the list of metric ids would never consist of all of an Item's metrics

I would love to use this implemenation. It just doesn't seem like MySQL is well suited for the task of frequent, large batch incrementations of counters. Please correct me if I am wrong.

Implementation 2:

Uses Redis -- really want to avoid implementing an in-memory solution if I can.

This uses Redis Hashes to store a particular Item's metrics. A Hash would be referenced with an Item's id and the counter fields would be referenced using the corresponding metric id.

The data structure seems well suited for the task. A slight performance slip would be that Redis doesn't seem to natively support batch incrementations for Hash's. So every time a counter field is incremented the Hash needs to be searched for again.

The biggest issue with this implementation is that Redis stores all values in Strings. And the seemingly most popular Java client only retrieves Hash's as Map<String,String>. Which means I would have to, at the least, convert each value to an Integer (potentially 100) -- like I said, Ideally I end up with a Map<Number,Number> after reads.

Anyone have any thoughts on what technology would be well suited for this use case?

Update:

It seems like Reddison another Java client for Redis, uses generics to create ReddisonMap's that represent Hash's. I'm not sure how it does the conversion from Hash to ReddisonMap, but it seems like I can get a Map<Number,Number> using it -- but as I said, I'd like to avoid using an in-memory solution if I can.

  • An in-memory solution that converted strings to numbers, added one, converted back to strings is likely to be a lot faster than an implementation that logically has to touch the disk on each record operation. If you really hate the Redis implementation, you can always modify it or roll your own. – Ira Baxter Jun 24 '16 at 14:51
  • @IraBaxter that's why i'm looking for a solution that would enable me to perform batch updates on the disk. wouldn't this prevent me from the overhead of going back and forth? also, the reason i don't want to use Redis is because the amount of ram i have at my disposal is limited. – khakis Jun 24 '16 at 14:59
  • RAM is dirt cheap, and if that solves your problem, it is the fastest and easiest way. So what's the objection? – Ira Baxter Jul 3 '16 at 1:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.