I'm trying to move my .NET program to a database engine but I'm having trouble finding what specific platforms/languages I'm supposed to use. Right now everything is done on the client-side application but I am now running into memory and performance issues.

Basically I have multiple large tables. The main table contains products that have a life cycle cost. For example, let's say the supply of "Beans" have a shelf life of 3 or 5 years. Every 3-5 years, the stock is replaced at a value of $500. Based on the conditions set by the client-side application, it can either be set it as 3 or 5 years. This scenario is applied to millions of products with different shelf lives and projected for 100 years. Finally, this result is summed up and displayed as reports.

I'm looking to set up a server-side engine that can help me process my complicated conditional code (I simplified it for my example). I looked into Map-Reduce for Azure's Hadoop engine (HDInsight), and it does exactly what I need it to do (taking a C# function and executing it on the server), but my dataset is much to small to utilize such a platform. At the same time, the long latency period for HDInsight is not good for my program's needs. Any suggestions?

2 Answers 2


For me it sounds like MongoDB would fit here. Some people say Headoop is for terabytes and MongoDB for Gigabytes. MongoDB offers a convenient aggregation framework which is easier to use than MapReduce functions. You could either use he console or any GUI application like MongoChef or implement a C# ASP.NET application or even in any other language you can think of.


I would suggest taking a look at python Pandas since both python and pandas are cross platform you can develop your code client side and then move it over to the server side and either have it run on a schedule, e.g. daily or hourly, or on demand in response to a URL post request.

Pandas works very well with large data sets, see this SO question, including using HDF5. Python/Pandas also has interfaces for most database back-ends so you would have an open ended solution where you can pick your database based on the size of your data and the available hardware.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.