I'm intending to write a web application for the following scenario:
- customer data in a database table on a datawarehouse-server (sql-server)
- customer scores received from a service provider, typically as one csv-file for each individual model. Scores maybe used for customer off- and online targeting
What do I want to do? Put simply: Sampling and A/B-Testing. Afterwards inserting data into a production system.
So, let's say we have scores for 2 models, then I split the customers in my customer database according to some splitting procedur (e.g. 50/50, 75/25, etc). Then I'm "joining" each of the csv-files with the corresponding sample, order by scores and pass the resulting data to the production system (Where someone else is doing simple stuff, but that person's retiring soon, so we need to automate). Of course there maybe further additional tests, like mailing variants etc. (but the socres only help to target the best customers).
Currently, I'm doing all the work using jupyter notebooks (with python) on my local machine. However, the work is repetitive (once the potential sampling and testing schemes are gathered, I'm currently working on this) and the work should be performed remotely.
Furthermore, I want others to perform my work and other to perform reasonable tests. Currently, I'm the only one to perform statistical sound tests. The ultimate solution I'm thinking of, should provide processing information for an administrator (me). The common user, should assign the scoremodels and potential testing variants in a drag-and-drop fashion
Python is relatively slow when writing data to sql server (executemany is not efficient here). Maybe apache spark might give a performance boost over jdbc here.
Which languages do I know:
- I'm somewhat familiar with pyspark
- currently I'm learning scala (coursera courses by odersky, completed first two courses).
- Lots of years ago I learned Java...
So my current, notebook implementation uses python and pandas.
Any framework/tool recommendations on what to use for the webbased project? I'm not requiring python, if there is something better suited in scala or Java I'd also consider this.
I haven't used spark yet, but I think it might be a good project to learn it. Or is this overkill/the wrong choice? On the backend side, I'm thinking of flask since it's lightweight and python based. What else would I need for the data side? What should I use for the frontend, particularly for the drag and drop abilities? (I think I won't have problems to learn the necessary js for this)
I hope, this is the right subforum to post this question. It doesn't fit on stackoverflow since it will be opinion based.