job scheduling and automatization with custom parameters

Question

tl;dr What I am looking for, basically, is a web interface for cron, where I am able to configure a set of additional parameters.

lets start with some facts:

pre-req's

OS: Ubuntu 12.04 / 14.04

Involved Software / Tools: cron, php

Current process:

• several php-scripts are executed on a regular base (daily, monthly) as well as manually
• regularly executed scripts are just activated by cron daemon (f.e. 10 * * * * php-cgi -q script.php param1=value1)
• manually executed scripts are activated via command line or remotely via the browser (f.e.: http://foobar.com/script.php?param1=value1 or php-cgi -q script.php param1=value1)

Special considerations - i have five different scripts (like import.php, export.php and so on) - each of my scripts has a particular set of default and mandatory parameters as well as optional parameters

Optimization potential

Currently it is quite hard to manage & control all those running scripts. This configuration lacks an useful web interface. If I want to schedule a script with a different set of parameters, I need to add it to the a cron.d-file with all the desired parameters.

All those scripts do have an error handling, of course. But I don't have one particular system where I am able to analyze the output of those scripts.

What I'd like to have

A web interface where an admin configures new scripts (script1.php, script2.php) with a set of default parameters.

The user may use one of those default settings to schedule scripts or run them manually. In both cases he must provide the mandatory set of parameter and he should be able to add additional parameters. This is quite important.

The interface should provide an overview for all scripts with the information of the last execution time and whether the execution was successful or not.

Nice to have

I have two servers running my script. It would be helpful to have on web interface where I can manage the scripts on different servers. But this is not important for me.

Besides, I'd appreciate an overview like charts to analyze execution times and stuff.

It would also be really great to create "chains" of dependencies: Run job A if job B ended successfully.

Estimated costs Frankly: I don't really care. If it is for free, it's good. But I also would like to show my appreciation with a fee or a price. But if it is to expensive, I am thinking about creating this for my own - simple make or buy-decision. ;)

Tested so far Most tools I tested so far are not able to handle mandatory and optional parameters. Users are not able to change parameters via the web interface - this is one of the most important features. There are a lot of pretty big software, like Slurm or Hudson, who are to complex and intended for cluster computing. Same with so called "workflow management frameworks" - Using them would be the same effort like developing an own one.

JobScheduler - crashes now and then MiniCron - looks neat, but also crashes, no way to configure custom parameters on webinterface AirFlow - looks promising first sight, to complex and no way to change parameters on the webinterface

Long story short What I am looking for, basically, is a web interface for cron, where I am able to configure a set of additional parameters.

Thanks in advance for your support

Answer 1

Check out https://dataportfol.io, I think it satisfies all your needs.

As for your required features, DataPortfolio has:

• A job-execution framework and scheduler - run jobs manually, on repeat or use cron-expressions to run jobs at scheduled intervals
• Support for flexible execution parameters - can specify multiple different parameters for different invocations
• Support for DAG pipelines - create pipelines with branching and condensing stages where a single failure will stop the flow / prevent other stages from executing ('pipeline' and 'stage' are product terms; I think this is what you mean by 'chains')
• Real-time dashboards that update stage progress as well as the ability to access historical data (pipelines and stages are rendered visually)
• Configurable execution environments - in other words, you tell it where to run the scripts. You can run a script only on one host or both hosts by configuring ProcessorGroups.

Features that are in development or road-mapped:

• log streaming (in development)
• Metrics (short-term roadmapped)

The core is a web-based interface, cloud-based infrastructure and an agent application that you install on each participating server (works on *nix/Win/Mac). Full disclosure: This is my product. It is in early-beta - reach out to me if you'd like to have it (free) in exchange for some feedback (use the 'Schedule My Demo' / contact form on the landing page and I'll get back to you shortly). Hope to hear from you!