First off, you might not need a game engine. It looks like what you need is more on pair with a graphics library.
Requirement by requirement:
image blending modes - add and multiply modes
There is browser support for various
fill area bordered by color - example: replace blue by red in are defined by black edges shape
What you describe here is a flood fill. There is no browser support for flood fill in the browsers. Yet, there are some good implementations such as floodfill.js. Which you probably could combine with whatever solution you choose for the other requirements.
draw by mouse/finger
The simplest implementation is to track the mouse (with
onmousemove event) and use
fill area by gradient
We are back to browser support. You can use either
createRadialGradient depending on what you want.
Therefore what you ask for is not in the realm of fiction. I had a look at libraries you may use. It appears to be that paper.js and processing.js are you best options to creating drawing tools. Another option worth looking into is fabric.js.
There is one library that the developers advertise to create games that you may find useful. It is EaselJS.
Note: None of these options have a real flood fill. The
fill function they provide is to specify the filling for shapes you draw afterwards, not to fill in the area defined by what is already drawn. In fact, none of the libraries listed in List of JS Game Engines (community effort) seem to have a flood fill. If we are accepting solutions without flood fill, have a look at Cocos2d.
Processing.js is, of course, a port of Processing which is designed as a tool to learn (and teach) programming. Given that your
programming skills are very poor, close to none I suggest to try this one. Addendum: Use the