Some games have very tedious/clunky menu systems for repetitive ingame tasks (crafting, auction house searches, some types of skill grinding, etc) and I was thinking it might be an interesting project to automate some of these in-game tasks.
Since these games usually use Directx/OpenGL, I can't use windows forms automation software which work based on control IDs.
My idea is that the player would put the game into the correct state and then launch the automation app. The app would take a screenshot and then use OCR to read the data out of the game by sending portions of the screenshot to the OCR engine.
Based on the data, the app would send the appropriate keyboard/mouse input, and take another screenshot to verify the task completed. This would be repeated/looped for all the necessary steps and desired tasks.
So my specific questions are: 1. Are there any existing free GUI automation frameworks that already work in a similar fashion to what I'm describing? 2. What free OCR libraries are available that might work well for this?