6

I'm looking for Python library which can help with window automation tasks (either on macOS or Linux).

By automation I mean finding the application/game window, gets the content of it and be able to send mouse and keyboard events.

Here is the similar code: jBugman/minesweeper at GitHub which aims to play Minesweeper game by simulating user interaction events.

I'm looking for something similar, but in form of library/framework that I can use for similar use (more universal way). Can work with Quartz or other similar libraries. Compatibility with macOS is preferred, but I'm open to Linux as well (X).

Improve this question
5
  • Do you only want to simulate user input, or do you also want to check the result? E.g if you simulate a click on File/open, then verify that the file selection dialog appears?
    – Mawg says reinstate Monica
    Jun 2, 2017 at 7:47
  • 1
    Mainly to simulate a user input (mouse/keyboard), like automation app testing or playing a game, so checking the results should be also nice to have, but I'm open for suggestions.
    – kenorb
    Jun 2, 2017 at 8:14
  • "finding the application/game window, gets the content of it " seems to imply checking output ... Sounds like you are looking for something like I am in this question
    – Mawg says reinstate Monica
    Jun 2, 2017 at 8:18
  • 1
    @Mawg Yes, we're looking for similar thing, however you're looking for any tool/library in any language for Linux, and I'm looking for library to be able to integrate within Python specifically (either Python library or using it within Python).
    – kenorb
    Jun 2, 2017 at 8:25
  • I would be very happy with Python. Otoh, if we find something non-Python, then we can add a wrapper. I think that you may have to broaden your search, as I didn't find anything satisfactory :-( If you do, please post details
    – Mawg says reinstate Monica
    Jun 2, 2017 at 8:27

3 Answers 3

Reset to default
2

On macOs there is pyatom / ATOMac. It allows access to text properties and meaningful actions without direct access to keyboard. Of course you need to add your app to accessible apps list in macOs settings. My student wrote script automating some steps in iTunes. The only problem is that flexible waits are not working well so he had to use hard-coded time.sleep calls.

There are not so many open source tools to access text properties. On Linux it's pyatspi2 though it's not so easy to use. LDTP might be easier but it doesn't look popular. It has API-compatible Windows and macOs friendly projects (pyatom is this one on macOs). But I'd say the place for popular cross-platform text-based automation tool is vacant in open-source segment.

P.S. I'm maintaining some rating of GUI automation tools for all desktop operating systems including the most popular mouse/keyboard based tools and image based libraries.

Improve this answer
4

There is Sikuli.

SikuliX automates anything you see on the screen of your desktop computer running Windows, Mac or some Linux/Unix. It uses image recognition powered by OpenCV to identify and control GUI components. This is handy in cases when there is no easy access to a GUI's internals or the source code of the application or web page you want to act on.

It supports Python language level 2.7 (supported by Jython).

Improve this answer
2

There are few Python libraries for user input automation, such as:

  • PyAutoGUI (works on Windows, macOS, and Linux)

    A Python module for programmatically controlling the mouse and keyboard.

  • PYAHK AutoHotKey (AHK is normally for Windows, but can be used with Wine)

    AutoHotKey is a powerful task automator with a terrible scripting language built-in. Allow exchange of data between both scripting engines, and the execution of AHK commands from Python.

  • Python-uinput (GPLv3+)

    Pythonic API to Linux uinput kernel module. It allows attaching userspace device drivers into kernel. In practice, Python-uinput makes it dead simple to create virtual joysticks, keyboards and mice for generating arbitrary input events programmatically.

There are also some command-line tools which can be executed from Python in order to interact with GUI applications, such as xdotool. It can be used to control the mouse and keyboard, also supports window manager actions such as moving, activating, and other actions on windows.

For general automation tools for Linux, check also: GUI scipting for Linux.

See also:

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.