0

I come from Java/Scala background, and I'm looking for an adequate replacement for Maven/SBT that I can use for Python projects.

I am looking for a tool that helps me to

  • Run unit and integration tests
  • Compute test coverage
  • Create "executable artifacts" (whatever this means in Python: there are no JAR's?)
  • Since Python isn't really compiled and statically type checked, I will probably sooner or later want to run additional linters (any recommendations here?)
  • Generate Reports
  • Generate Documentation
  • Distribute reusable packages in a way that they can be easily used by pip
  • Run the built application
  • Work with Python 3.6.x
  • Be available for all major linux distributions

So far, I've tried using PyBuilder, because it comes up as first search hit on Google. It worked with the simple examples described in the tutorial, but even basic things such as running the built application turned out surprisingly counter-intuitive and complicated. Moreover, so far I couldn't make test coverage work for projects that are split up into multiple interdependent packages. I would like to hear whether there are any other recommendations before I invest more effort into making it work with PyBuilder.

Maybe I misunderstood something, and I'm wishing for the wrong things? Does it work somehow completely differently in Python? I'm currently struggling even with a tiny project that consists of maybe a dozen of modules and is <2kSLOC: without support by a proper build tool, everything starts to fall apart very quickly.

2

Have you looked into setuptools yet? That's the de-facto standard method for doing most of what you want:

  • Run unit and integration tests

    Handled with minimal effort, and can be integrated into the build process directly.

  • Compute test coverage

    Not sure about this one, but if it isn't handled by setuptools, it can probably be done easily by a plugin or as part of a custom step.

  • Create "executable artifacts" (whatever this means in Python: there are no JAR's?)

    Python doesn't really have a concept of executables. There are a couple of options out tere that will bundle your application into an executable, but they're mostly for Windows, and are generally overly complicated in most cases. Python does have a concept of 'compiled' code, but it's only really used to speed up loading and execution of modules (if the interpreter finds a valid .pyc file alongside the .py file for the module, it will load that instead and avoid having to translate from the source to the internal bytecode representation), and setuptools does support pre-compiling modules.

  • Since Python isn't really compiled and statically type checked, I will probably sooner or later want to run additional linters (any recommendations here?).

    Use a third party tool, and probably integrate it into your source control and not your build. Pylint is the most widely used option I know of for this.

  • Generate Reports

    Not covered by setuptools, but probably not hard to do as a plugin or custom step. Some of the stuff offered by the above mentioned pylint may be of interest here (for example, it can generate UML diagrams from Python code).

  • Generate Documentation

    May or may not be covered by setuptools inherently, but easy to add as part of a custom step. If you just care about the embedded documentation that is the standard for most Python modules, it's handled automatically and you don't need to do anything. If you want it to be accessed through means other than pydoc, you will have to look at using an external tool, though there isn't really any one standard for that.

  • Distribute reusable packages in a way that they can be easily used by pip

    Setuptools together with the actual pip application covers everything you need to publish stuff to PyPI or bundle things as your own module (in fact, if you use setuptools, you can just distribute the source directory directly).

  • Run the built application

    Easy to do with Python without needing any special framework, just make sure your sanely coding your imports.

  • Work with Python 3.6.x

    Setuptools works with all Python versions since at least 2.7.

  • Be available for all major linux distributions

    Because it's so widely used, setuptools is packaged in pretty much every major distribution natively, but it's also trivial to just install using pip if your distro doesn't have it.

  • Thanks for a reply @AustinHemmelgarn! I will take a closer look at this library, but there is one thing that I find a bit strange: if I understand it correctly, setuptools is exactly the set of tools that are used to write setup.py. If this were so unproblematic, then why would someone sit down and write another 20kSLOC-long build tool PyBuilder that does more or less exactly that: generating a clean and simple setup.py? It seems that there must have been quite a large gap in expected vs. perceived usability of setuptools. – Andrey Tyukin Apr 12 '18 at 20:13
  • For example in SBT, I get 80% of the value just by copy-pasting libraryDependencies += foo % version % 123-snippets from MavenCentral into build.sbt, and then issuing short commands like sbt compile/test/run/doc. I for some reason suspect that it's not quite as simple in setuptools. It somehow feels more... "Apache-Ant-like". But I'm not sure, I will have to read more about it first. Thanks so far. – Andrey Tyukin Apr 12 '18 at 20:15
  • @AndreyTyukin No it's actually quite simple to declare dependencies, but I think one problem is that you don't need to - since python sees all installed modules and can use those. That might be a problem if your dependencies don't properly declare their own dependencies. I don't think another tool can solve that though. It might be the other 20% that are targeted by tools like PyBuilder, I think(?) – kutschkem Apr 13 '18 at 6:33
  • @kutschkem "installed" in where? I tried using virtualenv and following the setuptools tutorial. Result so far: nothing tested, no modules installed, but instead it added some "wheels" in ~/home/.cache/pip/wheels. I don't get why this program even dares to escape the scope of the virtualenv and dump anything in my /home. The most basic things don't work, but the tutorial goes on trying to lecture me about version numbers, as if it's the most urgent thing? Anyway, thanks for trying to give some advice, I appreciate that. – Andrey Tyukin Apr 13 '18 at 16:36
  • I would suggest also taking a look at cookiecutter (cookiecutter.readthedocs.io/en/latest) it sets up the framework for your project to use distutils, pytest, etc., and takes a lot of the pain away. – Steve Barnes May 23 at 5:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.