I want to analyse source code written in multiple languages. So the library should tell me at least, which lines are comments and which are not. Optimally it could also tell me more specific stuff, such as variable declarations, language-specific keywords etc.

So it actually is similar to a language specific syntax-highlighter, just that it does not highlight the source code but annotates it and prints something like JSON output.

I found markdown-it for Markdown, but well, I would need it for many languages, such as Shell, JavaScript, Python, Scala, Go …

  • Tools exist for each of those languages, but I don't think there's a generic one, unless you use a generic syntax highlighter for this purpose, which might be straightforward. For example, if the highlighter produces HTML with CSS classes that indicates which parts are keywords, comments, etc., then that's exactly what you want, right? Commented Oct 1, 2017 at 17:58

2 Answers 2


I don't know if it covers all the languages you need, or emits all the metadata you need, but Doxygen might suffice. It can output XML that includes things like function and variable declarations.

  • I don't think Doxygen is related to the question. It generates documentation, but not analyses source code. It isn't even a library, but a separate program.
    – Alejandro
    Commented Mar 16, 2018 at 15:34
  • You're right, it's not a library, I missed that. But it does analyze source code. It might be possible to use it as a library, I don't know.
    – Chris
    Commented Mar 16, 2018 at 15:47
  • What kind of analysis does it do? Can it identify which lines are comments, declarations and the like? Could you expand your answer explaining all this?
    – Alejandro
    Commented Mar 16, 2018 at 16:08
  • Based on the documentation it produces, its analysis includes classes/interfaces and their inheritance relationships, #include files, member variables and methods (including full type signatures), and function/method call graphs. I'm not sure exactly what the xml output contains. I can't find any documentation on it specifically. The best thing is probably to check the schema: github.com/doxygen/doxygen/tree/master/templates/xml
    – Chris
    Commented Mar 19, 2018 at 19:32

Our DMS Software Reengineering Toolkit is a general purpose program analysis and transformation tool parameterized by explicit language definitions. To date there are some 40 languages (not counting dialects) defined for DMS.

For each such language defined by an explicit grammar (yes, we handle C++17 this way), DMS can parse the source and produce an abstract syntax tree. [Yes, you an export it as XML (JSON would be easy, too) if you want, but it is easier when do other analyses to do them with other DMS-provide mechanisms, meaning you don't have to export them. From there labelling comments vs declarations vs function headers vs. code is pretty easy to figure out.

DMS can be configured to pull out a lot more information than just this.

  • 2
    When you work for the company that makes the product, it is normal practice to explicitly include a disclosure at the beginning or the end of the post (even though it is in your profile and you imply it by the use of "we") Commented Oct 16, 2017 at 8:36
  • 1
    It is official Stack Overflow policy that the phrase "Our" ... is adequate disclosure. That was resolved many years ago.
    – Ira Baxter
    Commented Oct 16, 2017 at 14:57
  • 3
    Fair enough, I think the standards of this site should be higher than the network default. So I have made a meta-post: softwarerecs.meta.stackexchange.com/questions/2700 Commented Oct 17, 2017 at 4:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.