4

I want to analyse source code written in multiple languages. So the library should tell me at least, which lines are comments and which are not. Optimally it could also tell me more specific stuff, such as variable declarations, language-specific keywords etc.

So it actually is similar to a language specific syntax-highlighter, just that it does not highlight the source code but annotates it and prints something like JSON output.

I found markdown-it for Markdown, but well, I would need it for many languages, such as Shell, JavaScript, Python, Scala, Go …

Improve this question
  • Tools exist for each of those languages, but I don't think there's a generic one, unless you use a generic syntax highlighter for this purpose, which might be straightforward. For example, if the highlighter produces HTML with CSS classes that indicates which parts are keywords, comments, etc., then that's exactly what you want, right? – Kodiologist Oct 1 '17 at 17:58
1

I don't know if it covers all the languages you need, or emits all the metadata you need, but Doxygen might suffice. It can output XML that includes things like function and variable declarations.

Improve this answer
  • I don't think Doxygen is related to the question. It generates documentation, but not analyses source code. It isn't even a library, but a separate program. – Alejandro Mar 16 '18 at 15:34
  • You're right, it's not a library, I missed that. But it does analyze source code. It might be possible to use it as a library, I don't know. – Chris Mar 16 '18 at 15:47
  • What kind of analysis does it do? Can it identify which lines are comments, declarations and the like? Could you expand your answer explaining all this? – Alejandro Mar 16 '18 at 16:08
  • Based on the documentation it produces, its analysis includes classes/interfaces and their inheritance relationships, #include files, member variables and methods (including full type signatures), and function/method call graphs. I'm not sure exactly what the xml output contains. I can't find any documentation on it specifically. The best thing is probably to check the schema: github.com/doxygen/doxygen/tree/master/templates/xml – Chris Mar 19 '18 at 19:32
0

Our DMS Software Reengineering Toolkit is a general purpose program analysis and transformation tool parameterized by explicit language definitions. To date there are some 40 languages (not counting dialects) defined for DMS.

For each such language defined by an explicit grammar (yes, we handle C++17 this way), DMS can parse the source and produce an abstract syntax tree. [Yes, you an export it as XML (JSON would be easy, too) if you want, but it is easier when do other analyses to do them with other DMS-provide mechanisms, meaning you don't have to export them. From there labelling comments vs declarations vs function headers vs. code is pretty easy to figure out.

DMS can be configured to pull out a lot more information than just this.

Improve this answer
  • 2
    When you work for the company that makes the product, it is normal practice to explicitly include a disclosure at the beginning or the end of the post (even though it is in your profile and you imply it by the use of "we") – Lyndon White Oct 16 '17 at 8:36
  • 1
    It is official Stack Overflow policy that the phrase "Our" ... is adequate disclosure. That was resolved many years ago. – Ira Baxter Oct 16 '17 at 14:57
  • 3
    Fair enough, I think the standards of this site should be higher than the network default. So I have made a meta-post: softwarerecs.meta.stackexchange.com/questions/2700 – Lyndon White Oct 17 '17 at 4:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.