python module to search for multiline pattern in text

Question

Sometimes I have to do complicated refactorings in a multi-language project. "Complicated" as in, far from stuff an LSP-server would allow, as a "rename an identifier". A recent example: find initialization of a specific array in a C file, for each element in it figure out if it's a string or an identifier (or combination of both), in case of identifier find its declaration in another file; then basically do some modifications across a few files based on the string/identifier naming pattern.

For C-language there's Coccinelle for that, but it is language-specific (so no cross-language refactorings) and is hard to debug too.

What I usually do is I write a parser which opens a file, goes through it line-by-line, trying to detect the pattern, while keeping the current state if pattern is multiline. So you basically get a state machine.

This is a monotonous work that I have to repeat, and I'd like to abstract it out. I'm wondering if there's any python module that, given a multiline pattern (perhaps in an EBNF grammar or something similar), would search for it in a file/text, and return all matches along with their locations.

What I tried so far:

• Lark. I was told on Gitter that it doesn't allow to just write the pattern to search for, instead it requires to have an EBNF that covers the whole file. This would be a lot of work. Even if I find an EBNF someone have already written, I feel like I could stumble upon corner-cases (such as syntax introduced in a newer language version or compiler-extensions); also, working with the AST of the whole file whereas I only need a specific portion of it sounds unwieldy.
• multiline regexp: I wasn't able to experiment with this too much to have a reasonable opinion. For starters, support is needed for both multiline matching and "nested structures" matching (e.g. balanced parentheses). regex python module is able to do both, but then I don't think it has a way to get "nth balanced expression". Substituting (?R) with (?n) where n is a number results in exception when text has more similar balanced expressions with depth less than n.

Answer 1

A few people told me to look at pyparsing. Having experimented with it for a day, I must say it does fit the requirements. It can parse multiline text and it can track nested structures (aka balanced expressions, e.g. pairs of parentheses). And although I haven't figured myself how to get nth balanced expression (it seems to be tricky, apparently involves using infixNotation rather than its wrapper nestedExpr), but judging by questions on StackOverflow it is possible.

Basically, pattern is created from token-classes (such as Word, Char, etc) in python code. There's Regex class too, so for example you could replace Word with Regex('\w+') if you want to. Then, once you got a pattern, you call a pattern.searchString(mystring) (or scanString) and you get matches in the text. There's a lot of helper functions too, like parseFile to parse a file.

pyparsing docs has a nice "getting started" section, but I think it is worth mentioning some essential things that left me confused for a while.

1. Match positions in text: by default pyparsing only prints them for the whole match (or not at all if you used searchString) which itself may consist of many small tokens. It is not very useful since most often a pattern comprises a non-interesting context (which is only used to sort out false-positives) and the actual core that one wants to extract.

   To make position of that "core" to appear, once you created it from tokens and before you meld it into the rest of the pattern, you call on it myPatternObj = locatedExpr(myPatternObj).

2. Token positions in match results: they can be referred by index, but it is error-prone, since adding a new token will make token positions change. Instead you can name the token, and then refer to it by name in results. Example of how to create a named token: token = Word(alphanums)('token_name')

3. Extracting group values from regexps: currently, getting them with index does not seem to be implemented. Instead, you have to use named groups (example of one: (?P<name>content follows)), which makes them behave as named tokens, see the prev. point.

4. Missing state from prints for ParseResults: there's a report for that. Basically, if you apply print() or str() to parse results, you will only see the matched text, you won't see key:value pairs for locn_start or the token name. It is purely a visual problem, e.g. you still can refer to the token by name if it was named. But as you will need to see state for debugging purposes, workarounds here are:

   1. call print(obj.dump())
   2. do debugging inside python interactive shell (result of evaulation of an object there shows all state)