looking for python script to extract and reorganize tagged text

Question

I am looking for a python script that will take a text file and extract and reorganize blocks of text in other files based on how those blocks are tagged. Before writing this myself (my coding is slow), wanted to check if someone had written a similar script that I could adapt. These are the steps:

1. opening the source / input file and reading to memory
2. identifying each text block delimited by tags of the type [#tag1]text or whitespace[#tag2]
3. identifying if there is already a file named with the found tag and creating a file with that tag if not
4. pasting the content found for that tag in the appropriately named file

How the tagging is applied to the source date is negotiable but at this point, I don't need hierarchical tagging. E.g. of the source data:

[#idea]this block contains an idea and 
where I stumbled on it
[#booknote]this block is a quote from a specific text
[#idea][#todo]todo item associated with an idea for some endeavor; would be duplicated in files corresponding to each of those tags

I originally posted this question (https://writing.stackexchange.com/questions/54338/software-for-organizing-blocks-of-text-within-files/54364#54364) on the Writing sub-site hoping to find an application to do this but didn't find one that met my needs.

Answer 1

Here is a piece of code to, first, make a list of your tags. If I have some time I will proceed next later ;).

    import glob
    import re
    
    list_of_tags = []
    
    for filename in glog.glob("*_yourfiles.dat"):
        with open(filename, 'r') as msg:
            data = msg.read()
        tags = re.sub(r"(?!tag[^\s]+\b)\b\w+","",data)
        list_of_tags.append(tags)

    list_of_tags = list(set(list_of_tags.append))

You may need this list to proceed with re.sub on each file between tag of interest and next tag. You remove all but not the blocks of interest (or the reverse and you match only groups of interest). And you append to file.

If you have a lot of data, using in and re.sub to match is a must.

Well, here is what comes next, finally had time:

    import re
    
    list_of_tags = ['[#tagA1]','[#tag2B]','[#tagC]']
    
    string = "here is [#tagA1] and even more here with [#tag2B] why not of repeat of [#tagA1] and a last [#tagC] which is fine"
    
    for tag in list_of_tags:
        list_of_match = []
        #print (string) #1
        wstring = string.replace(tag,"__@_")
        #print (wstring) #2
        wstring = re.sub(r"(_@_[^\[]+)",r'\1'+'_@_',wstring)
        #print (wstring) #3
        for match in re.finditer(r"_@_[^@]+_@_",wstring):
            print (tag,wstring[match.start():match.end()])
            list_of_match.append(wstring[match.start():match.end()])
    
        with open(tag+"_outfile.dat", "a") as msg:
            for match in list_of_match:
                msg.write(match.replace("_@_","").strip()+"\n")

Note that the format of your tags is pretty horrible, you should not use special characters that may have to be escaped. Things like "@" or others unlikely combination to embed your tag would clearly be better.

String processing for first tag:

here is [#tagA1] and even more here with [#tag2B] why not of repeat of [#tagA1] and a last [#tagC] which is fine
here is __@_ and even more here with [#tag2B] why not of repeat of __@_ and a last [#tagC] which is fine
here is __@_ and even more here with _@_[#tag2B] why not of repeat of __@_ and a last _@_[#tagC] which is fine

So basically, you replace your tag by something you can work with easily (something generic/not this format).

Then, you add the same 'end' generic tag to your expression starting with the generic tag you just used to replace the tag of interest.

Then, you re.finditer() to get all expression between two tags with no "[" in between (cause you don't want to embed other tags in that).

Then, you fill a list with the matches. And you append to the corresponding file with the list content.

Output:

[#tagA1] _@_ and even more here with _@_
[#tagA1] _@_ and a last _@_
[#tag2B] _@_ why not of repeat of _@_
[#tagC] _@_ which is fine_@_

In 3 different files with terrible names:

$ cat '[#tagA1]_outfile.dat'

and even more here with
and a last

For sure you can use the same glob.glob() for loop as seen in the first part, to loop over files and over tags, or the opposite, up to you.

Certainly not the best optimization overall but should still be pretty fast, with those very annoying tags that challenge my not too extended skills in regex.