4

I have a large intranet containing tens of thousands of web pages developed over the course of the past 15 years. These pages are in the process of being updated with the HTML5 doctype. Among the complications: the old doctype we were using did not care about the case of CSS class names and ID names.

In the old doctype, class="cellcolor" would work as well as class="CellColor". So, we have to fix the case on every page when we apply the HTML5 doctype.

I've been developing a regex script (using Funduc Search & Replace) to run against the entire intranet, fixing class and ID names so the case matches our style sheets.

The problems are twofold:

  • (1) SPEED: The tool runs very slowly. If we don't find an alternative, we're going to have to let several machines run day and night for days or weeks to hit every page. Is there a more robust tool that will work faster?

  • (2) SYNTAX: The Funduc S&R regex syntax is somewhat limited, missing some features that would be very helpful. I'm hoping I can also find a tool with a more robust implementation of the language.

Any insight or advice would be appreciated!

8
  • What operating system(s)? Gratis or paid? Open-source or closed-source? Apr 13, 2015 at 23:15
  • What is the page throughput rate you are getting? Perl is considered to be reasonably fast;why would that not be appropriate? It has pretty powerful regex, but that may not be enough.
    – Ira Baxter
    Apr 14, 2015 at 4:12
  • 1
    Any tool that uses regex for this job will likely fail at this scale; regex can't parse nested text structures. Imagine that you text is found inside a text string inside a fragment of JavaScript. Do you want to change that JavaScript stringliteral? How do you know? If you want a tool that can to this right, you need a tool that can actually parse the HTML accurately.
    – Ira Baxter
    Apr 14, 2015 at 4:21
  • @RockPaperLizard: I would run this from either Windows 7 or Windows 8 machines. I don't have a preference for open- or closed-source. What do you mean about gratis or paid? Apr 14, 2015 at 13:15
  • @Ira Baxter: I'm getting about .9 seconds per page right now. Not terrible, really, but there are a lot of pages. You raise a good point about JavaScript: This has been one of my concerns. So I'm dying to know: What is a tool that can parse HTML accurately? Apr 14, 2015 at 13:18

2 Answers 2

2

Other folks recommend solutions involving regex. That won't be reliable; regex does not work for parsing nested structures. In essence, you need a parser.

You may be able to hack together something in a procedural language using regex, where the procedural language code in essence is the parser, but then that's a solution that uses a little bit of regex and a parser. Writing a full parser for HTML5 this way is going to be a lot of work; you may be able to cheat a lot and build island parsers to pick out just what you want, but island parsers tend to fragile and pretty fiddly.

A parser based solution reads your text, parses it into the HTML5 structures, and then applies the patches you want want to make. Writing a set of procedures that crawl the tree, sniff nodes, and make local tree splices to fix things is possible but awkward, esp. if the set of changes is varied or is large. Then you need to prettyprint the tree with the changes. It is hard to assemble all these bits and make it work esp. for a one-shot task.

An alternative is to use a Program Transformation System (PTS), which has all the parsing and prettyprinting machinery built in, and will let you code surface syntax source-to-source transformations of the form "if you see this, replace it with that". The transformations then match the this part of the rules against the parse tree; where matches are found, the that part is spliced in as a replacement. This makes for accurate matching and replacement; you can only replace the structures you intend to replace. Typically you have to provide a PTS with a grammar (so it can parse) [and often some hints about prettyprinting embedded in the grammar]. PTS are usually applied to systems of files that make up an application.

Our DMS Software Reengineering Toolkit is such a PTS, explicitly designed to carry out mass-change tasks to source code.

DMS has a full HTML4 (with CSS) parser and prettyprinter already available. DMS provides the ability to manage language dialects; it should be straightforward to define HTML5 using the HTML4 base.

With that in place, OP could write his patches directly using DMS rules for HTML5 syntax source-to-source transformations such as:

 rule replace_cellcolor(): CSS_attribute -> CSS_attribute
  "   class=\"cellcolor\"   "  ->  " class=\"CellColor\" ";

This is a rule named replace_cellcolor. It will only match CSS_attributes, as defined by a nonterminal in the grammar; it can't match anything else. The rule transforms a CSS_attribute into a CSS_attribute (you could map it to something else, but that's not useful for this particular activity). The pattern is

      class=\"cellcolor\"

written inside rewrite-rule-language metaquotes ".....". Because of the metaquotes we have to meta-escape \" the quotes actually used in the CSS syntax. The rule shows this -> that as source pattern in metaescapes, and target pattern in meta-escapes. That's all that is needed for OP's specific patch. What you write is essentially fragments of the targeted langauge; DMS interprets these as specifications for tree patterns, and uses these tree patterns to match the parsed tree. Since DMS parses these patterns using the same parser as it uses for parsing the source files, the tree patterns it gets are exactly the ones you want.

OP likely has a big set of these capitalization fixes he wants done. He writes one DMS rule per capitalization fix, and organizes them in a set:

  ruleset fix_CSS_capitalization = 
        replace_cellcolor 
        + replace_xyx
        + ... ;

DMS will apply the entire set of rules automatically in one of its modes of operation. You need a bit of DMS metaprogramming code (not shown here) to open a designated set of files, parse each file, apply the rules, prettyprint the file; this is pretty standard for uses of DMS. Other applications of DMS that make such changes can run through thousands of files per hour.

Summary, DMS:

  • Uses a parser, not regex, to accurately interpret the text
  • Allows one to easily express exactly the change desired, using the syntax of the target language (HTML5)
  • Allows one to express the set of changes
  • Applies the rules automatically
  • Prettyprints the answer.
  • Can apply changes to multiple files.
2
  • Ira, Thanks very much for the incredibly detailed and thoughtful answer. You've given me much more than I asked for -- a real education on a broader topic of maintaining large code systems. I appreciate it. I looked at the SD web site for pricing information about DMS but was unable to find it. Can you give me an idea of how much it costs? Apr 15, 2015 at 14:22
  • SO isn't the place for discussions about price. Contact the company.
    – Ira Baxter
    Apr 15, 2015 at 15:17
1

As usual I would recommend python for the following reasons:

  • Full and powerful regex
  • You can even comment your regex if you use verbose mode
  • You will probably need os.walk to traverse local file systems.
  • Cross platform & free - you could install it on each machine and simply let each machine update it's local pages.
  • A lot faster than I expected.
  • You could also use the scapy library to scrape the intranet so as to locate the pages that need updating.

One thing to remember is that pages that do not contain HTML5 will not need updating so you can test for that first.

An alternative approach

The other approach - not to use regular expressions would be to parse each document, modify the elements, and write it back out. Luckily there is also a python library for parsing HTML called Beautiful Soup which is a Python library for pulling data out of HTML and XML files.

With it you could load the document, replace the appropriate attributes, and output it in the appropriate format, (hopefully remembering to create a backup first). If you have more than a few items to potentially modify it would probably be quicker to pick python+BS up than developing and testing all the regexs.

1
  • Good advice, Steve. Thanks very much. I don't know Python and don't have time to learn it in the timeframe of the project I'm working on. But I definitely need to make a point to become familiar with it, and your comment has increased my interest in it quite a bit! Apr 14, 2015 at 14:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.