Tool assisting 32 to 64 bits C++ migration on a huge code base

Question

TL;DR : I need tools to locate problematic code in a 32 to 64 bits C++ Linux migration (tools can be under Linux or Widows).

So here it is, I have a huge c++ application (we are talking more that 8M lines of code) which I wish to port from a 32 bits platform to a 64 bits one (on linux). The reason of porting the code is not questionable, my task is to do so. The app is totally functional on 32 bit but the code can be really old in some part and is clearly not coded with recent standards and good practices. In order to succeed I need to locate all code fragments which will be problematic when compiled in 64bits without spotting too much false positive since we are talking millions on lines and i don't have resources to refactor all code to make it clean and pretty. So finally my question is what tools can help me in this task?

I considered PVS-Studio but it seems designed for Windows app. I also considered using compilers messages like Gcc and Clang warning but there is way too much false positive since in the 32 bits case there is already a lot of them. I also considered static analyzing tools like Cppchack, Klowork, Gimpel Pc-Lint and FlexeLint or "Parasoft C++ test" but i don't really know those tools and if they can really help me.

In short, the patterns I need to locate are (ask me if you need more detail) :
- problematic use of types with different size in 32 and 64 bits
- overflows induce by the migration
- problematic magic values
- memory alignment change in structure, union and so
- bad use of format specifier (like in printf(“%u”, val); if val is a long) - problematic implicit cast
- methods and functions that no longer match (virtual and overloaded method)

The 4 key points are that :
- I need to find all problems induced by the migration or to know what types of problems i missed if i can't find them.
- I need to have few false positive or a way to quickly eliminate them (need automation)
- The solution have to be industrial (i.e scriptable or automatic). It's a huge code base, by hand work is not possible.
- The solution can be one tool or a set of tools, free or not, Linux or Windows.

It would be a plus if the tool comes with a nice and clear documentation.

If you ever experienced this kind of problem, thank you for any advice you can give me.

Answer 1

Yes, I have done tasks with large-scale analysis and modification of code, including C++. I build tools explicitly to handle this.

Our DMS Software Reengineering Toolkit with its C++ Front End might be of help.

(Since this is not a off-the-shelf solution to OPs problem, I can't make a recommendation here, only point out its existence. I will assert it is a viable solution to his problem as stated).

What DMS offers is the ability to configure a custom ("scripted") analysis tool, focusing on the specific needs of a particular code base. Its C++ parser can read C++14 source code of GCC or MSVS style, build full ASTs, compiler-accurate symbol tables, and local control and data flow analysis for methods/functions. It has built-in support to analyze ranges of values which a variable can be. With these foundations, it is often practical to build an focused analyzer that detects specific problems of interest.

To the extent that a problem is detected and has a known cure (imagine copying a 32 bit pointer into a 32 bit int; on a 64 bit machine, the 32 bit int has to be redeclared to be 64 bits), DMS can apply source-to-source program transformations to modify the ASTs and then regenerate source code from the modified trees. This can be a big help in managing the cost of change after detection. DMS has been used for other mass change tasks applied to C++ source code.

Regarding custom analyzers needed for OP's tasks:

• data type size differences: DMS has full type information for every declared entity, and computes types of every level of every expression just as a C++ compiler does. You can use this information to detect mismatches between the type of a source value (e.g., pointer [which now must be 64 bits]) and its target (e.g., a struct slot of declared 32 bit size).
• overflows induce by the migration: knowing the types/ranges of summands, one can estimate the range of the sum, and whether the target can accomodate a carry-out of an add.
• problematic magic values: if you can enumerate these, they are easy to detect. I think the problem here isn't the values, but the context in which they are found. DMS has the dataflows to find the context so it can be checked.
• memory alignment change in structure, union and so: DMS has the type information. Where the "new" type must be different from 32 bits, you can check the alignment of the target types (e.g., struct slots, etc.)
• bad use of format specifier (like in printf(“%u”, val); if val is a long): This will require interpreting the format identifier, and checking the arguments. DMS has the type information necessary
• problematic implicit cast: again, you need type information for the original types, and check whether the implicit cast streches/shrinks source value inappropriately
• methods and functions that no longer match (virtual and overloaded method): DMS knows (for the valid 32 bit version) what the proper overload matches are. If operands change size, one can do a relatively straightforward check to determine if the argument size is declared accordingly (or have DMS simply change the operand size).

None of these are likely as simple as I have suggested. All of them together are still likely to be a lot less work than trying to hand-detect/patch problems across 8M SLOC.

Your key points:

• Find all problems: You can build custom analyzers for each specific issue. Some will be easier than others.
• Minimize false positives: Some custom analyzers will be hueristic vs. algorithmic depending on effort invested and difficulty of analysis. Welcome to automated program analysis, and the Turing tarpit. Take the help you can get.
• Industrial: DMS has been used for a huge variety of complex tasks, in av variety of languages, including with C++. It is scriptable in a variety of ways; DMS was designed to handle large scale.
• DMS is a "tookit"; an assemblage of tools and libraries. It is commercial. It runs natively under Windows; it has been thoroughly tested u under Linux/Wine.
• Documentation: DMS is designed to be used by 3rd parties. Thousands of pages of online documentation, tutorial slides, and we offer a training class if you want it.

DMS can run on Windows, or on Linux under Wine.