Code clone detection software

Question

I am looking for a simple tool that can identify similar parts in one or multiple files.

I am working on a fairly old codebase where developers used to use copy paste a lot. Eventually we have hundered of C files that are about 10k lines long.

I would like to identify the similar code with a not too complex tool.

What software or tool can I use for this job?

A first approach will be to consider one 10'000 lines file which contain C functions. Inside these long functions I can manually identify blocks that are very similar to some others. The difference is sometime the name of the variables foo_m0 and foo_m1 or the index of a table foo[0] and foo[1].

Pointing out these regions of similar code is not always easy in such long files. Using a tool that can identify the longest redundant block and the most repeated along the file will be very useful to plan refactoring actions.

Answer 1

I have used Simian, which

identifies duplication in Java, C#, C, C++, COBOL, Ruby, JSP, ASP, HTML, XML, Visual Basic, Groovy source code and even plain text files

It is a commercial software with different licenses. and it can ignore identifiers, braces etc. during the comparison, so you can not only find exact duplicates but structural duplicates.

It worked fine, but seems always a bit behind state of the art (e.g. support for LINQ queries in C#) at the time I used it. Therefore I doubt it would work very well if you use the latest C++ standard. It'll still do a good job though.

I'm not sure how it integrates into your IDE. It was possible to use it in C# with a ReSharper plugin called Agent Ralph. Maybe something similar exists for C++.

Answer 2

Let me recommend these 3 tools, that might solve your problem:

http://www.solidsourceit.com/products/SolidSDD-code-duplication-cloning-analysis.html - best, but not free (in fact a bit expensive)

http://www.harukizaemon.com/simian/

http://duplo.sourceforge.net/

Answer 3

There are a number of tools that can help to find direct copies/duplicate code in C/CPP code, as are pointed out in the answers to this stack overflow question. As pointed out many of the detection mechanisms fail simply because of simple indentation changes.

Personally I have found that one of the best mechanism for code duplication detection is to refactor the large files into smaller files grouped by purpose and given meaningful names, from a maintenance and compile time prospective this is almost always worth doing anyway. You soon find that the name you need already exists and can then look to either use only one version or to generalize/abstract the code. I have seen examples where what the code was supposed to be doing was identical other than the values of some indexes, actually in the case I am thinking of there were several copies, taken at different times, some missing bug fixes that had been found, generalizing could be by passing an enum to specify which use case and using that to select the index(es) or by directly passing in the index and range checking before use.

Answer 4

Our CloneDR tool can do this for large code bases with many files (or just one file if you insist). It finds:

• exact clones. Using the language structure (ASTs) to guide the matching process, ensuring that clones detected are sensible blocks of code. This also means that it will find clones in any language structure (whole functions, statement sequences, big expressions, ...) in spite of changed code layout (indentation, whitespace, presence/absence of comments, constants of same value but different radix, ...). If there is a set of several (or many!) blocks of code that are clones, the set gets reported, instead of pairs. Exact clones are reported by exact source location.
• near miss clones, including expressions substituted for variable names and vice-versa, as OP gives as examples, and in many cases places where statements have been inserted or deleted. (This often detects broken clones, where a fix was not propagated, as discussed by Steve Barnes in his answer). Near-miss clones are reported by location, with parameters showing how they differ, and how the parameters can be bound to produce the exact clones in the source code. This is almost a subroutine declaration for the commonality.

CloneDR typically finds 10+% duplicated code in software that is relatively well engineered. These numbers can be significantly larger in sloppy source code (we have a real 100K SLOC COBOL example that is 55%+ duplicated code, yikes).

CloneDR uses a language-precise parser to discover the code structure. There are versions for many languages, including various dialects of C (through C11) and C++ (through C++14 with C++17 coming along) and especially including MS-specific dialects. [We try to keep our langauge front ends up to speed. CloneDR works for C# 6, including the LINQ commands mentioned by another answer].

Setting CloneDR up to run is fairly easy: you construct a project file listing the source files you would like analyzed, plus a few parameters that define what "near miss" means (x+y is a clone of a+1 with a variable substution, but you really don't want to see those).

Because of complications with parsing source code with preprocessor directives without expanding them, for C and C++, you may have to configure CloneDR for your specific code base, by telling it about your more egregious macros or conditionals that make standalone parsing difficult. This can be done for million line systems with pretty modest effort. C usually requires a bit of effort here. C++ often does not; C++ programmers don't use the preprocessor much. Other languages supported by CloneDR without preprocessors don't need this special step.

There are example reports of detected clones at the site for a variety of languages. The download demo will determine all clones (even in a big system) and provides an analysis of how clony the analyzed software is; it will also show some of the clones.

Since this is my tool, don't take this as a recommendation; just documenting its existence. It is based on my long experience writing code, and there's a technical paper describing essentially how it works: Clone Detection Using Abstract Syntax Trees.