How Remixware Was Tested: 2880 Planted Copies of Real Open Source Code

A tool that says “this code was copied” has to be right in two directions. It has to find the copies, including the ones that were tidied up, renamed or cut down. And it has to stay quiet about code that only looks like something else, because a report full of false alarms gets ignored. Remixware’s tests measure both, on real open source, and every test and benchmark number here comes from the files the tests write.

There are two kinds of test. The first is a set of small projects written for the tests, where every copy is planted by hand and every expected finding is written down. The second is a benchmark that copies thousands of functions out of real open source projects and counts how many Remixware traces back.

Hand-Made Projects

The test projects copy from a set of small open source projects under nine licence situations: MIT, BSD, ISC, zlib, Apache 2.0, MPL 2.0, GPL, AGPL and no licence. An online shop under MIT holds nine copies, a firmware tree under GPL-2.0-only holds three, and a clean project holds code that looks like the others without copying any of it. The tests check every finding by file, line range, source, licence and status, and fail on any finding that wasn’t expected. They also check that a scan gives the same answers when the project has Windows line endings, that a damaged or made-up index file is refused or can’t make a scan read files outside the folders it was made from, that a results file can’t make Remixware read files outside the project, that a file header can’t hide a stricter licence, and that the check, notice and check-again loop ends with a pass once the entries are finished. You can read the sample reports these projects produce.

The Benchmark

The benchmark clones 14 real open source projects at fixed versions: nine in Go (logrus, cobra, mux, uuid, errors, hcl, juju/errors, coyim and writefreely), requests and click in Python, express in JavaScript, and zlib and cJSON in C. Together they’re 2,784 source files and 514,366 lines, under MIT, Apache 2.0, BSD, MPL 2.0, LGPL, GPL, AGPL and zlib licences. Remixware indexes them, and the benchmark checks that it read each project’s licence correctly. It did for all 14, including coyim, which has two licence files: the GPL for the project and a BSD licence for code it took from another project.

Then it picks 300 Go functions and 60 each in Python, JavaScript and C, and copies each one into a test project six ways:

• verbatim, as it is;
• layout, reformatted;
• comments, with the comments changed;
• renamed, with every local name changed;
• partial, with only the middle part copied;
• combined, reformatted, with new comments and renamed, all at once.

That’s 2,880 copies. A copy counts as found when Remixware traces it to the function it came from.

What It Found

Remixware only reports a match of at least 50 tokens and 5 lines, so the results depend on the size of the copy:

Size of the copy	Found
Under 50 tokens	0 of 578, by design
50 to 99 tokens	960 of 1,188 (81%)
100 to 199 tokens	722 of 737 (98%)
200 tokens or more	372 of 377 (99%)

Copies of 100 tokens or more were found 98% of the time, 1,094 of 1,114, across all six kinds of change. Below that, the kind of change matters. Of the copies of 50 to 99 tokens, the ones copied as they were, reformatted or given new comments were found 96% of the time. Renamed ones of that size were found 55% of the time, because once the names are gone, a short function has the same shape as plenty of other code, and Remixware asks for more evidence before it reports one.

None of the 2,880 copies was traced to the wrong project. Across all sizes, including the copies too short to report, Remixware found 2,054 of them, 71%. Every miss is listed in the results with its size and source.

False Alarms

To count false alarms, the benchmark scans nine projects that aren’t in the index: urfave/cli, gin, BurntSushi/toml, chi and fatih/color in Go, koa in JavaScript, typer in Python, and sds and parson in C. That’s 924 files and 99,198 lines, written by different people for similar jobs, so they’re full of code that looks alike.

Remixware reported five matches in four of typer’s functions, and all five are real copies. Typer is built on click, which is in the index, and it reuses some of click’s code. One of those functions carries the comment “Copy from click.parser._split_opt”, another “Duplicate all of Click’s logic only to modify a single line”, and a third is click’s help formatter with arguments added. Remixware traced each match to the right place in click. There were no false alarms.

The SCANOSS Connection

With --scanoss, version 0.2 sends fingerprints of your files to the SCANOSS service. They have to be the ones SCANOSS’s own scanner makes, or the service would find less. A test compares Remixware’s fingerprints with the ones SCANOSS’s scanner wrote for files picked for their awkward cases: Windows and old Mac line endings, a file with no line break at the end, a very long first line, minified JavaScript, files on either side of the size where SCANOSS starts hashing, and text outside ASCII. They match byte for byte. Another test runs SCANOSS’s own sample of its service’s answers through Remixware and checks every finding, licence and status. Others play a service that limits requests, answers slowly, sends something that isn’t SCANOSS results, leaves files out of its answer, answers about files Remixware never sent, puts terminal commands in its text, or redirects to another host.

Then the real service. When 0.2 was finished, the free service’s limit for GitHub’s test machines was used up until the start of November, so the check was made from the owner’s own connection: the request Remixware sends for a test project with two copies of logrus code went to the service, and its answer went back through Remixware. SCANOSS traced both copies to logrus code, under the name of a fork with the same files: the whole file, and the copied function as a snippet. The record has the request and Remixware’s report.

Independent Reviews

Before 0.1.0, separate reviewers went through the code three times looking for ways to make Remixware wrong. The worst problems the first review found let a licence problem pass the check: a file header that only said where the code came from (“based on code released under the MIT license”) could make GPL code read as MIT, EUPL licences were read as GPL, a made-up index could make the HTML report show other files from the machine, and a check passed when it had scanned nothing.

The second review checked those fixes and found more problems, several of them in the fixes themselves. An index of a folder above the project could hide a vendored GPL copy inside it. A copy that needed both a review and a notice passed the check without the notice. A dual-licence header worded “Alternatively, this software may be distributed under the GPL” was read as asking for both licences. A made-up index could still crash a scan.

A third, shorter check of the new code found a few more. A GPL notice in the wording GCC uses, “either version 2, or (at your option) any later version”, was read as a choice between licences when a second licence followed it. A vendored GPL library inside the project counted as the project’s own code when the index held the whole project. All of them were fixed with tests, and every reviewer’s reproductions were run again on the fixed version, before the release.

One more turned up after 0.1.0 was out, in a review of this website. The MPL 2.0 text has an item that starts “(c) under Patent Claims”, and the Apache 2.0 text, wrapped the usual way, has a line that starts “(c) You must retain”. Remixware took lines like these for copyright lines. A copyright line counts as credit, so a notices file carrying the Apache or MPL text for any other library made a copied file under the same licence look credited. Version 0.1.1 fixes it. Lettered items, a licence document’s own copyright, such as the Free Software Foundation’s line at the top of the GPL, and templates like “Copyright (C) year name of author” no longer count as copyright lines. The fix was checked against every text in the SPDX licence list and against real licence files from the benchmark projects and a Linux system.

Version 0.2 had reviews of its own, aimed at the SCANOSS connection. The first found that an answer leaving files out was accepted, so a check could pass on files SCANOSS never looked at, that redirects were followed with the API key, and that text in an answer could clear the terminal and print a fake PASS. A second reviewer checked the fixes and found that the cleaning had gone too far: code credited by its own header failed a check, and a file with an everyday Unicode name, such as one with a wide space, could lose its finding. A third found a few smaller holes in that round. All of it was fixed with tests, and on the test projects and the benchmark projects 0.2 gives the same results as 0.1.1.

Speed

On a cloud machine with two processor cores, Remixware indexed the 514,366 lines of the benchmark projects in about a second, around 495,000 lines a second, into an index file of about 6 MB. It scanned the unrelated projects at about 262,000 lines a second.

Read the Results Yourself

Everything above comes from the files in test/results: the recall tables in benchmark/recall.txt, every miss in benchmark/misses.txt, every match in the unrelated projects with both sides of the code in benchmark/false-alarms.txt, the licence check in benchmark/corpus.txt, the live SCANOSS check in live/, and the full test log in tests.txt. scripts/record-results.sh makes them all again from scratch, apart from live/, which was made by hand with scripts/scanoss_live.py.