We evaluated all approaches against the RIPE security testbed.1 RIPE is a synthesized C program that tries to attack itself in a number of ways, by overflowing a buffer allocated on the stack, heap, or in data or BSS segments. RIPE can imitate up to 850 attacks, including shellcode, return-into-libc, and return-oriented programming.
To evaluate security of approaches, we disabled all other security features:
- Linux ASLR was disabled via
sudo bash -c 'echo 0 > /proc/sys/kernel/randomize_va_space'
- All compiler optimizations were disabled via
- Compiler-inserted stack canaries were disabled via
- Compiler-enforced fortify-source was disabled via
- Executable stack was enabled via compiler flag
Even under these relaxed security flags all compilers were susceptible only to a small number of attacks. Under native GCC, only 64 attacks survived, under ICC – 34, and under Clang – 38.
|MPX (GCC) default*||41/64 (all memcpy and intra-object overflows)|
|MPX (GCC)||0/64 (no working attacks)|
|MPX (GCC) no narrow bounds||14/64 (all intra-object overflows)|
|MPX (ICC)||0/34 (no working attacks)|
|MPX (ICC) no narrow bounds||14/34 (all intra-object overflows)|
|AddressSanitizer||12/64 (all intra-object overflows)|
|SoftBound||14/38 (all intra-object overflows)|
|SAFECode||14/38 (all intra-object overflows)|
Note 1. In Col. 2, 41/64 means that 64 attacks were successful in native GCC version, and 41 attacks remained in MPX version.
Note 2. The “default” version of GCC-MPX means without
-fchkp-first-field-has-own-bounds and with
BNDPRESERVE=0, see below.
Surprisingly, a default GCC-MPX version showed very poor results, with 41 attacks (or 64% of all possible attacks) succeeding. As it turned out, the default GCC-MPX flags are sub-optimal. First, we found a bug in the
memcpy wrapper which forced bounds registers to be nullified, so the bounds checks on
memcpy were rendered useless.2 This bug disappears if
BNDPRESERVE is manually set to one. Second, the MPX pass in GCC does not narrow bounds for the first field of a struct by default, in contrast to ICC which is more strict. To catch intra-object overflows happening in the first field of structs one needs to pass the
-fchkp-first-field-has-own-bounds flag to GCC. When we enabled these two flags, all attacks were prevented; all next rows in the table were tested with these flags.
Other results are expected. MPX versions without narrowing of bounds overlook 14 intra-object overflow attacks, where a vulnerable buffer and a victim object live in the same struct. The same attacks are overlooked by AddressSanitizer, SoftBound, and SAFECode. Interestingly, AddressSanitizer has 12 working attacks, i.e., two attacks less than other approaches. Though we did not inspect this in detail, AddressSanitizer was able to prevent two shellcode intra-object attacks on the heap.
We performed the same experiment with only-writes versions of these approaches, and the results were exactly the same. This is explained by the fact that RIPE constructs only control-flow hijacking attacks and not information leaks (which could escape only-writes protection).
Below are the logs which show which attacks worked under each approach.
- Native versions:
- MPX versions:
- AddressSanitizer versions:
Raw results can be found in the repository.
Bugs in Benchmark Suites
During our experiments, we found 6 real out-of-bounds bugs (true positives). Five of these bugs were already known, and one was detected by GCC-MPX and was not previously reported.
The bugs found are:
- incorrect black-and-white input pictures leading to classic buffer overflow in
- wrong preincrement statement leading to classic off-by-one bug in
- out-of-bounds write in
- benign intra-object buffer overwrite in
- benign intra-object buffer overread in
- intra-object buffer overwrite in
|Approach||Bug 1||Bug 2||Bug 3||Bug 4||Bug 5||Bug 6|
|MPX (GCC) no narrow bounds||✔||✔||✔|
|MPX (GCC) only writes||✔||✔||✔||✔||✔|
|MPX (GCC) no narrow bounds + only writes||✔||✔||✔|
|MPX (ICC) no narrow bounds||✔||NA||✔||NA||NA|
|MPX (ICC) only writes||NA||✔||✔||NA|
|MPX (ICC) no narrow bounds + only writes||✔||✔||✔||NA|
A more refined summary table as well as descriptions of the aforementioned bugs can be found in the Usability page.