Security Evaluation

RIPE testbed

We evaluated all approaches against the RIPE security testbed.1 RIPE is a synthesized C program that tries to attack itself in a number of ways, by overflowing a buffer allocated on the stack, heap, or in data or BSS segments. RIPE can imitate up to 850 attacks, including shellcode, return-into-libc, and return-oriented programming.

To evaluate security of approaches, we disabled all other security features:

  • Linux ASLR was disabled via sudo bash -c 'echo 0 > /proc/sys/kernel/randomize_va_space'
  • All compiler optimizations were disabled via -O0
  • Compiler-inserted stack canaries were disabled via -fno-stack-protector
  • Compiler-enforced fortify-source was disabled via -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0
  • Executable stack was enabled via compiler flag -Wl,-z,execstack

Even under these relaxed security flags all compilers were susceptible only to a small number of attacks. Under native GCC, only 64 attacks survived, under ICC – 34, and under Clang – 38.


ApproachWorking attacks
MPX (GCC) default*41/64 (all memcpy and intra-object overflows)
MPX (GCC)0/64 (no working attacks)
MPX (GCC) no narrow bounds14/64 (all intra-object overflows)
MPX (ICC)0/34 (no working attacks)
MPX (ICC) no narrow bounds14/34 (all intra-object overflows)
AddressSanitizer12/64 (all intra-object overflows)
SoftBound14/38 (all intra-object overflows)
SAFECode14/38 (all intra-object overflows)

Note 1. In Col. 2, 41/64 means that 64 attacks were successful in native GCC version, and 41 attacks remained in MPX version.

Note 2. The “default” version of GCC-MPX means without -fchkp-first-field-has-own-bounds and with BNDPRESERVE=0, see below.

Surprisingly, a default GCC-MPX version showed very poor results, with 41 attacks (or 64% of all possible attacks) succeeding. As it turned out, the default GCC-MPX flags are sub-optimal. First, we found a bug in the memcpy wrapper which forced bounds registers to be nullified, so the bounds checks on memcpy were rendered useless.2 This bug disappears if BNDPRESERVE is manually set to one. Second, the MPX pass in GCC does not narrow bounds for the first field of a struct by default, in contrast to ICC which is more strict. To catch intra-object overflows happening in the first field of structs one needs to pass the -fchkp-first-field-has-own-bounds flag to GCC. When we enabled these two flags, all attacks were prevented; all next rows in the table were tested with these flags.

Other results are expected. MPX versions without narrowing of bounds overlook 14 intra-object overflow attacks, where a vulnerable buffer and a victim object live in the same struct. The same attacks are overlooked by AddressSanitizer, SoftBound, and SAFECode. Interestingly, AddressSanitizer has 12 working attacks, i.e., two attacks less than other approaches. Though we did not inspect this in detail, AddressSanitizer was able to prevent two shellcode intra-object attacks on the heap.

We performed the same experiment with only-writes versions of these approaches, and the results were exactly the same. This is explained by the fact that RIPE constructs only control-flow hijacking attacks and not information leaks (which could escape only-writes protection).

Up to table of contents


Below are the logs which show which attacks worked under each approach.

Raw results can be found in the repository.

Up to table of contents

Bugs in Benchmark Suites

During our experiments, we found 6 real out-of-bounds bugs (true positives). Five of these bugs were already known, and one was detected by GCC-MPX and was not previously reported.

The bugs found are:

  1. incorrect black-and-white input pictures leading to classic buffer overflow in ferret (PARSEC);
  2. wrong preincrement statement leading to classic off-by-one bug in h264ref (SPEC);
  3. out-of-bounds write in perlbench (SPEC);
  4. benign intra-object buffer overwrite in x264 (PARSEC);
  5. benign intra-object buffer overread in h264ref (SPEC);
  6. intra-object buffer overwrite in perlbench (SPEC).
ApproachBug 1Bug 2Bug 3Bug 4Bug 5Bug 6
MPX (GCC) no narrow bounds   
MPX (GCC) only writes 
MPX (GCC) no narrow bounds + only writes   
MPX (ICC) no narrow boundsNANANA 
MPX (ICC) only writesNANA  
MPX (ICC) no narrow bounds + only writesNA  

A more refined summary table as well as descriptions of the aforementioned bugs can be found in the Usability page.

Up to table of contents

  1. John Wilander and Nick Nikiforakis and Yves Younan and Mariam Kamkar and Wouter Joosen. RIPE: Runtime Intrusion Prevention Evaluator. In ACSAC’2011. 

  2. Note that in the final table under “Quick overiew of results” we show the RIPE results with this bug fixed.