Seeking concrete research on the security impact of memory unsafety


#1

Hi.

As I tend to do, lately I’ve been collecting evidence about the impact of memory-unsafe languages on computer security, for the purposes of strengthening the Rust argument. I’ve got plenty of (yet unsorted) links to opinions and clever quotes and some ad-hoc analyses (mostly from w/in the Rust community). I don’t think I’ve got enough firm data from reputable sources to be convincing to skeptics.

Is anybody aware of strong sources for concrete numbers like: quantity of code written in unsafe languages; percent of security vulnerabilities that are due to memory unsafety; financial impact of specific vulnerabilities, and in aggregate; financial impact of maintenance due to debugging segfaults; what proportion of the C++ industry defected to Java when it rolled out. I’m also interested in historic perspective - what was our industry thinking about this subject 30 years ago (need real sources, not anecdotes)?

I did some cursory searches of arxiv.org, but I have not much patience.

So does anybody have any great sources on this subject?


#2

roc did some analysis of security critical bugs in Firefox related to WebAudio. His original email is here: https://groups.google.com/d/msg/mozilla.dev.servo/ufJM51SF3G4/5nfMejsQd5UJ


#3

This is an excellent paper on the state of memory safety, but I’m not sure if it has the numbers you’re looking for. https://people.eecs.berkeley.edu/~dawnsong/papers/Oakland13-SoK-CR.pdf


#4

https://www.vvdveen.com/memory-errors/ The corresponding papers links to more references.

OWASP top 10 is slated to be updated in the near future


#5

Thanks @jethrogb and @metajack!


#6

This blog post about curl’s reason for using C led some commenters to go through all the known curl vulnerabilities from the past seventeen years and count how many were directly caused by memory safety violations.

23 of the 61 reported security bugs (about 38%) were caused by memory unsafety according to this tally.


#7

One paper that comes to mind is “A Large Scale Study of Programming Languages and Code Quality in Github” by Baishakhi Ray et al (DOI). They analyze a large dataset (728 projects) for relationships between languages and code quality. You can read the PDF version.

The dataset is built by taking the top 19 programming languages on Github, and then taking the top 50 projects for each language. They filter out projects with very short development histories (< 28 commits), leaving 728 projects.

They separate out “security” as a defect impact class, while “memory” is a defect cause. Looking at their examples, I would potentially consider many of the “memory” defects to be potential security vulnerabilities (although they might not be simple or directly exploitable). I’ll include results for “security” and “memory” defects below.

They break down languages into a few different axes:

Some potentially relevant results:

  • 432 of the 728 projects in their sample have at least 20 commits in C, C++, or Objective-C. In total those projects have 35,035 KLOC (out of 62,840 KLOC total across all projects).

    • It is unclear if these SLOC numbers are for that language or for all projects that include that language
  • They find that procedural, static typed, weak typed, unmanaged memory languages (C, C++, Obj-C) are more defect prone (see Tables 6 and 7):

    Comparing the other language classes to the grand mean, Proc-Static-Weak-Unmanaged languages are more likely to induce defects. This implies that either weak typing or memory management issues contribute to greater defect proneness as compared with other procedural languages.

  • They visualize the relationships between language class and bug categories using a heat map [page 8], and also investigate the relationships using regression models.

    • They find a strong relationship between proc-static-weak-unmanaged and memory errors.
    • They find that Erlang, C++, and Python produce more security errors than average.


#8

For code “demographics”, there’s a lot that can be done using the GitHub datasets in Google BigQuery. There are historical event log records going back many years, as well as commit and file data for every project. For example, it would be possible to do things like write a query to count the number of commits to C files in the last year. If there’s specific data you’d like to investigate, I’d be happy to help write up the queries. For more information on these datasets and how people are using them:


#9

Awesome writeup @cthompson. Thank you!