forked from ShiftLeftSecurity/Benchmark
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathOWASP_Benchmark_Home.html
189 lines (157 loc) · 9.33 KB
/
OWASP_Benchmark_Home.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
<meta name="description" content="">
<meta name="author" content="">
<link rel="icon" href="../../favicon.ico">
<title>Guide to the OWASP Benchmark v1.1,1.2</title>
<!-- Bootstrap core CSS -->
<link href="content/css/bootstrap.min.css" rel="stylesheet">
<!-- Custom styles for this template -->
<link href="content/dashboard.css" rel="stylesheet">
<!-- Just for debugging purposes. Don't actually copy these 2 lines! -->
<!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]-->
<script src="content/js/ie-emulation-modes-warning.js"></script>
<!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]-->
</head>
<body>
<nav class="navbar navbar-inverse navbar-fixed-top">
<div class="container-fluid">
<div class="navbar-header">
<a class="navbar-brand" href="OWASP_Benchmark_Home.html">OWASP Benchmark v1.1,1.2</a>
</div>
<div id="navbar" class="navbar-collapse collapse">
<ul class="nav navbar-nav navbar-right">
<li><a href="OWASP_Benchmark_Home.html">Home</a></li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Tools<span class="caret"></span></a>
<ul class="dropdown-menu">
<li><a href="Benchmark_v1.2_Scorecard_for_FBwFindSecBugs_v1.4.0.html">FBwFindSecBugs v1.4.0</a></li>
<li><a href="Benchmark_v1.2_Scorecard_for_FBwFindSecBugs_v1.4.3.html">FBwFindSecBugs v1.4.3</a></li>
<li><a href="Benchmark_v1.2_Scorecard_for_FBwFindSecBugs_v1.4.4.html">FBwFindSecBugs v1.4.4</a></li>
<li><a href="Benchmark_v1.2_Scorecard_for_FBwFindSecBugs_v1.4.5.html">FBwFindSecBugs v1.4.5</a></li>
<li><a href="Benchmark_v1.2_Scorecard_for_FBwFindSecBugs_v1.4.6.html">FBwFindSecBugs v1.4.6</a></li>
<li><a href="Benchmark_v1.2_Scorecard_for_FindBugs_v3.0.1.html">FindBugs v3.0.1</a></li>
<li><a href="Benchmark_v1.2_Scorecard_for_OWASP_ZAP_vD-2015-08-24.html">OWASP ZAP vD-2015-08-24</a></li>
<li><a href="Benchmark_v1.2_Scorecard_for_OWASP_ZAP_vD-2016-09-05.html">OWASP ZAP vD-2016-09-05</a></li>
<li><a href="Benchmark_v1.2_Scorecard_for_PMD_v5.2.3.html">PMD v5.2.3</a></li>
<li><a href="Benchmark_v1.1_Scorecard_for_SAST-01.html">SAST-01</a></li>
<li><a href="Benchmark_v1.1_Scorecard_for_SAST-02.html">SAST-02</a></li>
<li><a href="Benchmark_v1.1_Scorecard_for_SAST-03.html">SAST-03</a></li>
<li><a href="Benchmark_v1.1_Scorecard_for_SAST-04.html">SAST-04</a></li>
<li><a href="Benchmark_v1.1_Scorecard_for_SAST-05.html">SAST-05</a></li>
<li><a href="Benchmark_v1.1_Scorecard_for_SAST-06.html">SAST-06</a></li>
<li><a href="Benchmark_v1.2_Scorecard_for_SonarQube_Java_Plugin_v3.14.html">SonarQube Java Plugin v3.14</a></li>
<li><a href="Benchmark_v1.1,1.2_Scorecard_for_Commercial_Tools.html">Commercial Average</a></li>
</ul>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Vulnerabilities<span class="caret"></span></a>
<ul class="dropdown-menu">
<li><a href="Benchmark_v1.1,1.2_Scorecard_for_Command_Injection.html">Command Injection</a></li>
<li><a href="Benchmark_v1.1,1.2_Scorecard_for_Cross-Site_Scripting.html">Cross-Site Scripting</a></li>
<li><a href="Benchmark_v1.1,1.2_Scorecard_for_Insecure_Cookie.html">Insecure Cookie</a></li>
<li><a href="Benchmark_v1.1,1.2_Scorecard_for_LDAP_Injection.html">LDAP Injection</a></li>
<li><a href="Benchmark_v1.1,1.2_Scorecard_for_Path_Traversal.html">Path Traversal</a></li>
<li><a href="Benchmark_v1.1,1.2_Scorecard_for_SQL_Injection.html">SQL Injection</a></li>
<li><a href="Benchmark_v1.1,1.2_Scorecard_for_Trust_Boundary_Violation.html">Trust Boundary Violation</a></li>
<li><a href="Benchmark_v1.1,1.2_Scorecard_for_Weak_Encryption_Algorithm.html">Weak Encryption Algorithm</a></li>
<li><a href="Benchmark_v1.1,1.2_Scorecard_for_Weak_Hash_Algorithm.html">Weak Hash Algorithm</a></li>
<li><a href="Benchmark_v1.1,1.2_Scorecard_for_Weak_Random_Number.html">Weak Random Number</a></li>
<li><a href="Benchmark_v1.1,1.2_Scorecard_for_XPath_Injection.html">XPath Injection</a></li>
</ul>
</li>
<li><a href="OWASP_Benchmark_Guide.html">Guide</a></li>
</ul>
</div>
</div>
</nav>
<div class="container">
<div class="starter-template">
<div>empty</div>
<div>empty</div>
<h2>Introduction</h2>
<p>The OWASP Benchmark is a test suite designed to evaluate the speed, coverage, and accuracy of automated vulnerability detection tools. Without the ability to measure these tools,
it is difficult to understand their strengths and weaknesses, and compare them to each other. The Benchmark contains thousands of test cases that are fully runnable and exploitable.</p>
<p>The chart below presents the overall results for this set of tools scored against version 1.1,1.2 of the Benchmark.
The score for each tool is the overall true positive rate (TPR) across all the test categories, minus the
overall false positive rate (FPR). To see the detailed results for any particular tool, select the tool
from the menus above. For an explanation of all the metrics calculated for each tool, see the
<a href="OWASP_Benchmark_Guide.html">Guide</a> page.</p>
<p>For more information, please visit the <a href="https://www.owasp.org/index.php/Benchmark">OWASP Benchmark Project Site</a>.
<img src="benchmark_comparison.png"/>
<p>
<p>
<h2>Summary of Results by Tool</h2>
<table class="table">
<tr><th>Tool</th><th>Benchmark Version</th><th>TPR*</th><th>FPR*</th><th>Score*</th></tr>
<tr ><td>FBwFindSecBugs v1.4.0</td><td>1.2</td><td>47.64%</td><td>35.99%</td><td>11.65%</td></tr>
<tr ><td>FBwFindSecBugs v1.4.3</td><td>1.2</td><td>77.60%</td><td>45.21%</td><td>32.39%</td></tr>
<tr ><td>FBwFindSecBugs v1.4.4</td><td>1.2</td><td>78.77%</td><td>44.64%</td><td>34.13%</td></tr>
<tr ><td>FBwFindSecBugs v1.4.5</td><td>1.2</td><td>95.20%</td><td>57.74%</td><td>37.46%</td></tr>
<tr ><td>FBwFindSecBugs v1.4.6</td><td>1.2</td><td>96.84%</td><td>57.74%</td><td>39.10%</td></tr>
<tr class="danger"><td>FindBugs v3.0.1</td><td>1.2</td><td>5.12%</td><td>5.19%</td><td>-0.07%</td></tr>
<tr ><td>OWASP ZAP vD-2015-08-24</td><td>1.2</td><td>18.03%</td><td>0.04%</td><td>17.99%</td></tr>
<tr ><td>OWASP ZAP vD-2016-09-05</td><td>1.2</td><td>19.95%</td><td>0.12%</td><td>19.84%</td></tr>
<tr class="danger"><td>PMD v5.2.3</td><td>1.2</td><td>0.00%</td><td>0.00%</td><td>0.00%</td></tr>
<tr ><td>SAST-01</td><td>1.1</td><td>28.96%</td><td>12.22%</td><td>16.74%</td></tr>
<tr ><td>SAST-02</td><td>1.1</td><td>56.13%</td><td>25.53%</td><td>30.60%</td></tr>
<tr ><td>SAST-03</td><td>1.1</td><td>46.33%</td><td>21.44%</td><td>24.89%</td></tr>
<tr ><td>SAST-04</td><td>1.1</td><td>61.45%</td><td>28.81%</td><td>32.64%</td></tr>
<tr ><td>SAST-05</td><td>1.1</td><td>47.74%</td><td>29.03%</td><td>18.71%</td></tr>
<tr ><td>SAST-06</td><td>1.1</td><td>85.02%</td><td>52.09%</td><td>32.93%</td></tr>
<tr ><td>SonarQube Java Plugin v3.14</td><td>1.2</td><td>50.36%</td><td>17.02%</td><td>33.34%</td></tr>
</tr>
</table><p>*-Please refer to each tool's scorecard for the data used to calculate these values.
<p>
<p>
<h2>Key</h2>
<table class="table">
<tr>
<th>True Positive (TP)</th>
<td>Tests with real vulnerabilities that were correctly reported as vulnerable by the tool.</td>
</tr>
<tr>
<th>False Negative (FN)</th>
<td>Tests with real vulnerabilities that were not correctly reported as vulnerable by the tool.</td>
</tr>
<tr>
<th>True Negative (TN)</th>
<td>Tests with fake vulnerabilities that were correctly not reported as vulnerable by the tool.</td>
</tr>
<tr>
<th>False Positive (FP)</th>
<td>Tests with fake vulnerabilities that were incorrectly reported as vulnerable by the tool.</td>
</tr>
<tr>
<th>True Positive Rate (TPR) = TP / ( TP + FN )</th>
<td>The rate at which the tool correctly reports real vulnerabilities. Also referred to as Recall, as defined at
<a href="https://en.wikipedia.org/wiki/Precision_and_recall">Wikipedia</a>.</td>
</tr>
<tr>
<th>False Positive Rate (FPR) = FP / ( FP + TN )</th>
<td>The rate at which the tool incorrectly reports fake vulnerabilities as real.</td>
</tr>
<tr>
<th>Score = TPR - FPR</th>
<td>Normalized distance from the random guess line.</td>
</tr>
</table>
</div>
</div>
<!-- Bootstrap core JavaScript
================================================== -->
<!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
<!-- Include all compiled plugins (below), or include individual files as needed -->
<script src="content/js/bootstrap.min.js"></script>
</body>
</html>