WSVDBench

Benchmarking Vulnerability Detection Tools for Web Services

This project contains the source code of the services that serve as workload of a benchmark for tools able to detect SQL injection vulnerabilities in web services. The benchmark was introduced in [1], the papers explains the results and how they were obtained. The benchmark was later reused and extended in [2].

The ground truth of this workload is presented in the detailed results file. In practice, the column Review represents what is considered as the true vulnerabilities (TV).

1. Nuno Antunes and Marco Vieira, "Benchmarking Vulnerability Detection Tools for Web Services", IEEE 8th International Conference on Web Services (ICWS 2010), Miami, Florida, USA: IEEE Computer Society, ISBN: 978-1-4244-8146-0, July 5-10, 2010, DOI:10.1109/ICWS.2010.76.
2. Nuno Antunes and Marco Vieira, "Assessing and Comparing Vulnerability Detection Tools for Web Services: Benchmarking Approach and Examples", IEEE Transactions on Services Computing, vol. 8, no. 2, pp. 269–283, 2015, DOI:10.1109/TSC.2014.2310221.

How to use it

1. Setup your database. A script to fill a PostgreSQL database can be found in database dump. It is a minimal version, just for demo.
2. Configure the class Database.java according to the database and configuration used in step 1.
3. You can (must) replace the class Logging.java for your favorite logging framework.
4. Install the missing maven dependencies, located inside install. Use the install.sh to help you.
5. Build the .war package, using mvn package or mvn install
6. Deploy the services in your app server.
7. Run your vulnerability detection tool against the sources or services, and compare the results with the ground truth.

Sources Characterization

The benchmark consists of 21 web services, adapted from three standard benchmarks developed by the Transactions processing Performance Council, namely: TPC-App, TPC-C, and TPC-W. Both vulnerable and non-vulnerable versions of the services are included. These services include a total of 80 operations (# operations) with 158 known SQL Injection vulnerabilities (# vulnerabilities). The table below presents, for each service, the number of known vulnerabilities, the number of lines of code per operation (LoC/Op), and the Average Cyclomatic Complexity of the code (calculated using SourceMonitor). As we can see, the services are quite diverse in terms of complexity, size and even in number of vulnerabilities. This diversity is important to evaluate tools that have different effectiveness in different scenarios..

Source Benchmark	Service Name	# operations	# vulnerabilities	LoC/Op	Avg. C. Complexity
TPC-App	ProductDetail	2	0	242	5
	NewProducts	2	1	206	4.5
	NewCustomer	6	35	1230	5.6
	ChangePaymentMethod	2	2	198	5
TPC-C	Delivery	9	10	2043	21
	NewOrder	7	15	2317	33
	OrderStatus	7	18	1463	13
	Payment	13	34	4251	25
	StockLevel	4	6	320	4
TPC-W	AdminUpdate	2	2	162	5
	CreateNewCustomer	6	27	978	3
	CreateShoppingCart	2	0	414	2.67
	DoAuthorSearch	2	1	88	3
	DoSubjectSearch	2	1	90	3
	DoTitleSearch	2	1	90	3
	GetBestSellers	2	1	124	3
	GetCustomer	2	1	92	4
	GetMostRecentOrder	2	1	258	6
	GetNewProducts	2	1	100	3
	GetPassword	2	1	80	2
	GetUsername	2	0	80	2
	Total	80	158	14826	-

Examples of works using the benchmark

• M. Elsayed and M. Zulkernine, "Offering security diagnosis as a service for cloud SaaS applications," Journal of Information Security and Applications, vol. 44, pp. 32–48, 2019.
• J. Thomé, L. K. Shar, D. Bianculli, L. Briand, “An Integrated Approach for Effective Injection Vulnerability Analysis of Web Applications through Security Slicing and Hybrid Constraint Solving”, IEEE Transactions on Software Engineering, 2018.
• J. Thomé, L. K. Shar, D. Bianculli, L. Briand, “Security Slicing for Auditing Common Injection Vulnerabilities”, Journal of Systems and Software, 2017.
• J. Thomé, L. K. Shar, D. Bianculli, L. Briand, "Search-driven String Constraint Solving for Vulnerability Detection", In Proceedings of the 39th International Conference on Software Engineering (ICSE 2017). ACM.
• A. P. Matsunaga, R. Moraes, N. Antunes, "Coverage Metrics and Detection of Injection Vulnerabilities: An Experimental Study", 12th European Dependable Computing Conference (EDCC 2016), Gothenburg, Sweden, September 5-9, 2016.
• M. A. Laverdière, B. J. Berger, E. Merloz, "Taint analysis of manual service compositions using Cross-Application Call Graphs", 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), Montreal, QC, 2015, pp. 585-589.
• A. Neto, "Security Benchmarking of Transactional Systems", PhD thesis, University of Coimbra, 2012, Coimbra, Portugal.