We are frequently asked to help advise on the appropriate scope for application penetration tests. Time and budget constraints can often raise the question of whether to use a black box, gray box, or white box penetration test.
What is a black box application pentest?
A black box penetration test is an application pentest where the tester is provided nothing more than the target location of the application. This is usually conducted against an application which requires authentication, however, credentials are not provided to the tester.
The primary objective is to determine “Can an external attacker with no prior access, obtain access to the application or data?”.
What’s performed in a black box pentest?
Although the ability of the pentester is greatly limited without credentials, there are a number of attacks and tactics used to obtain access or sensitive information.
Content Discovery
Pentesters will use a number of tools to enumerate files, directories, and functionality which may be accessible on the web server. Burp suite for example has built-in functionality to quickly discover web content.
Default Credentials
Chances are low that admin / admin password combinations exist in modern applications, but a tester must check for them. A pentester should also use any information available to test for easy to guess credentials.
Username Enumeration
Username Enumeration is a common theme in application pentesting, but especially relevant for black box pentests. As a first step to gaining access, a pentester will use a large dictionary of usernames against the login form to see if the application indicates if a username is valid.
It is recommended that sensitive applications do not reveal whether usernames are valid or not.
What is a gray box penetration test?
A gray box penetration test is performed with credentialed access. This allows the pentester to assume the role of legitimate users of all privilege levels. The tester can then perform attacks from the perspective of users to determine the impact a bad actor could have.
What’s performed in a gray box penetration test?
The word performed from a gray box perspective is substantially more comprehensive than a black box. This includes some of the following:
-
Testing of role based access controls – Identifying if low privilege users can access data or functionality belonging to higher privileged users.
-
Testing of data handling – Authenticated applications often process and store a wide array of user submitted data. A gray box pentest will analyze encoding schemes that may protect against Cross-site Scripting (XSS) and SQL Injection vulnerabilities.
-
Testing of session management – cookie based authentication, JWT tokens, and custom session management schemes all can be susceptible to unique attacks.
-
Testing of application technology – Does your application use a framework like .NET, Django, or Rails? Do you integrate with cloud components? Many common functionality like uploading documents to S3, or using wkhtmltopdf, all have their very specialized security considerations needed.
-
Testing of business logic – One of the biggest differentiators in a good vs. average application pentest. This is where a tester uses all available technical pieces of an application to undermine your business. Consider a shopping cart that allowed a quantity of ‘.5’, or a financial transaction that didn’t validate a negative deposit.
What is a white box penetration test?
A white box penetration test includes the scope of a gray box penetration test, but also allows access to source code, design documents, code comments, and just about everything a developer would typically have. This allows the most insightful perspective into the inner workings of the application and can potentially reveal the most security findings.
This assessment, however, is also the most labor intensive and time consuming. Not all applications are appropriate to assess in this fashion.
Cost Analysis of a Black Box vs Gray Box vs. White Box Pentests
Many people first consider a black box pentest as a cheap way to satisfy compliance requirements. Although this is the lowest cost method of performing a pentest, it returns significantly lower value.
At a high level it’s safe to say the following about Black vs. Gray vs White box assessments:
- Black box tests cost the least, but produce a very low number of vulnerabilities.
- Gray box tests fall in the middle range of cost, but return a good number of substantially useful vulnerabilities.
- White box tests are the highest in cost. These should produce the highest number of vulnerabilities, but the time needed to identify them is high. This leads to a high dollar per vulnerability ratio.
To illustrate this, let’s look at an average number of vulnerabilities returned from each pentest type:
Speaking very generally, black box pentests produce vastly lower vulnerabilities than gray box. White box pentests produce more, but not vastly more issue than a gray box.
Now let’s quantify the money spent. To measure the “weight” of high, medium, and low risk issues, we will give the following ‘value’ to each:
| Risk | Points |
|---|---|
| High | 1 |
| Medium | .5 |
| Low | .25 |
We end up with the following value for each:
| Risk | Black Box | Gray Box | White Box |
|---|---|---|---|
| High | 0 | 9 | 11 |
| Medium | 2 | 8 | 12 |
| Low | 3 | 10 | 15 |
| Total Points | 1.75 | 15.5 | 20.75 |
And now for the final dollar per vulnerability point:
| Black Box | Gray Box | White Box | |
|---|---|---|---|
| Cost | $4,000 | $12,000 | $30,000 |
| Points | 1.75 | 15.5 | 20.75 |
| Cost Per Vulnerability | $2,285 | $774 | $1,445 |
When we show this into our highly scientific chart, you can see the vulnerability points significantly increase after the black box category, vulnerabilities decrease when getting into the white box territory:
Which one is right for you?
Consider some of the downsides of a black box test:
- The findings returned will generally be minimal. The test is essentially only covering the login page and leaves the vast majority of the application untouched.
- The cost of engagement for a vendor for small projects can still be expensive. Pentest consultancies usually have minimum engagement costs which can lead to wasteful spending.
- A black box pentest report will not satisfy all requirements. For software/SaaS companies, potential clients will generally expect to see a gray box pentest report. This is the industry standard way to give clients assurance of your application security.
Despite the downsides, there are still times where a black box pentest is appropriate:
- Applications that are no longer maintained, and a minimum level of security assurance is needed.
- Applications which contain little to no sensitive information.
- Circumstances where a gray box pentest is simply not financially feasible. In times where budget cannot be allocated to a gray box pentest, a black box pentest is of course better than nothing. In summary, if you’re finding yourself in the Black Box vs. Gray box dilemma, chances are the gray box is the more prudent choice. A black box test is better than nothing, and a White Box test is nice if you have a large budget for critical application, but the most value will typically be found in the gray box.
Conclusion
When in doubt, it may be best to seek the advice of the pentesting company performing the assessment. Weighing internal constraints against industry norms may help decide which pentest is right for you.