There are a lot of different attitudes and expectations clients have for penetration tests. Many people think of a penetration test as quality assurance, expecting the result of the test to be a simple yes or no answer to the question “is this software bug-free?” (or at least security bug-free).

There is a problem with this thought: The penetration tester cannot possibly determine whether any given software is completely free of bugs in a typical penetration test. That would at least require the source code which we (as penetration testers) are rarely given in the typical penetration test. Sometimes further adding to this obscurity are vague error messages or a lack of experience on the penetration tester’s part with the application or framework in question.

These conditions often lead penetration testers to report something in between “pass” and “fail”. For instance, I’ll report that while I was not able to exploit a particular piece of software, it has some suspicious behavior which I recommend changing. I will make such recommendations if I suspect that someone with more time or better knowledge of the application or its surrounding software environment might be able to exploit it.

I think that more bugs can be found and fixed if penetration tests are thought of as education rather than a simple “pass/fail” result. For example, if a vulnerability is found, think about how and why it was exploited. If a recommendation is made about something that we could not exploit, think about what else the pen tester might need to actually exploit the vulnerability and who else might have that. One of the things a penetration test gets you is not only the actual test and report, but the ability to ask questions of someone with a different perspective. Thinking about a penetration test this way can lead to far more improvements than simply the correction of the actual bugs that the test finds.