Do you build reports for your penetration tests? Want to make them more useful and more readable? This article is for you. Various tips are spelled out that have proven effective for the author over the years.

Recently I’ve had the, ahem, pleasure of reading and reviewing a number of penetration test reports from various internal and third-party resources. For fear of getting someone into a good deal of trouble my source shall rename nameless. My first thought was “wow, this is a wild and varied industry we are in.” My second thought was, “how is this stuff useful to someone trying to fix the issues found?” In the majority of the reports it wasn’t obvious what the problem evenwas let alone what one would do to fix it. This is a pretty well known problem in the infosec industry as a whole, we tend to “overtalk” the
problem. That is, and I know a number of folks who do this regularly, we simply cannot distill a problem or issue down into an actionable item. At least not in writing. We like to explain exactly how clever we are and exactly how we found an issue but how to fix it is usually the last thing on our minds. This of course isn’t in all cases but out of these reports I got to look over it was the case in most instances. I am going to attempt to outline how to build a better penetration test report and I’m going to use a category then list format for easy digestion. If you don’t like lists because they are overused please
withhold your anger, they are overused for a reason, people like them.

Executive Summaries
This is often a VERY overlooked part of any report. Folks think you just include some charts and graphs and off you go. This is not or should not be further from the case. Remember whether you’re a third-party tester or an internal tester, the executives pay your bills, be nice to them! The reports I reviewed basically had an executive summary that said either “Your stuff is broken beyond repair” or “You need to pay us to come fix it for you.” Neither are really good messages to send to non-technical executives. Below are some tips for driving the point home without being insulting and without over or understating the problems.

1) Charts and graphs can be useful and powerful tools as long as they’re properly used. The ones I like to see break down the issues by severity and then by severity and by category. This will tell the executive exactly where the discovered issues are and how bad they are. This provides a snapshot they can use to allocate resources appropriately. See useful and powerful!

2) Provide a set of scenarios for each issue or category of issues ranging from best case to worst case as it relates to the business you’re (or your client is) in. This allows them a better understanding of what the vulnerability could lead to. Save the drama and stick to something that might have a reasonable shot of happening. Someone is probably not going to leverage an XSS vulnerability on a toy maker’s website to set off a nuclear explosion that brings about the zombie apocalypse. I mean, it COULD happen but maybe save that one for your fiction writing class. The more realistic the better here.

3) Provide a non-technical explanation of the vulnerability. I know this one is tough but stay with me. Executives, for the most part, aren’t technical people and that’s no reason to be insulting to them. They have different skills, like balancing checkbooks and managing people, so, in English (or the language of your land), explain the problem, clearly. Some German physicist once said “You do not really understand something unless you can explain it to your grandmother.” He was a pretty smart guy, we should carry that quote with us when building our executive summaries or in reports in general.

4) Try to include some criticality rankings as it relates to their industry. For instance, if the industry best-practice is to rate an XSS hole as a “high” criticality say that, then explain why you may have marked it as “medium” if that is the case. There are mitigating circumstances in a lot of cases that may cause you to do this. It is okay to not mark everything as high especially when the vulnerability doesn’t warrant it. Another example from my report reviews, every case of being able to find version information about a service was marked as a “high” criticality issue. It made me very sad.

5) Ultimately your executive summary should tell what is wrong, in what places and how bad. The keep it simple principle applies here as it does to most things.

Vulnerability Reporting
This section of a report is often were the rubber meets the road so to speak. This is where you get to report your findings in all their deep, technical glory but wait! You might want to consider your audience first. Typically you’re not presenting your results to security people. Typically they’re being presented to a more-technical-than-our-hypothetical-manager-above manager. That manager will then break apart your report for distribution to the appropriate parties. Web developers, database administrators, network administrators, etc will be getting their own chunked up version of your report. This requires some forethought on your part for how you design your vulnerability reporting. Here’s a list of steps I think are useful for vulnerability reporting, you should consider including these along with any specific requirements your client or company might have.

1) Assign a unique vulnerability ID, well unique to you, your client or company. This helps with tracking the issue throughout the remediation process. It also helps so the tech folks to communicate the status of a particular issue up the chain. Something like department code-vulntype-number would work for this. So, 001-web-010 would identify a vulnerability in department code 001, it is classified as a web attack and then an identifying number.

2) Remember to try to speak your end audience’s language. Even the best web developer may not be aware of CSRF by that name so make sure you take the time to explain the issue. If she were to try to explain advanced web usability standards to you, you probably would drift off too. I know we think what we do is the most important thing in the whole galaxy (and it is) but not everyone shares that opinion. This is especially true if they have experienced a rather unenlightened security person before that tried to tell them not only was their code “insecure” but it lacked comments and readability. Yes, that was in
one of the reports I had a chance to peruse. Try not to be THAT security person.

3) Make each vulnerability stand on its own remembering that your report is probably going to be distributed in chunks. If one vulnerability depends or is related to another, you should note that in a “relates to” section. This helps the manager to know to distribute that issue with its related issue whether it belongs to that specific group or not.

4) Include an evidence section that is fairly detailed as to what actually happened! Screenshots, videos, packet captures and details, these are all welcome here. This is where you can be more technical to cover what the problem actually is. Again, it is not enough to just insert “Suzie waz h3r3” into a database. Be as descriptive aspossible. What page was the problem located at? What fields were vulnerable? What did you do to find the problem? How can it be reproduced? These are essential questions to answer in a vulnerability report.

5) Include some industry references. It’s awesome to include a CVE or OSVDB for some packaged piece of commercial software where you discover the application that hasn’t been patched. You should go further. If you find a SQL Injection in a custom piece of code, point to some industry standard resources on the issue. This helps educate the folks involved with the applications and will hopefully result in more secure code being written. That is our end goal, right?

Tool Data and Use
Everyone uses scanners, if you claim otherwise, either you are not paying attention or not being honest with yourself. You may not rely 100% on the scan data but you use it. The question is, should you just, by default, provide the scan data with your report? This one causes a pretty heated argument in my head. On one hand I think you should just include the full tool data with your report, you used a scanner, some web application discovery tool, etc, you should provide the raw data. On the other hand I balance this with the fact that the number one complaint I hear about penetration testing reports is “they dropped 60 pounds of paper on my desk and just left it there, what am I supposed to do with that?” Here is what I think should be done with tool data:

1) Don’t provide the raw data by default, let the necessary parties know it is available upon request but generally providing it straight away leads to comments like the one above and a lot of confusion. You are paid to interpret the results, not the end reader of your report. The tools you use should just be that, tools that provide to-be-analyzed data, not to be used for end reports.

2) As I said I am conflicted about this so if you are too then by all means, provide some tool data with your vulnerability reports. You just want to avoid copying and pasting the whole scan log into your report for instance. Perhaps note in a “Identified by” section that details how you found the particular issue. This is a great place to include some tool data as it relates specifically to that vulnerability. Maybe even include a “here’s how to test for this vulnerability yourself with tool x” bit in there. Again, it’s about educating your audience.

3) Don’t be shy about saying what tool you used to find what. This show proficiency in your craft, unless of course, you only used “Web Killer 2020” as the only tool. That just shows a lack of attention to detail. Yet another example from my report review, EVERY vulnerability was “found with AppScan”. Now I’m not denigrating AppScan at all (that’s another article) but it should never, ever be the only tool you use in your work. No tool should be. You should develop a varied ecosystem of tools that you use in particular circumstances and don’t be afraid to add new ones, after an appropriate test period of course.

4) Open Source tools. Do you use them? If not, you should add some to your arsenal. My philosophy is and always has been that an attacker, kiddie or otherwise, is NOT going to go out and spend $50,000 on a tool to exploit some hole, they will probably pirate it. Seriously though, they will go get some open source tool or write their own. Either way, the giant price tag tools will not help you in all these circumstances. One word of caution, don’t just blindly include open source tools data into your reports (you really should never blindly do anything of course), the reason being the authors sometimes like to use colorful language you may not want your end audience to see.

5) Generally speaking though, keep your report to the analysis of the results of all these various tools, learn to manually verify the results. This analysis is the value you provide. It’s one thing to penetrate a network but if you can’t explain how you did it then did you really do anything of worth? If you aren’t providing value then you are just a paid button-pusher, that’s not really what you want to be is it?

Reporting from Experience
Your client or employer didn’t hire you for your looks. You were hired, presumably, because you have some experience in this field so why are you not using it? Reporting from experience is quite powerful. The term “I’ve seen this before” is the ultimate confidence builder, it is what manager live for. Being able to relate an issue or vulnerability back to past experience is a powerful way of saying “I’ve seen this before and here’s what we should do.” You should be doing that.

1) If you’re a third party tester, tell your client if you’ve seen the exact issue before and explain what your previous client did about it. This is a great reporting technique and relays your experience with confidence. It allows you to provide a track record of a particular remediation technique for your customers. Possibly you could even build a “trust factor” into your remediation solutions. This could be an article in and of itself but basically you’re putting a score on your confidence a given solution will work for a particular issue. Customers and other departments will like that.

2) If you are an internal resource without much exposure to other networks or apps or don’t have a lot of experience then do some research about the vulnerability. “Here’s how XYZ fixed it according to Google”. Okay maybe it’s not THAT simple but a little research and effort can go a really long way to building an effective report.

3) This applies to everyone. Once you have tested and manually verified the vulnerabilities then do some research on the issues. Did a major breach occur somewhere else because of this vulnerability? That’s something you should be reporting. “XYZ leaked 10,000 credit card numbers due to a SQL Injection closely related to vulnerability XXX.” That’s quite a bit more powerful than “You have some SQL Injection and I can insert my name in your database.” The more convincing you make your argument the less likely a client or other department is to counter with the old “well it’s not that big a deal,
we’ll fix it later.” Sadly that is a prevailing attitude and can be correct with this approach.

Remediation
I have noticed both from the reports I’ve reviewed and just being in the industry for a while that a vast majority of penetration testers shy away from even mentioning remediation techniques. This is a shame to me as I think testers can bring a very unique perspective to the remediation process. I’m not completely sure on the reasoning behind the shyness but I can guess. At any rate I think offering some remediation advice in reports is invaluable when looking at these things from an operations perspective. A question I get quite a bit from my customers is “okay I’ve got this report from vendor X, how do
I fix this stuff?” No one is an expert in everything so while you can provide some advice be honest about where your expertise lay. Go gather some external resources to help you out where your skills might be weak, this helps build the experience I discussed earlier. On with the list.

1) Each vulnerability should have a remediation section. It should explain some possible fixes to the issues along with well known industry best practices for fixing the issue.

2) Stop it with simply listing links to remediation resource. While I think a resource and reference section is useful and valuable it cannot be the only remediation advice you supply. You are being paid for your advice, don’t chicken out.

3) If you have previous experience with a particular issue the note that in your report. Offer some sort of special assistance with that issue, don’t just be a paper generator.

4) Some would argue that offering remediation advice limits your objectivity. I would argue it helps you know your client better, builds your testing skills and keeps your other skills sharp. It’s really a win all the way around.

5) I touched on this one before but it is worth repeating I think. Learn to speak the language of your audience. It is worth the time investment to communicate to your end audience in a way they’re accustomed to. Don’t make them learn a new language to read your report. That approach leads to reports sitting on a shelf with no action taken.

Re-test
Once vulnerabilities are fixed, they should be re-tested to be sure the fixes work. We typically include it as part of an up-front engagement but however you do it make sure it gets done. This does a lot of things to reinforce your initial testing but primarily it provides a metric to show how serious a company or department is to fixing vulnerabilities. It can even be used to show how a company or department measures up to another organization or department as it relates to remediation. Not to say that people respond to competition but people respond to competition. It is just human nature.

1) Simply running through your evidence sections in the initial vulnerability might suffice for a re-test. However, I recommend a more thorough re-testing, as an applied patch or a fix in one script might have introduced some new exploitable hole. You will want to try to find those on a good re-test.

2) Once you have completed the re-test then you should update your report with the results of that testing. I like the “change log” format that shows when something was discovered and when something was resolved. If a given issue was not resolved then you can note that in the same format.

3) Don’t be afraid to update your remediation advice on a re-test. No one is perfect, it’s okay. If you find that a particular technique doesn’t work for resolving an issue, note that and find a technique that does. This can be included in the “trust factor” mentioned above allowing you to scale your confidence in a given fix more easily.

4) What if you a find a new issue during the re-test? Then you are doing your job. Make a new vulnerability report and let your client know, then begin the same process all over again. This just makes you a better tester as you will learn to be more and more thorough as you go on.

Report Tips
Before I wrap up I wanted to just review some general reporting tips.

1) Provide a “cheat sheet” so that your end audience can quickly see a general summary of what is in the report. I am a huge fanboy of cheat sheets, I’m fairly convinced that if properly used they can do anything.

2) Remember that these reports will probably be very chunked up when delivered to the various remediation teams. Provide the report in a format that is easy to break apart and can stand on its own.

3) Provide a guide to using your report. Help files are wonderful, a tutorial on how to read the report would be great. This helps end audiences know where they’re supposed to look for things, etc. Now needing this violates the keep it simple principle but it removes any confusion. Remember what is simple to you might not be to someone else.

4) Make sure you provide a method for your end audience to comment on your report or ask questions. A wrap up meeting or conference call is always the best way to accomplish this. You should always be able to defend and explain your work.

5) Do not just leave a stack of paper on your customer’s desk. Be available to help, explain, defend, etc. You should not be a hit and run penetration tester.

As I re-read this article I realize it’s probably very condescending sounding to experienced testers, if that is the case, then this article is not for you. Based on this review of some big name firms’ testing reports though a lot of folks need a refresher course in report writing. These tips would apply to whatever sort of testing you are doing. From physical penetration to social engineering to remotely exploiting a server, all of these things require proper and useful reporting. Your hard work and your end customers deserve nothing less.

Read more in PenTest Magazine – May 2011