A surprising number of “penetration test” reports are a scanner export with a cover page. They are not useless — a scan does find real problems — but they answer a different question from the one a customer, auditor or insurer is asking when they request a pen test.
What a scanner actually does
An automated scanner sends a large set of known-bad inputs at every page and parameter it can find, and compares the responses against signatures. It is excellent at finding outdated software with published vulnerabilities, missing security headers, obvious injection points and misconfigured TLS. It is fast and cheap, and you should run one regularly.
What it cannot do is understand intent. A scanner does not know that invoice 1042 belongs to a different customer from invoice 1043. It does not know that the approval step is supposed to happen before the payout. It sees a 200 response and moves on.
Where the serious findings are
Broken access control has been at the top of the OWASP Top 10 since 2021, and it is still there in the 2025 edition. In the API Security Top 10, broken object level authorisation is number one. These are, overwhelmingly, the findings that expose whole databases — and they are almost invisible to automated tools, because the “attack” is a perfectly valid request made by the wrong person.
“A scanner sees a 200 response and moves on. A tester asks whose data that was.”
The same is true of business-logic flaws: stacking discounts, racing a withdrawal limit, skipping a verification step by calling the next endpoint directly. None of these involve malformed input. They involve using the application in an order nobody intended.
Related engagement One tenant could read every other tenant’s invoices →How to tell which one you are buying
Ask three questions. How many days of manual testing are included, and by whom? Will the tester have accounts for each role, and test access between them? Will each finding include the specific request that reproduces it? If the answers are vague, you are probably buying a scan.
Also ask what methodology is followed. The OWASP Web Security Testing Guide and NIST SP 800-115 are the common references; ASVS 5.0 gives a checklist of what was verified. A good report tells you what was covered, not just what was found.
When a scan is enough
Continuous automated scanning is the right tool for keeping known vulnerabilities out of your infrastructure between tests, and for meeting requirements that specifically ask for scanning, such as PCI DSS external scans. It is the wrong tool for proving that one customer cannot see another’s data.