Penetration testing vs vulnerability scanning: what each proves and when you need both

What is the difference between a vulnerability scan and a penetration test?

NIST SP 800-115 draws the line in one sentence: scanners "check only for the possible existence of a vulnerability", while the attack phase of a penetration test "exploits the vulnerability to confirm its existence". Most tests also look for combinations of vulnerabilities that give more access than any single one.

The rows draw on PCI SSC's penetration testing guidance (version 1.1, 2017), NIST SP 800-115 and NIST SP 800-53. The guidance adds that some entities may have to test more often.

Is a vulnerability assessment the same as a scan?

No. The NIST glossary defines a vulnerability assessment as a systematic examination of an information system or product to determine the adequacy of security measures and identify security deficiencies, among other goals. NIST SP 800-115 lists port and service identification, vulnerability scanning, security configuration review and password cracking as activities a vulnerability assessment plan may authorize. The NCSC adds that a penetration test is a method for gaining assurance in your vulnerability assessment and management processes, not a primary way to find vulnerabilities.

How does vulnerability scanning work?

A scanner identifies the operating systems and major software on each host, submits attack patterns, compares the responses with the signatures of known vulnerabilities and reports any matches. That can find outdated software versions, missing patches and misconfigurations. It relies on a repository of signatures, so its database should be updated before each run. It sees surface vulnerabilities, weaknesses that exist in isolation, so it cannot detect ones revealed only by combinations of attack patterns. NIST says scanners can have high false positive and false negative rates.

Scans come in several kinds.

The OWASP testing guide suggests building DAST, static application security testing (SAST) and software composition analysis (SCA, or dependency tracking) tools into CI/CD workflows, or running them on a regular schedule, to catch "low hanging fruit" weaknesses. Our secure coding checklist shows when each check runs, the website security checklist lists free scans for your own site, and the SBOM guide covers the component inventory behind dependency checks.

What does a penetration test do that a scan cannot?

NIST SP 800-115 splits a test into four phases: planning, discovery, attack and reporting. Discovery includes the scanning and vulnerability analysis that many scanners automate, and manual analysis can identify new or obscure vulnerabilities that scanners may miss, though it is much slower. The attack phase then tries to exploit what was found, and a feedback loop returns the tester to discovery, because penetration testing is an iterative process that uses minimal access to gain greater access.

An example, invented for illustration. A scan of a small web business reports three findings: a login form that reveals whether a username exists, an old admin panel on a high port, and a database that accepts connections from the whole internal network. The scanner rates each low or medium. A tester chains them: confirm a staff username, guess a weak password on the admin panel, run commands on the web server through the panel, then read the customer database. NIST warns that a scanner may assign "a low risk to each vulnerability", leaving the assessor "falsely confident", and names penetration testing as the more reliable way to find the risk of vulnerabilities in aggregate.

OWASP's testing guide says automated tools do a poor job of testing for vulnerabilities in part because they do not think creatively, and that creative thinking is needed case by case because most web applications are built in a unique way. PCI SSC's guidance gives an example of a finding a report should include even though it was not exploited: credentials obtained by manipulating a web application error message, which an ASV scan had not flagged because of a low CVSS base score.

A test has limits too. NIST says penetration testing is labour-intensive, requires great expertise and may damage systems or render them inoperable, a risk experienced testers can reduce but never fully eliminate. The NCSC says a test can only validate that your systems are not vulnerable to known issues on the day of the test, and that checking whether defined security controls work is not a valuable use of penetration testing resources.

Which type of penetration test do you need?

Pick by what you expose and what you cannot afford to lose.

Each type can run with different amounts of knowledge. PCI SSC's guidance defines black-box testing as testing without prior knowledge of the internal structure, grey-box as partial knowledge and white-box as knowledge of it, and says PCI DSS penetration tests are typically white-box or grey-box, which give more accurate results than a pure black-box test. The NCSC calls the two ends closed box and open box, and says closed-box testing models unknown attackers more accurately but can leave vulnerabilities undiscovered in the time available.

A red team exercise goes further. NIST SP 800-53 says red team exercises extend the objectives of penetration testing by examining an organization's security and privacy posture and its capability to implement effective cyber defences, and that they may include social engineering. A penetration test is conducted on systems or individual system components.

How do you rank what scans and tests find?

Scanners often use their own proprietary methods for defining risk levels, so findings are hard to compare between tools. NIST's example: a scanner may call an FTP server a moderate risk because it sends passwords in cleartext, when the real risk is lower if the server is an anonymous public one that uses no passwords. Assessors should set the risk level for each vulnerability and not simply accept the scanner's.

Three public inputs help you set that level:

• CVSS v4.0 rates severity, and its qualitative scale runs from Low (0.1 to 3.9) to Critical (9.0 to 10.0). FIRST's FAQ says Base scores "are not risk" and should not be used alone for patch prioritization.
• EPSS is a calibrated probability that exploitation activity will be observed across EPSS data partners in the next 30 days. FIRST says it is not a severity score and not a complete risk score.
• CISA's KEV catalog is, in CISA's words, the authoritative source of vulnerabilities that have been exploited in the wild. CISA says organizations should use it as an input to their prioritization framework.

Our guide to which vulnerabilities to patch first compares these inputs and ends with a patch policy a small IT team can adopt. Testers rank findings too. The NCSC says they may lack your business context and rate an issue lower or higher than you would, and that "risk assessment and decisions on the application of fixes are your responsibility".

What do PCI DSS and NIST ask for, and how often should you test?

The rows below are from PCI DSS v4.0, as printed in its SAQ D for Merchants (April 2022). The Council's note on v4.0.1 calls it a limited revision with no new or deleted requirements and says that once v4.0 was retired on 31 December 2024, v4.0.1 became the only active version, so confirm the current text before you quote it to an assessor.

PCI DSS leaves "significant change" to the entity. PCI SSC's guidance says it depends on your risk-assessment process, and that a change which could affect network security or allow access to cardholder data may count. The same guidance says simply running an automated tool does not satisfy the penetration testing requirement, because tools cannot interpret vulnerabilities or exposed services to assess the true risk.

Outside PCI DSS, NIST SP 800-115 says that, given its high cost and potential impact, annual penetration testing may be sufficient, and that regularly scheduled scanning interspersed with periodic penetration testing can help prevent many types of attacks and reduce the impact of successful ones.

Scan between tests, because the NCSC notes it is not uncommon for a year or more to pass between tests, so vulnerabilities could exist for long periods without you knowing. The program below is an example for a small or mid-sized company, not a standard. Where PCI DSS applies, it sets the floors.

What should you agree in writing before a test starts?

NIST notes that issuing attacks to compromise systems is usually prohibited by an organization's own policies, and that the rules of engagement (ROE) provide the authorization for assessors to do it. It recommends involving legal advisors for intrusive tests such as penetration testing.

Settle these points in the scope document and the ROE, drawing on the NCSC's scoping guidance and PCI SSC's list of ROE considerations:

1. Scope and exclusions. List the hosts, applications, APIs and accounts in scope, and any system not authorized for testing, which NIST calls the "exclude list". NIST says assessors should verify that public IP addresses are under your organization's purview before testing begins.
2. Test window. State the hours, and name legacy systems with known issues with automated scanning. NIST's sample ROE notes it may be prudent to test an operational site in the evening rather than at peak business periods.
3. Contacts and stop rules. Name a technical contact who is available at short notice. PCI SSC's example for a previous or active compromise is to activate incident response and stop the test until it is resolved.
4. Data handling. Decide whether the tester must disclose passwords or sensitive data they reach, and whether they keep any. PCI SSC says only a proof-of-concept test should be performed, and any cardholder data obtained must be secured in accordance with PCI DSS.
5. Security controls. If an intrusion prevention system or web application firewall protects the application, decide whether it may block test traffic. PCI SSC advises avoiding interference, because the test should evaluate the services' susceptibility to exploitation, not the protection system.
6. Success criteria and retest. Define when the test is complete, for example compromise of the domain used by privileged users. Without that, PCI SSC says, the tester may exceed your boundaries and expectations. Include a retest of the fixes.

Cloud providers add rules of their own. AWS lets customers test their AWS infrastructure on a list of permitted services, such as EC2 instances, RDS and Lambda, without prior approval, but not AWS's own infrastructure or the AWS services themselves. It prohibits, among other things, denial of service and port, protocol and request flooding; DDoS simulation falls under a separate policy. Microsoft's rules allow fuzzing, port scanning and vulnerability assessments on your own Azure virtual machines, prohibit denial-of-service testing and network-intensive fuzzing, and apply equally to the resource owner and their authorized agents.

What does a good penetration test report contain?

PCI SSC's guidance is blunt: "Merely reporting lists of vulnerabilities is not helpful" and does not meet the intent of the penetration test.

Use PCI SSC's suggested outline and the NCSC's list as a checklist. A report worth paying for has:

• A statement of scope, including the systems tested and why.
• Methodology and limitations, such as the approach followed, test hours and restrictions.
• A testing narrative: how testing progressed, and any issues met along the way.
• Findings with a risk ranking, affected targets, references such as CVE or CWE, and how each was exploited.
• Findings that were not exploited but still pose a risk.
• How each severity was derived, with reasons for any change to a standard score.
• An opinion on your own vulnerability assessment, with advice to improve it.
• Cleanup steps, and a retest report listing the original findings and the retest results.

A report that mainly lists scanner findings with the scanner's own severity labels is a scan export, not a test report. Ask for a debrief, where the testers walk through their findings and you can request clarification, and for a retest.

What drives the cost, and what should you ask a provider?

This guide quotes no prices, because none is honest without your scope. The NCSC calls penetration testing a powerful but expensive operation and says the scoping document should state the time frame and effort, usually in resource days. Three things the sources link to that effort:

• Size of scope. The technical boundaries and the types of test expected. The NCSC says changing scope is likely to alter time frames and cost.
• How much the tester is told. PCI SSC says black-box testing may need more time, money and resources.
• Constraints and reporting. The NCSC says the scope should record out-of-hours testing, systems that need special handling and reporting requirements such as CVSS scores, plus any time constraints a provider must consider when allocating resources.

Put these questions to every provider, against the same written scope:

1. Who will test, and what have they tested before? The NCSC says a test's quality is closely linked to the abilities of the testers. PCI SSC suggests asking about years of experience (the tester's and the firm's), work for organizations of similar size and scope, experience with your technologies, and references.
2. What do their certifications show? PCI SSC lists OSCP, CEH, GIAC, CREST and CHECK as examples, says certifications may indicate skill, does not endorse them, and says qualifications cannot be met by certifications alone, so ask about actual engagements.
3. Are they independent of your systems? PCI SSC says the tester must be organizationally separate from the management of the target systems.
4. Which method will they follow? PCI DSS asks for industry-accepted approaches. PCI SSC's guidance names examples, among them OSSTMM, NIST SP 800-115, the OWASP Testing Guide and the Penetration Testing Execution Standard.
5. Is a retest included, and what does it cover?

A plan you can start this month

1. List what you expose. Domains, public IP ranges, web applications, APIs, cloud accounts, anything a third party hosts for you, and any reverse proxy in front of them.
2. Get the requirement in writing. Ask the customer, insurer or auditor which document asks for the test, how often, who may perform it and what the report must show.
3. Scan and fix by risk. Run internal (authenticated) and external scans, rank findings as above and rescan. Routine updates, covered in our website maintenance guide, address a common root cause NIST names: insufficient patch management.
4. Write one scope sheet from the checklist above, and collect two or three quotes against it.
5. Book the test after your next big change, and book the retest with it.
6. File the evidence. Keep reports, fix records and retest results for at least 12 months, the period PCI DSS 11.4.1 sets for penetration testing results and remediation activities results.

Computese offers security testing, which covers vulnerability scanning and penetration testing.