I designed the scanning solution for one of the ASV's and also handled the process of getting the solution approved as an ASV. All of the PCI Scanning solutions out there are supposed to be programmed to look for pretty much the same thing, and in (almost) exactly the same way. You would think that this would lead to accurate results. You would also think that this would lead to consistent scan results across ASV PCI Scan solutions. Unfortunately, I don't believe that is the case.
'PCI Scans' are not the same as a Vulnerability Assessment
Here are a few examples:
- The PCI Scan finds that port 53 tcp is open inbound to a branch of your restaurant. The PCI Scan doesn't find any known DNS service vulnerabilities so there are "no vulnerabilities detected" and the PCI Scan report shows that the host passed. The results do not raise any questions about this port being open. The tools don't understand the context of the environment. Why in the world would you need to have DNS traffic allowed inbound to a restaurant?
- The PCI Scan checked a few limited ports and concluded that the host is "dead" and therefore the scan service did not perform a vulnerability scan. Your PCI report doesn't show any vulnerabilities, so it's considered compliant. The problem is that the host is not "dead" and has some TCP ports open and could very well be running a vulnerable service on one of those ports.
- The PCI Scan found that port 443 (SSL) is open on a host and reports it as a Level 1 (Informational) finding. The PCI Scan report shows that this host is compliant. The problem is that this is one of your remote locations that uses a UTM (Unified Theat Management) device and has web-based remote management enabled from the Internet without the use of two-factor authentication. This system does not meet the requirements of the PCI DSS.
- The PCI Scan found that port 3389 (RDP) is open on a host and reports it as a Level 1 (Informational) finding. The PCI Scan report shows that this host is compliant. The problem is that this is an Internet-facing host that’s part of the card data environment (CDE) and is setup to allow connections from any source address via Terminal Services without the use of two-factor authentication. This system does not meet the requirements of the PCI DSS.
PCI Scan products can vary greatly in their findings
Earlier this year I did an ASV Scan Vendor Bake-off. I had narrowed it down to two vendors. The test scans were performed against the same hosts and under the same conditions. One report concluded that the scan FAILED and the systems were not compliant. This first report shows these vulnerabilities categorized as: 2 Critical, 1 High, 3 Medium, and 1 Low. The other report concluded that the scan PASSED and that the systems are compliant. According to the report, this scan solution had found 0 Vulnerabilities, and 16 informational comments.
What does this say about the effectiveness and value of PCI Scanning? To me it seems that PCI Scanning today is robotic and inconsistent. And since scan tools don't understand the role or context of a system being scanned, it's my feeling that they should not be trusted as the sole method of assessing vulnerabilities.