Pen-testing is a great idea; if you're a pen-tester. Other than that, I think there are serious problems with the entire concept. Sure, pen-tests give you a comfort factor and sense of "CYA" - but I suspect most of their value is in keeping auditors off your back or showing clueless managers that, "Hey! Someone really can break into our system!" That represents a lot of money, time and effort spent on appeasing the clueless. With recent standards and legislation such as PCI, SarBox, etc., we have what amounts to the "Pen Tester Permanent Employment Act."
My mother taught me everything I needed to know about computer security back in 1969, when she asked me, "If all your friends were jumping off a cliff, would you jump too?" Just because a lot of people are doing something doesn't make it smart. The problem with pen testing is that it doesn’t actually measure what people want to believe it measures. Gary McGraw likes to refer to pen testing as the "badness-o-meter" - it's a test that registers, at one end of the dial "your network sucks" and at the other end, "we don't know."
A more logical way of looking at it is simply that pen testing is trying to prove a negative - namely "there are no holes in the system." Any student of logic knows you can't prove a negative; what you can prove is a positive, "our pen tester doesn't know any way to get in." Really, what you're doing is paying a pen tester a hefty amount of money to see how good they are.
The only favorable or useful outcome of a pen test is the worst one: the pen testers walk in and demonstrate, conclusively, that system security is horrible. Then you've got a 50/50 chance you'll end up with a mandate to fix it, or nothing will happen. Because here's the sad fact: organizations with sucky security already know it, and it is not going to be improved a great deal by having an outsider show up and point that out. I suppose there's some value in the "consultant effect" that you get from a pen test - you know, the "management never listens to employees but if a consultant comes in from the outside and says the same thing, that gets their attention." None of this is about security, though - that's just plain old bad management and organizational dysfunction.
So what's the realistic alternative to pen testing? It's obvious: have a good security design, and then verify that it is in place and working correctly. If your management wants to hire outsiders because they don't trust you, or they think you're stupid, then hire outsiders to review your security design and help you improve it; then you'll actually have something to test. Isn't that a bit more scientific and logical? Your security design is your plan, then you validate your implementation against the plan, note deviations from the plan, and re-assess where necessary. The pen testing approach is to look at your network as a great big unknown, from which you try to derive clues using ping sweeps and port scans. I've got bad news for you, Dear Reader, if your network is so uncontrolled that the only way you can figure out what's on it is by scanning, then your badness-o-meter is probably pegged on "sucks" already. All you are going to find is large, uncontrolled tracts of TCP/IP swamp-land, great unknowns populated with backdoor wireless access-points, keylogger-infected laptops, and wide-open hosts.
That's the part, I think, that scares me the most about pen testing: it is not a substitute for knowing what should and shouldn't happen in your network. Pen testing is just a revisiting of the old philosophy of "penetrate and patch" - take something fundamentally flawed and keep adding more layers of duct tape and bandage to it, and eventually the flaws will be cured. It doesn't work. Fundamentally, most security problems are a result of poor design, and it's impossible to get a good design by taking a bad one and throwing away all the bad parts. Having a pen test done just helps you identify a few of the bad parts, but that's an endless effort-sink because there will always be more bad parts! That's bad news for you - and good news for your pen tester.
Marcus Ranum is the CSO of Tenable Network Security – the company that produces the Nessus Vulnerability Scanner. Lots of pen testers use Nessus because it does a better job than they can.