@jacobadam: I agree with you, but in general, security is a non-functional requirement that developers tend not to understand properly. So they may build the required system, build it to spec, but since they are not experts at security, we try to help them see what they have missed.
I don't know if that's what you were refering too, even if internal knowledge of the code is not required (I agree with you!), pentesting is about finding the little hole that was forget. So to me, understanding the internal mechanism speed up the process of finding vulnerabilities, thus providing better value to a client.
Finally, I have yet to see one system with perfect requirements and perfect specs, so developer "mistakes" end up happening in the end...
@Sil: Great post, as always. But even if I agree that filtering input data is sometime a very difficult task, it's usually possible to do quite well. There's always the easy cases of validating phone numbers (using regular expressions), age (must be an integer), etc. But the more difficult ones are the Text Area, which includes comments field, descriptions, etc. I use things like the Ship Validator (
http://sourceforge.net/projects/shipvalidator/) and AntiSamy (
https://www.owasp.org/index.php/Antisamy). They may not do miracle, but by adding these calls in my code, I know I get rid of most hackers.
Using these libraries combined with things like white and black lists, least privileges, use of prepared statement, etc make the application quite secure. Then of course, like you said, we limit access to the system. Like you know, security in depth is the key!
But I found that the easier the solution is to implement and support, the more people will do it. So I tend to focus on that for "regular" systems. Highly secure systems are a different ball game!
But great comments guys!