Top 10 Tips For Hiring Web Application Pen Testers

My 2 cents...

• Match Individuals to your Application - If you are doing black-box testing, then knowing the actual language might not be necessary. It would definitely help because each language has its weak points, but this alone wont/shouldn't stop a good hacker. In most of the cases the webapp problems lie in logic-implementation or coder's understanding of new technology or concepts. By the time, you finish testing 20-30 good sized web applications - you kind of know where to start hitting.
• Road Test - I would also suggest, that when doing road test - take away the man's tool. Lets see what he can do with his bare hands :). This way you can actually see his methodology as mentioned in the step 6. Many people that i have met talk a lot about SQL injection and how/what it can do, but when you ask them to do it. ... Silence is golden....
• Be Prepared to pay for quality - This is currently anywhere from $200 to $300 USD per hour.. WoW ! are you guys hiring :)
• Methodology - Every hacker has its own methodology, you cannot follow something from a book or what your predecessors wrote. For this same reason, an year back when I started hacking web applications I jotted down the WebApp Cheatsheet, its a single page pdf to keep track of my methodology.
• ToolBox - I hate web application tools, just hate them. Why ? Because those tools are so dumb that they will make you dumb too. Most of the webapp scanning tools, don't perform well across all applications. Even when you use it, you have to make sure that there are no false positives. You have to get personal with the web-application to understand what its doing and what it shouldn't. Just like any piece of art, after having a complete look at the web-application you can guess what the coder (artist) was thinking while coding (painting), but webapp tools cannot. Use the basic tools like Paros, or any of other advanced web proxy tool.
• Fixed Fee = Fixed Time - There is nothing more painful even for the pentester if he finds a potential flaw on the last day of testing while wrapping everything. Its like you find the cheat for a game, but cannot use it. Luckily, my manager is not someone in suit-tie and CISSP on resume; therefore he allows me to extend my stay and cause havoc :)

Apart from these, I would suggest rotating pentesters. Fresh eyes can sometime bring out flaws that were missed by previous pentester. Some pentesters are good with one technology and not others, so rotating them can help.