I’ve seen plenty of talk these days about audits, compliance, pen-testing, and security-reviewers and how they are canned or unskilled or unknowledgable. (Auditors who understand they lack understanding are exempted as there is a place for checklist checkers.)

To me, it is pretty simple to tell when someone or some group knows what they’re talking about and are qualified to help with security. There are two traits:

1) Ability to discuss (accurately!) a scan or pen-test report. Don’t just hand the report back and/or quote it verbatim in a meeting. Be able to casually talk about the issues, and if necessary, be able to reword them for understanding by managers and techies alike. Being a fellow geek, I can usually quickly figure out when someone’s knowledge is limited or outright bullshit.

2) Ability to discuss the pros and cons of security measures. This takes practical knowledge on how business and IT works, including practical knowledge in implementing protections or workarounds. Is it responsible to demand an entire web application be wrapped in SSL? In a way, empathize with the IT manager, the techs, and the business as a whole.

The bottom line: Be able to add something beyond just the deliverable report! The unfortunate reality of this is the need for security persons to have some practical implementation experience in business. The more you can basically make decisions for a manger, the more she will like you!

In the long term, there are two more traits one can evaluate sec geeks/groups by.

3) Ability to learn new techniques and tools. This ability still leaves open the option to use automated tools, but forces testers to be open and able to learning new things, including code, other automated tools, or manual options. I’m not one to rag on people who run automated scanners,* but I would rag on someone whose entire repetoire is an automated scanner.

4) The depth and breadth of their knowledge. It is hard to be both deep and broad in security, but the more you can be both, the more value you have. Practice, practice, practice! (And for god’s sake, manage expectations properly and admit when you are in over your head rather than flounder and deliver less value.) As an alternative, be extremely deep in your chosen area.

* It is my observation that there are three main phases to automated tools use. First phase is developmental where automated tools are fun and awe-inspiring. Then there’s the in-between phase which is where people start crying and yelling at “script kiddies” because they use automated tools. Then there is the third phase where realization sinks in that automated tools, even skiddie tools, can have value. Strive for phase 3!