Exploring Protocols - Part 1

In the process of doing software security analysis, it is pretty common to encounter unknown network protocols or file formats that are part of the attack surface you’re investigating.
Not too long ago, we wrote a post entitled Reversing a ZLib-obfuscated? Network Protocol where we talked about reversing an undocumented protocol to look for security weaknesses.

We got several good questions about some of our deductions about the protocol as we picked it apart. I’d like to take the opportunity to talk more about protocol reversing in general and hopefully help explain how that deduction process works while getting some broader coverage on the subject in.

This will be the first of at least 2 blog posts. I’m going to start by discussing building blocks and see where that takes us. In the early phases of talking about this process, I’m not making a distinction between whether a protocol is “unknown” because of lack of documentation or because it’s simply “unknown to you/me” because we’re unfamiliar with it.

Of course an undocumented protocol is going to be tricker to reverse. If there’s a point to these initial posts, it’s that working with documented protocols helps us understand the undocumented ones.