February 21, 2015

Packets Don't Lie Wireshark, and Sniffer Days of Old

What more could anyone want?

I have been using protocol analyzers since the days of Network General and have always loved the analytic view of the network.

Not the smoke and mirrors we usually show the users, but the hardcore, what is on the wire, proof.

Packets don't lie.

One knowledgeable look and you see it. You just see it in real time. It magically appears.

OK, it's not a magic application one just opens, and it tells you exactly how to fix your issues. There is some analyzing and knowledge required of how things are supposed to work to get real use out of it.

Like any tool it helps if you know how to use it.

Protocol Analysis is a tool to look at how devices are actually communicating. Not how they are supposed to be communicating.

The Awakening

My first exposure to sniffing was a production issue at a previous employer 20+ years ago. We had a subnet having intermittent communication issues with the onsite data center.

Youngling Geek

This was early in my career and tools like wireshark or any free sniffer tools simply didn't exist to the average public. Heck it wasn't even switches back then it was Synoptics hubs.

At the time I was one of the onsite server/PC geeks and our network people were located in the Midwest. Three days of local troubleshooting and finally we get a visit from the network home office cavalry.

The Awe Effect

I remember the color of the carpet tiles being bright yellow because in walked in this blonde network goddess that put the color to shame. A presence larger than life. Little did I know then but she would become my mentor, friend, and confidant for many years. She was the brightest, most intelligently complex person I have ever met. The kind of intelligence that reminds me of standing room only in a conference hall listening to Radia Perlman speak. If you got 50% of what was said you were doing extremely well.

She solved the issue in less than 5 minutes and spent the next two days showing me the sniffer, packets, and answering the barrage of questions which spewed forth.

The Windy City and Italian Beef Sandwiches

Within three months, I transferred teams and moved to Chicago, and was working and learning in the network group for three years under her tutelage. I look back fondly on those times and continue to build upon that foundation today.

One thing that hasn't changed is that protocol analysis is an awesome resource to be able to use when the need arises. Which in my opinion, happens too late in the troubleshooting process, most times.

Most tech people today know sniffers exist and most can easily step through setting up a capture. Some may even open the capture and see something standing out in red that solves the issue. But usually that is where it stops.

Sniff something? Well did ya, rat boy?

Most organization I've worked with, the sniffer is pulled out late in the troubleshooting process. The initial troubleshooting being done by the front line teams. Usually in time, the issue gathers attention and the old neck beards come out of the cave. Usually they were stating it wasn't a network problem and see they can prove it. Often the protocol analysis helped pin point the issue and usually that it wasn't a network issue. They would then get in their late model sedan and go grab lunch early to avoid the "people" rush.

I'm a big proponent of front line techs knowing how and where to take the capture. Even if it's just to pass the pcap to an engineer later. A sample of whats actually going on can go a long way when troubleshooting. Verse all to familiar "Everything is slow" or "the computers are down".

I think the Wireshark Certified Network Analyst WCNA Certification objectives are a good starting point for many admins.