Author: subnetwork

Wireshark: Capture CDP and LLDP

A couple of years ago, I wrote a short piece about filtering CDP and LLDP packets using Wireshark. Since that time, I have simplified the way that I filter these packets, and based on feedback, and additional use of that information, I wanted to post an update. This will hopefully guide people to the best answer immediately.

CDP

CDP sends all packets to the L2 multicast address of 01:00:0C:CC:CC:CC. Therefore, our filter can be:

ether host 01:00:0c:cc:cc:cc

However, VTP (VLAN Trunking Protocol) also sends packets to this address. Since the default timer for VTP is 300 seconds, and the default timer for CDP is 60 seconds, this shouldn’t be an issue. Additionally, since VTP packets are only sent out trunk ports, if you see VTP packets on a port that a user should be connected to, you may have just found your problem.

LLDP

Link Layer Discovery Protocol, AKA 802.1AB, is an IEEE standard. While Cisco doesn’t support LLDP out of the box, it can be enabled on your Cisco gear. HP, Juniper, Dell, and everyone else that I have ever worked with supports LLDP by default. The L2 multicast address for LLDP is: 01:80:C2:00:00:0E. However, LLDP has the benefit of a unique EtherType. That type is: 0x88cc. Based on that information, we can filter with either:

ether host 01:80:C2:00:00:0E

OR

ether proto 0x88cc

The default timer for LLDP seems to vary across vendors, although 30 seconds is the default for Cisco and quite a few others.

Wireshark Portable

If you are the roving type that walks out to the users desk, Wireshark can be ran as a portable app from a USB device.

Cisco IP Phones

Cisco IP Phones will send out CDP packets onto the PC port. What good does this do? I don’t know. However, hit the webpage hosted on the phone and you can find the CDP and LLDP info on the Network Statistics >Network page.

Firewalls

Embarrassing story time. Like a lot of engineers, I regularly use Wireshark to look at packet captures from other devices. After doing this for months, I needed to use Wireshark on my local LAN port. I started by spending 20 minutes trying to figure out why I wasn’t seeing CDP packets. Of course, once I remembered that I had a local firewall to contend with, I quickly fixed the issue, and haven’t made that mistake since. Don’t make that mistake. Disable the local firewall.

Cisco Live 2013 Final Thoughts

Image of JD on his bike in West Virginia — Image by Klaus Jones

I spent the last 5 days on the seat of my motorcycle driving hundreds of miles through the mountains of West Virginia. I do some of the best thinking on my motorcycle. The sound and vibrations of my pipes, driving with my whole body, leaning in and out of curves, the awareness of everything on, in, or around the road. Somehow, with all of that going on, I think A LOT.

As I continued to process everything I learned at Cisco Live, there were some thoughts that stuck out. These have very little to do with the social aspect, as I have already written about that here.

Cisco Live Itself

1) Why isn’t there a “lessons learned” document or post from the team who setup the wireless network? That was an incredible undertaking. I heard no complaints. I want to know what the Cisco Live Team has done over the last few years to scale the wireless network. Maybe the article is out there, but I haven’t seen it. This article wouldn’t be theory or sales, this is open communication about a real-life incredibly complex environment.

2) Ditto on the WAN connection.

3) As a first time, late registering attendee, I didn’t fully understand the Meet the Engineer, or the Table Topics at lunch. Now that I understand both, I will take better advantage of them next year.

4) There is a special program for Netvets. There is a special party for CCIE’s. Why isn’t there a session on Sunday or early Monday for first timers? Make it a welcome party, initiation, meet and greet, and Q&A. I would have felt overwhelmed if it wasn’t for the great group of engineers that I hung out with at the Social Media Hub. It would have also answered #3 above.

World of Solutions

I was surprised by the number of engineers running through the WoS chasing cheap plastic swords and other bits of junk. I liked a few of the T-Shirts, and grabbed a few of those, I picked up some buttons from Solarwinds, who clearly understands geek humor, and I avoided the rest. I realized on my ride this week, that the attendees were following the design. Run from booth to booth conquering and claiming prizes. Vendors, can I make a few suggestions?

1) If you plan to give away shirts, make it a good design. If I like the design, I will wear it. Other engineers will see it. Conversations will be started about your company. Isn’t that the goal? If the design is bad, it will end up in the “donate” pile, as the yard work t-shirt, or used to wash cars. None of those are good for brand recognition. Special points given to geek humor and high quality shirts. If you want to guarantee that it sees the office, make it a polo shirt.

2) Stop trying to win customers with a 5-minute pitch thrown out at the speed of sound by a mouthpiece that can’t answer questions. Your audience is technical. Do you think the audience can’t tell when the speaker is reciting words that they don’t understand?

3) Find a way to engage potential customers. Make it easy for them to talk with a technical person, who can answer technical questions, and provide technical solutions. (Noticing a theme?)

4) Don’t scoff at me when I refuse to provide my information for your cheap junk.

5) Most importantly, don’t scoff when I sit through your presentation, give you my information, and then refuse your cheap junk. I am the person you are trying to reach, someone who is genuinely interested in your product, and who could easily be convinced to become a customer. I’m not there for the cheap junk, I’m there for more information about your product. If you could only answer my technical questions…

Now is a great time to register for next year!

Experiencing Cisco Live for the First Time

I’ve been back from Cisco Live 2013 for 5 days. I’m still not caught up at work on email or task that were assigned while I was away. It will most likely take a couple of weeks of working extra hours each day to finally get back to the point where I am only drowning a little. With such morose statements, it probably seems like I’m not happy with my first trip to Cisco Live.

In-fact, you cannot be more wrong. Cisco Live is like Space Camp for adults. Once you have been there, coming back to normal life is difficult, but I’ll get to that later.

I understand that I had an unusual first experience but I’m not special. If 2014 will be your first Cisco Live; your trip could be just as good as mine, or even better. If you came away from Cisco Live 2013 unimpressed, YOU ARE DOING IT WRONG.

From my time on Twitter (@subnetwork) and this blog, I knew a lot of other engineers via the Internet. The very first stop I made after checking in was The Social Media Hub. On Sunday there was a Twitter Meetup (tweet-up). This gave me chance to meet in-person engineers that I have been conversing with online for years.

The engineers who attend the Tweet-up are a special breed. They write blog post about network engineering. They post to twitter about network engineering. They think about network engineering a LOT. This was an opportunity to learn, teach, and otherwise geek out about network engineering without getting that look that says the other person checked out as soon as you mentioned LISP.

From there, Cisco took over their role, and managed to host an incredible event. I sat through classes, ate meals, walked the World of Solutions, went to the Customer Appreciation Event (CAE), and attended vendor parties each evening.

All of this falls into the standard experience. However, in my case, everywhere I went, there was someone from social media crowd. Lunches were discussions about problems at work, new technologies, classes we attended, difficulties in finding good coworkers, geek lore, and the list continues on.

From the Social Media Lounge, I was able to participate in various contests Cisco posted online. I scored a special pass to the CAE, which allowed me to meet the band Journey. Thanks to Twitter, I was invited to the CCIE event. While sitting in the lounge, between sessions, I was invited to participate in a Tech Field Day featuring Open Gear.

I met Journey, attended the CCIE party as a non-CCIE, and participated in a Tech Field Day. These aren’t part of the normal experience. None of this would have been part of my experience if I wasn’t active in social media and hadn’t sought out these great engineers who sleep, eat, and breathe networking.

Your task, if you choose to have an extraordinary experience next year is simple. Begin participating in the conversation now. Make virtual acquaintances now, then turn them into friends in San Francisco.

If you do it right, next year, you will have difficulty adjusting back to normal life. You will be overflowing with fresh knowledge, and will be looking for people to share it with. You will find yourself often wishing that you were back at Cisco Live so that you could share your excitement with someone else who LOVES what they do for a living as much as you do.

Now is a great time to register for next year!

Finding new coworkers

We have once again started the process of expanding our team at my workplace. We always bring new employees in as a contractor first, and if things work out, the contractor is usually offered a full time position.

Our interview process is fairly hard. First, all recruiters are provided with 10 questions, of which each candidate must answer 8 out of 10.These are all basic CCNA level questions.

Next, we schedule a phone screening, where we ask more specific questions that are broken up into different areas. Layer 2, Layer 3, OSPF, and QOS are all on tap for this portion of the interview process. If we feel that the person was able to answer enough questions correctly without frantically searching for answers to recite back to us on the internet, we move them onto the third phase.

In the third phase, the person comes into our offices, and we provide them with equipment and instructions. They have 2.5 hours to configure a router, switch, and an AP per our instructions and answer a few questions based on those configurations. Once they have completed the configuration, we move into a Q&A portion of the interview, where we ask off the wall questions, mixed with troubleshooting scenarios of increasing complexity.

Every person who has ever left an interview felt well abused. If they paid attention, they know their weaknesses, and could use it to start a personal improvement plan. We in-turn, have a solid read on each candidates abilities, strengths, and weaknesses, and whether they would make a good addition to the team.

This process is long and arduous; the last time we went through this process, we started the interview process on almost 60 people before we found three we liked.

I can’t knock our process though. In-fact, our team is so strong that I have turned down multiple offers at other positions, which payed more, simply because I like my teammates in my current role.

It seems that every time I consider taking a different role, I get pulled into interviewing more candidates, and am reminded what it is like out there in the rest of the world. Case-in-point, here is an email excerpt from a potential job candidate:

What did I say about scheduling issues earlier in one of your calls. All day long not a single trouble call comes in. 5 minutes before the time for the phone interview I get a call and 3 tickets logged into our dell kace service desk. Figured since I was finally done withe the remote assistance calls working from my terminal I would drop you a line while I am on the phone with one of the users that is having problems at the entire locationlocations that is having a problem that I am trying to get through to them it is sunding like a provider problem to let me let them go and get a hold of the provider. Always love a network that uses back up internet connections that are all from the same cable provider(comcast) so come off the same pole and think that it is a gfood redundancy feature. Not the one I am working with but we have one service center location that has all 3 retail branches of our company and instead of getting an upgrade on the connection type with 3 static ips for the way they like to do things but really makes no sense what so ever they have 3 cable modems all coming off the same pole so that they can supposedly have a better more stable connection makes me have nightmares about wasted money and the stupidity of the outside consultants that engineered this network.

After speaking to you prior to the interview time and you mentioning questions about switch configurations I will kinda admit you got me thinking it has been almost 10 years since I have programmed a true cisco switch do little netgear knock offs almost weekly and switches had always been my weak point give me a router or a pix device and I could make it sing but even on those I am rusty. Put me in a lab environment and it would be just like riding a bicycle but just giving me verbal questions I would be stumbling all over myself. Which looking at things makes me belive that this wouldn’t be the right position for me until I get back into the game and work away some of this rust.

If you don’t feel sick after reading that email, then a part of your soul is dead already.

Hey Apple, Help Us, Help You!

When the iPhone debuted on the AT&T network, AT&T was clearly not expecting the demand that was created. They were caught off-guard by the influx of customers, and more importantly they were surprised by the data consumption of users, who had purchased a device created to consume data. Problems were reported at a ridiculous rate, and rumors abounded everywhere within the Tech blogs that Apple was threatening to take their ~~ball~~ phone and go ~~home~~ to Verizon if AT&T didn’t do something fast.

In the mean time, Apple began working on ways of optimizing the iPhones use of the carriers network, and kept pushing AT&T for improvements. It took AT&T a couple years, and a LOT of money to build their network up. Some people will argue that if the iPhone had not been made available on other carriers that AT&T would still be having issues.

Apple studies, lives and dies by user experience. They knew that a poorly performing network would reflect on their device. It was not enough to simply blame the network. If the network wasn’t available, then features of their phone weren’t available either.

With that in mind… Apple DOES NOT provide developer access to wireless API’s in IOS. Troubleshooting WLAN issues for IOS devices can only be accomplished from the infrastructure side. Without jailbreaking an iPhone, there is no way to access RSSI, SNR, or other WLAN statistics.

Which device is best for troubleshooting iPad connectivity issues on a WLAN? If you answer anything other than “another iPad”, go directly to jail, do not pass go, and do not collect $200. This is an ~~oversight~~ decision that Apple needs to quickly reconsider.

Apple, we are the network. Without WLAN Engineers, iPads and iPhones won’t function correctly on corporate networks. Without the proper tools, WLAN engineers cannot support IOS devices when there are issues on the WLAN. Without tools, our network problems reflect on your devices. Help US, help YOU.

Supporting Apple devices on the WLAN

Since the iPad was released, it has received a mixed welcome within Enterprise environments. While a lot companies have at least some plan to move forward with iPads, these drivers are usually coming from the business side, instead of IT. In-fact, most IT shops are being dragged into IOS support with strong reluctance.

The broad questions which are causing resistance can be summed up in one word: SUPPORT. IT departments must figure out how to support the device in multiple areas. Information integrity and control, end-user support, and connectivity support all must be dealt with. Since this is a networking blog, I want to deal with the last one; and will do so in the next two articles.

Supporting iPads on the network is more complex than connecting them to an SSID and providing login credentials. If we look at the standard iPad user in most organizations, we see a highly mobile user, users who also have laptops. Most of these users requested an iPad after having a positive experience with their company issued iPhones. That translates to a user having three wireless devices at there desk at any given time: their laptop, their iPhone and their iPad.

To understand the problem this creates, let’s look at how we survey for a wireless network. There are two considerations: coverage and capacity.

Wireless Coverage
A survey can be based on square footage, and provide a certain RSSI from wall-to-wall. This is a perfectly acceptable way to survey if everyone has their own office. However in Cube-ville, a single AP may cover 100 desk or more. If each desk has one wireless device, you now have a physical medium (the channel or airspace) that is incapable of supporting all of the connected clients.

Wireless Capacity
The other way to perform a wireless survey is based on capacity. In a high capacity environment, the wireless spectrum, not the AP is the bottleneck. More on this later…

In a capacity based scenario, a number of desk are chosen, lets say 25. For every 25 desk, there is an AP. Those AP’s are placed based on coverage area, and in to minimize channel overlap. For the same 100 desk in Cube-ville, you now have 4 AP’s. Since there will be channel overlap, the radios are turned way down, and in general, the physical medium is now capable of handling the number of clients.

Taking this environment to the next step, each desk gets an iPhone, and a few months later, 1 in 4 request an iPad. We can safely assume that complaints will begin coming into IT about the wireless network. The AP airspace that was previously servicing 25 clients now contends with 62 per AP. Time for another wireless survey and at least twice as many AP’s!

Now we can see the problem that many companies are facing. The i-devices are here, and businesses seem to love them. The network team must begin planning and building now. I would like to make a few suggestions which might keep network teams from finding themselves behind the eight ball.

Budget to begin surveying your high density environments now.
Develop a plan for support, complete with timelines and cost. Present this to the highest management level you can reach, so that it can be considered as the business begins planning device deployments.
If your company has a charge-back system for devices, be certain a cost is associated with each IOS device to support the wireless network going forward.
Be certain to include a survey and additional equipment as a cost of any iPad rollout projects, make certain the business can see the total cost of deploying iPads and iPhones.
Finally, be first in line to get an iPad if you don’t already have one. You can’t support what you don’t understand; besides, it really is a great device.

I realize that there are other options out there other than the “i” devices. However, I haven’t heard of, or seen, a single enterprise level roll out. However, these rules apply to the world of Android and Windows too. More devices per square foot equals more demand on the wireless network.

Explaining wireless overlap to non-techies.

Yesterday I was called about a problem in a new warehouse where I had recently rolled out wireless. I knew what the problem was before I ever logged into the wireless LAN controller. My organization leases approximate two thirds of a large warehouse, and the remaining space is occupied by various organizations. Those various organizations are broadcasting from 29 unique AP’s all crowded into the 2.4Ghz space.

I knew the issue, because I had raised the red flag before the project had even begun. I explained the problems that would be experienced, due to the other networks, and that there was little I could do to mitigate the problem. I was able to work with the building owner to disable AP’s that existed on our side of the warehouse.

Since I had already explained the problem once, I thought I would take a different tack. I typed out a quick short story that explains the overlap problem, and sent it off. It seems the story made a positive impact and helped the manager understand the root of the problem. I thought I would share this to help bridge the gap between engineers and business managers, that need to understand wireless problems.

Bob is excited to finally be going to the XYZ annual conference in Podunck, Al. This year, the conference is bigger than ever, and he was lucky to even get a ticket. When he arrives, he learns that all sessions will be taking place in room 1, room 6, or room 11. Since he paid extra, he has two days of additional classes which he can choose to attend, and quickly fills his schedule.

On the first day, each session is taught from the stage, with the latest in PA equipment. The speaker is easily heard, and the presentation is clear and effective. Bob ask a few questions, and gets answers he both understands, and appreciates. He leaves feeling like he has learned an incredible amount in a very short period of time.

On the second day, more people have arrived at the conference, and he is surprised to find that each room has two classes going on at the same time. There is now a stage at each end of the room labeled A and B. Also, since the focus of the second day is Q&A, audience participation is paramount for the day to be effective. After breakfast, Bob gets a seat near Stage A, and while Stage B is distracting at times, he is still able to understand things that are being said. After lunch, however he isn’t so lucky. Near the middle of the auditorium noise from the Stage B often overwhelms the sound from stage A. Also, when the Stage B audience participates, he gets distracted, and forgets the question he wanted to ask the presenter on Stage A. Once he finally remembers, and gets the attention of Stage A, it is clear that they can’t understand him, so he repeats his question multiple times. Finally the presenter understands the question, but Stage B creates so much noise that Bob never hears the answer. Bob leaves that day feeling frustrated.

On the third day, everyone has arrived. Bob is horrified to learn that each room will have 4 sessions running simultaneously. The scene is pure pandemonium, and Bob does something smart…he spends the day playing golf.

Archiving IOS configurations

Ethan Banks has a great article over at the PacketPushers website detailing the simple setup required to archive switch configurations to an FTP Server. I’ve looked for this type of solution before, but haven’t ever seen it explained so well.

There are valid security concerns that come with clear text passwords and usernames in the config, but in my opinion, if an outsider is already looking at my config, the ftp account is the least of my problems.

Be sure to give it a read, and leave Ethan a comment.

[LINK]

And now for something completely different…an iPad background for Network Engineers

I’ve never really found a lock screen for my iPad. I’ve been looking for something that has some geek humor, with a little bit of, “yes, I am a network engineer” mixed in. Finally, I decided to create something. I’m tossing it out for anyone to use.

What would you like to see as an iPad background? What IOS commands scream “Look at me every time you turn on your iPad”?

Like Swiss Cheese – The road to being certifiable – Part 2

I tested, and received my first certification in 2000. I had been in IT for only 6 months, and I passed the Windows NT Server exam, which gave me the title of Microsoft Certified Professional. I did so after spending $7000 on a 6 month MCSE course. Finishing the course, just so happened to coincide with Microsoft announcing the end of the 4.0 track, and the beginning of the 2000 track. I didn’t have enough time to pass all 6 exams, to complete my MCSE, so I spent $7100, including the exam fee to attain my MCP. Needless to say, I wasn’t happy.

Moving forward over the next 5 years, I worked in every aspect of IT. I worked as help desk support, DBA, .Net programmer, and Web Developer. Eventually, I got sick of programming, and decided to plot my return to servers and networking. It was that or walk away from IT all together.

It took me a year, but I finally found a job that would trust me with their network, and I quickly made up for lost time. I fell in love with networking, and realized that I had finally found my niche in IT. Wired, wireless, firewalls, it all just made sense to me on a level that nothing else I had ever touched had.

Since that time, I have considered getting certified multiple times. In my opinion, the Cisco certifications are the most well respected vendor certifications available, and since I was working with about 90% Cisco equipment, there was no reason for me NOT to be certified. The only problem was, there didn’t seem to be any reason for me TO be certified either.

Salary surveys and employment studies seemed to indicate that certifications didn’t equal better pay, or higher level of employment. I have always been a busy guy, and passing certifications would require me to give up a lot of personal time that could be used to pursue other interest.

I was facing a motivation crisis. Couple that with my past experience in certifications, and the fear of, dare I say it, not passing an exam (also known as failure). I had plenty of reasons NOT to take a certification exam.

This all changed a couple of months ago. I made a couple of realizations that made getting certifications important to me, not for resume building, but for me as an individual.

I had just finished having a conversation with a junior level engineer over TRILL. I had explained in detail the finer points of TRILL vs. every vendors’ competitor. I discussed how it would most likely push L3 routing back into the Core and Distribution layers and out of the Access layer. I explained IS-IS.

Then, I was asked for help to setup a static Frame Relay map. My response was “Google for it” and I walked away quickly. I could discuss complex new technologies and yet somehow, a basic CCNA level task had escaped me. There were holes in my knowledge that I couldn’t escape.

I thought about that experience over the next couple of weeks. I realized that I was suddenly surrounded by real network experts through twitter: @etherealmind, @amyengineer, @matthewnorwood, @jtie_6ee7, @networkingnerd, @ecbanks, and many more. I liked the conversations that were taking place through blogs and other avenues. I also felt like I had a dirty little secret that would one day be discovered. I didn’t know (some) basic CCNA level stuff about networking.

It didn’t matter how well I could discuss PAGP vs. LACP, OSPF vs. EIGRP, IPv6, TRILL, or any other topic. It didn’t matter that my home network included an ASA and aironet AP. I could be easily stumped (without the internet) on basic topics that I never bothered to learn and memorize.

That was when I decided it was time to begin my certification journey. I would start with ICND1, taking no shortcuts. I wouldn’t take the CCNA composite test, in-case it didn’t cover a topic in-depth enough. I would become certified, and more importantly I would fill in the gaps, and know where I stood.

I easily passed the ICND1. According to Cisco, I have at least entry level experience and knowledge (surprising, right?). I quickly scheduled the ICND2, and there is where the holes appeared. On the portions of the test I knew, I didn’t miss any, or at least not more than one. ACL’s, OSPF, STP, and IP subnetting wasn’t a problem. There were problems though, and despite a few HORRIBLY worded questions, I can only blame myself. I missed passing by 21 points out of 1000.

Needless to say, I will be retaking the exam next week. I expect to pass, and more importantly, I will have filled in a few more holes.