Cisco Live 2013 Final Thoughts

Image of JD on his bike in West Virginia

Image by Klaus Jones

I spent the last 5 days on the seat of my motorcycle driving hundreds of miles through the mountains of West Virginia. I do some of the best thinking on my motorcycle. The sound and vibrations of my pipes, driving with my whole body, leaning in and out of curves, the awareness of everything on, in, or around the road. Somehow, with all of that going on, I think A LOT.

As I continued to process everything I learned at Cisco Live, there were some thoughts that stuck out. These have very little to do with the social aspect, as I have already written about that here.

 Cisco Live Itself

1)   Why isn’t there a “lessons learned” document or post from the team who setup the wireless network? That was an incredible undertaking. I heard no complaints. I want to know what the Cisco Live Team has done over the last few years to scale the wireless network. Maybe the article is out there, but I haven’t seen it. This article wouldn’t be theory or sales, this is open communication about a real-life incredibly complex environment.

2)   Ditto on the WAN connection.

3)   As a first time, late registering attendee, I didn’t fully understand the Meet the Engineer, or the Table Topics at lunch. Now that I understand both, I will take better advantage of them next year.

4)   There is a special program for Netvets. There is a special party for CCIE’s. Why isn’t there a session on Sunday or early Monday for first timers? Make it a welcome party, initiation, meet and greet, and Q&A. I would have felt overwhelmed if it wasn’t for the great group of engineers that I hung out with at the Social Media Hub.  It would have also answered #3 above.

World of Solutions

I was surprised by the number of engineers running through the WoS chasing cheap plastic swords and other bits of junk. I liked a few of the T-Shirts, and grabbed a few of those, I picked up some buttons from Solarwinds, who clearly understands geek humor, and I avoided the rest. I realized on my ride this week, that the attendees were following the design. Run from booth to booth conquering and claiming prizes. Vendors, can I make a few suggestions?

1)   If you plan to give away shirts, make it a good design. If I like the design, I will wear it. Other engineers will see it. Conversations will be started about your company. Isn’t that the goal? If the design is bad, it will end up in the “donate” pile, as the yard work t-shirt, or used to wash cars. None of those are good for brand recognition. Special points given to geek humor and high quality shirts. If you want to guarantee that it sees the office, make it a polo shirt.

2)   Stop trying to win customers with a 5-minute pitch thrown out at the speed of sound by a mouthpiece that can’t answer questions. Your audience is technical. Do you think the audience can’t tell when the speaker is reciting words that they don’t understand?

3)   Find a way to engage potential customers. Make it easy for them to talk with a technical person, who can answer technical questions, and provide technical solutions. (Noticing a theme?)

4)   Don’t scoff at me when I refuse to provide my information for your cheap junk.

5)   Most importantly, don’t scoff when I sit through your presentation, give you my information, and then refuse your cheap junk. I am the person you are trying to reach, someone who is genuinely interested in your product, and who could easily be convinced to become a customer. I’m not there for the cheap junk, I’m there for more information about your product. If you could only answer my technical questions…

Now is a great time to register for next year!

Hey Apple, Help Us, Help You!

When the iPhone debuted on the AT&T network, AT&T was clearly not expecting the demand that was created. They were caught off-guard by the influx of customers, and more importantly they were surprised by the data consumption of users, who had purchased a device created to consume data. Problems were reported at a ridiculous rate, and rumors abounded everywhere within the Tech blogs that Apple was threatening to take their ball phone and go home to Verizon if AT&T didn’t do something fast.

In the mean time, Apple began working on ways of optimizing the iPhones use of the carriers network, and kept pushing AT&T for improvements. It took AT&T a couple years, and a LOT of money to build their network up. Some people will argue that if the iPhone had not been made available on other carriers that AT&T would still be having issues.

Apple studies, lives and dies by user experience. They knew that a poorly performing network would reflect on their device. It was not enough to simply blame the network. If the network wasn’t available, then features of their phone weren’t available either.

With that in mind… Apple DOES NOT provide developer access to wireless API’s in IOS. Troubleshooting WLAN issues for IOS devices can only be accomplished from the infrastructure side. Without jailbreaking an iPhone, there is no way to access RSSI, SNR, or other WLAN statistics.

Which device is best for troubleshooting iPad connectivity issues on a WLAN? If you answer anything other than “another iPad”, go directly to jail, do not pass go, and do not collect $200. This is an oversight decision that Apple needs to quickly reconsider.

Apple, we are the network. Without WLAN Engineers, iPads and iPhones won’t function correctly on corporate networks. Without the proper tools, WLAN engineers cannot support IOS devices when there are issues on the WLAN. Without tools, our network problems reflect on your devices. Help US, help YOU.

Explaining wireless overlap to non-techies.

Yesterday I was called about a problem in a new warehouse where I had recently rolled out wireless. I knew what the problem was before I ever logged into the wireless LAN controller. My organization leases approximate two thirds of a large warehouse, and the remaining space is occupied by various organizations. Those various organizations are broadcasting from 29 unique AP’s all crowded into the 2.4Ghz space.

I knew the issue, because I had raised the red flag before the project had even begun. I explained the problems that would be experienced, due to the other networks,  and that there was little I could do to mitigate the problem. I was able to work with the building owner to disable AP’s that existed on our side of the warehouse.

Since I had already explained the problem once, I thought I would take a different tack. I typed out a quick short story that explains the overlap problem, and sent it off. It seems the story made a positive impact and helped the manager understand the root of the problem. I thought I would share this to help bridge the gap between engineers and business managers, that need to understand wireless problems.

Bob is excited to finally be going to the XYZ annual conference in Podunck, Al. This year, the conference is bigger than ever, and he was lucky to even get a ticket. When he arrives, he learns that all sessions will be taking place in room 1, room 6, or room 11. Since he paid extra, he has two days of additional classes which he can choose to attend, and quickly fills his schedule.

On the first day, each session is taught from the stage, with the latest in PA equipment. The speaker is easily heard, and the presentation is clear and effective. Bob ask a few questions, and gets answers he both understands, and appreciates. He leaves feeling like he has learned an incredible amount in a very short period of time.

On the second day, more people have arrived at the conference, and he is surprised to find that each room has two classes going on at the same time. There is now a stage at each end of the room labeled A and B. Also, since the focus of the second day is Q&A, audience participation is paramount for the day to be effective. After breakfast, Bob gets a seat near Stage A, and while Stage B is distracting at times, he is still able to understand things that are being said. After lunch, however he isn’t so lucky. Near the middle of the auditorium noise from the Stage B often overwhelms the sound from stage A. Also, when the Stage B audience participates, he gets distracted, and forgets the question he wanted to ask the presenter on Stage A. Once he finally remembers, and gets the attention of Stage A, it is clear that they can’t understand him, so he repeats his question multiple times. Finally the presenter understands the question, but Stage B creates so much noise that Bob never hears the answer. Bob leaves that day feeling frustrated.

On the third day, everyone has arrived. Bob is horrified to learn that each room will have 4 sessions running simultaneously. The scene is pure pandemonium, and Bob does something smart…he spends the day playing golf.

Like Swiss Cheese – The road to being certifiable – Part 2

I tested, and received my first certification in 2000. I had been in IT for only 6 months, and I passed the Windows NT Server exam, which gave me the title of Microsoft Certified Professional. I did so after spending $7000 on a 6 month MCSE course. Finishing the course, just so happened to coincide with Microsoft announcing the end of the 4.0 track, and the beginning of the 2000 track. I didn’t have enough time to pass all 6 exams, to complete my MCSE, so I spent $7100, including the exam fee to attain my MCP. Needless to say, I wasn’t happy.

Moving forward over the next 5 years, I worked in every aspect of IT. I worked as help desk support, DBA, .Net programmer, and Web Developer. Eventually, I got sick of programming, and decided to plot my return to servers and networking. It was that or walk away from IT all together.

It took me a year, but I finally found a job that would trust me with their network, and I quickly made up for lost time. I fell in love with networking, and realized that I had finally found my niche in IT. Wired, wireless, firewalls, it all just made sense to me on a level that nothing else I had ever touched had.

Since that time, I have considered getting certified multiple times. In my opinion, the Cisco certifications are the most well respected vendor certifications available, and since I was working with about 90% Cisco equipment, there was no reason for me NOT to be certified. The only problem was, there didn’t seem to be any reason for me TO be certified either.

Salary surveys and employment studies seemed to indicate that certifications didn’t equal better pay, or higher level of employment. I have always been a busy guy, and passing certifications would require me to give up a lot of personal time that could be used to pursue other interest.

I was facing a motivation crisis. Couple that with my past experience in certifications, and the fear of, dare I say it, not passing an exam (also known as failure). I had plenty of reasons NOT to take a certification exam.

This all changed a couple of months ago. I made a couple of realizations that made getting certifications important to me, not for resume building, but for me as an individual.

I had just finished having a conversation with a junior level engineer over TRILL. I had explained in detail the finer points of TRILL vs. every vendors’ competitor. I discussed how it would most likely push L3 routing back into the Core and Distribution layers and out of the Access layer. I explained IS-IS.

Then, I was asked for help to setup a static Frame Relay map. My response was “Google for it” and I walked away quickly. I could discuss complex new technologies and yet somehow, a basic CCNA level task had escaped me. There were holes in my knowledge that I couldn’t escape.

I thought about that experience over the next couple of weeks. I realized that I was suddenly surrounded by real network experts through twitter: @etherealmind, @amyengineer, @matthewnorwood, @jtie_6ee7, @networkingnerd, @ecbanks, and many more. I liked the conversations that were taking place through blogs and other avenues. I also felt like I had a dirty little secret that would one day be discovered. I didn’t know (some) basic CCNA level stuff about networking.

It didn’t matter how well I could discuss PAGP vs. LACP, OSPF vs. EIGRP, IPv6, TRILL, or any other topic. It didn’t matter that my home network included an ASA and aironet AP. I could be easily stumped (without the internet) on basic topics that I never bothered to learn and memorize.

That was when I decided it was time to begin my certification journey. I would start with ICND1, taking no shortcuts. I wouldn’t take the CCNA composite test, in-case it didn’t cover a topic in-depth enough. I would become certified, and more importantly I would fill in the gaps, and know where I stood.

I easily passed the ICND1. According to Cisco, I have at least entry level experience and knowledge (surprising, right?). I quickly scheduled the ICND2, and there is where the holes appeared. On the portions of the test I knew, I didn’t miss any, or at least not more than one. ACL’s, OSPF, STP, and IP subnetting wasn’t a problem. There were problems though, and despite a few HORRIBLY worded questions, I can only blame myself. I missed passing by 21 points out of 1000.

Needless to say, I will be retaking the exam next week. I expect to pass, and more importantly, I will have filled in a few more holes.

ICND- Is Cisco oN Drugs – The road to being certifiable – Part 1

So I have a dirty little secret that I’m going to let you in on. Until recently the only IT certification that I held was an expired MCP certification dating back to the days of NT4.0. That’s right, I wasn’t a CCanything, and didn’t really see the need. I had years of experience on my resume, and didn’t want to put myself through the emotional distress caused by chasing certifications. There was also the question in the back of my mind: “if I begin taking exams, when do I stop, ccna, ccnp, ccie?”

So for reasons that will be explained later, I decided that it was time to begin the journey to become certifiable certified. Rather than jump directly into the Route, Switch, and Tshoot exams, which I really wanted to do, I instead decided to make myself step through it one step at a time, beginning with ICND1. I spent a week going over the material, just to be certain I knew what to expect, and scheduled the exam.

I wouldn’t say that I have “test anxiety” but anytime you spend $125 on an exam there are going to be strong emotions involved. I went into the exam a little nervous, but still expecting to pass easily.

This brings me to the reason that I HATE certification exams. I was shocked throughout the exam at how many poorly worded questions there were. I felt like I was arguing semantics with someone over whether or not “yes” means “yes” always or just on the odd and even numbered days. It finally led to the question that if I had been in an argument, I would have walked away before resorting to violence.

A loose paraphrase of the question was:

Which are swapped to change a straight-through cable to a crossover cable?
1 and 2
2 and 4
1 and 3
etc.
etc.

Now, the very first answer was “1 and 2” , which I understood to mean “the orange pair that includes wires 1 & 2” so I clicked the check box and then began looking for “3 and 6” to indicate the green pair. The only problem was, there was no “3 and 6” as an answer. I re-read the question, and all of the answers, still no “3 and 6”, I re-re-read, and still no dice.

At this point, I had seen a couple of poor questions or examples, and I was about to chalk this up as “another screw up on this stupid exam” and just click a box so that I could move on. But, I couldn’t stand to be beaten by such a simple question. I re-re-re-read the question, and finally figured out what they were really asking.

The question was really asking which STRANDS, WIRES, or PINS are swapped, not which PAIRS. Why one of those simple words was not used, I cannot tell you. It would have made the 4 minutes I spent on that question less than 20 seconds. At this point I was so frustrated with the many poorly worded questions, I spent 5 minutes writing a comment on this question before I moved on.

By the end of the exam I was sure that I had passed, and when my last two questions were “complex” subnetting questions, I guessed because I didn’t feel like doing the math, and wanted to be done with the exam. I passed with a great score, and ~20 minutes left.

I don’t mind challenging questions. I like to think, and want to know that when I complete an exam I have accomplished something. At the end of the ICND1 exam,  I had figured out what Cisco was TRYING TO ASK enough times to pass. That is all that I feel I accomplished.

Making people decipher and decode poorly written questions does not vet them as a capable certification candidate.

I now know that ICND really stands for “Is Cisco oN Drugs”

RSA can’t be trusted. Death to RSA.

RSA has finally admitted that it’s root certificates were compromised, which affects ALL SecurID tokens.

I personally feel that this shows absolute failure on the part of RSA. First, their root certificate was compromised. Second, rather than admit it, begin contacting customers immediately, and notifying the public, they chose to hide behind NDA’s while their customers were being compromised. RSA’s excuse for their lack of communication was that they didn’t want to give the attackers more information that could be used to exploit further companies. Based on the target of the attacks: Lockheed, Northrup Grumman, and L3 Communications, it is clear that the attackers knew everything already.

A company that was built on trusts and security has now been found completely untrustworthy and insecure. I expect to see major lawsuits resulting from this. I hope to see heads roll.

The company I work for uses these tokens. We have asked RSA for more information multiple times, but they have been slow in providing anything.

http://www.net-security.org/secworld.php?id=11122

Google warns of World IPv6 Day

Google is warning users of tomorrow’s test of IPv6, and more importantly of the fact that current IPv4 addresses have been depleted. I was only able to see the yellow banner in Linux running Firefox4, it never appeared on my Windows 7 machine. Google warning of IPv6 testing on June 8th.

While the banner is sure to cause some discussion among the non-networking crowd, I wish Google had included a link to more information. Instead they only include a link to test a users internet connection for IPv6 readiness. I don’t think the average user understands that their ISP is responsible for providing IPv6 connectivity, or of the problems that currently face IPv4.

I will give Google credit for starting the conversation. Hopefully, tomorrow there will be a lot of companies asking themselves what they must do to be ready for IPv6. Enterprise must lead IPv6 adoption, because as we all know, carriers are more than happy to sit on their butts as long as no one complains. The fact that so many ISP are considering CGN is a perfect example of that.

The velociraptor died after choking on a rib bone, so creating IPv7 is out of the question

OK, I admit it. I’ve had my head stuck firmly in the sand for almost 11 years. 11 years ago, to the month, I was sitting in my first TCP/IP class. I had fought through the first two days of class feeling mentally exhausted. I was finally beginning to wrap my head around IPv4 and variable length subnet mask. In fact, I was understanding IPv4 well enough that I could help my fellow students decipher the statements coming from our newly minted (and very proud of it) CCIE.
I was feeling pretty good about myself, and may have started to strut, just a little, as I moved from desk to desk, helping other students.
I should mention now, that I’m fairly quick on the up-take. I’m not bragging, simply stating that I meet the minimal requirements to be a geek. For some reason, I had really struggled with IPv4, so once I felt like I had a firm grasp of the concept, I was feeling pretty good.
My CCIE instructor, from his seat of power, saw a little pride develop in his class as more people caught the basics of VLSM. He, in the ultimate wisdom which comes with that coveted CCIE number, decided it was time to strangle those good feelings until they were most certainly dead. He did so, by launching into a 30 minute diatribe of how IPv4 would die a “quick death” and how IPv6 would take its place.
I’m sure you can imagine the look of horror on the faces of the students in the room. He certainly saw it, and fed off the fear as he blew through the broad topic that is IPv6. He delighted in mentioning that every device would have multiple IP’s, that each IP would be part of a different subnet. He threw out new words like anycast to a group of people who barely understood muilticast and unicast.
Wait, what?
In 30 minutes, he convinced three students that IT was not really the field they wanted to pursue, and the rest that IPv6 was EVIL. I was so affected and confused by that 30 minute rant, it took me five years before I had a practical understanding of subnetting IPv4 networks again.
Since that time, I have done my best to ignore the existence of IPv6. I used the fact that vendors were still releasing new products without IPv6 support as a reason to keep my eyes and ears firmly closed.
<My IPv6 Rant>
I believe that when IPv6 was being created someone said, “Yes, we COULD do that, but SHOULD we do that”. The rest of the attendees sat silently as he was taken from the room, and forced to watch his organs being fed to a genetically engineered, but very bored, velociraptor. The group then hired a soothsayer to read the velociraptor droppings, which gave us IPv6, reality TV, and the song “Friday”. The velociraptor died after choking on a rib bone, so creating IPv7 is out of the question.
</My IPv6 Rant>
With that said, IPv6 is here to stay, and it’s time for us, as Network Engineers, to get on board. We can’t complain about NAT64, without being willing to make the commitment to IPv6. When new protocols like TRILL are brought up for discussion, it’s easy to get excited. TRILL takes something that we already know (IS-IS, L2, etc) and simply builds on it. It is also transparent to layers 4-7, so it doesn’t affect non-network types.
IPv6, causes us to backtrack. It changes all of the rules. It’s not just IPv6, it’s new routing protocols, DNS, application stacks, etc. We have to forget what we learned in IPv4, and relearn it for IPv6. Server admins and developers will also have to update their skills. It’s painful.
With that acknowledged, we can’t put off learning to subnet, route, and filter IPv6. It’s time to begin examining IPv6 routing protocols, and buying equipment or ordering circuits which don’t support IPv6 should be out of the question. Yes, it does feel like starting from scratch. Yes, you will have to learn every protocol that you thought you knew all over again. Yes, IPv6 makes everything more complicated.
System Admins and developers can’t support IPv6 until we do. We must move forward, so that they can move forward.
Most network engineers agree that NAT is a poor solution to the problem staring us down. There are only a few other options. We can upgrade our skills, beginning the long arduous task of becoming experts in IPv6. We can ignore the change, until we are required to upgrade; then deal with entire IT teams being unprepared, learning on the fly, while implementing poor solutions in the near-term. Finally, we can make the same choice that those three classmates of mine did. “Maybe networking isn’t for me, I’ll go do something easier, like lion taming.”

Could Cisco Prime be the first step towards OpenFlow competition?

As it has been clear from my previous post, I have a love/hate relationship with Cisco. I love some of their products and I love working in IOS. There are also things that I hate: Cisco’s management platform and the lack of consistency between product lines; subnet mask vs. wildcard mask being a great example. Another thing I hate, Cisco’s management tools. CiscoWorks is a joke, and in smaller environments, where CiscoWorks would be overkill, companies are left with Cisco Network Assistant(CNA).

<RANT>
I realize that CNA is free, that Cisco doesn’t make any money on it, and that it was never meant for large enterprise. However, if there has ever been a product deserving of a “Beta” tag, I’m not sure what it is. What a piece of junk!
</RANT>

Now Cisco has released Cisco Prime. In all of the articles that I have read, the primary function is listed as “unified access across wired and wireless networks”. Clearly Cisco intends this to be a security solution. However, as you read further, things get a little more interesting. Here are the features as per a Cisco Blog post: http://bit.ly/gWBijM

Centralized Policy. Support any user on any device and provide secure access across the entire network by setting a single set of policies that can be distributed and enforced across the entire network.

Network Management. Unified management via Cisco Prime for wired and wireless networks helps increase IT efficiency, reduce IT training, and decrease time to resolve IT issues by providing a converged service-centric management platform.

Automation for Voice and Video. Ensure consistent high-quality user experience on any end-point. The latest innovations using Cisco Medianet enhancements provide automation and troubleshooting in the network to deliver application quality of experience, particularly video. Plus, organizations can reduce cost and time when resolving application choke points in the network, and scale applications to any endpoint with greater speed and efficiency.

The last two items are what piqued my interest. Unified network management, bandwidth control and shaping for audio and video; aren’t these features discussed when OpenFlow comes up? Is it possible that Cisco has recognized the need to address OpenFlow now, before it gets a stronger foothold in the market?

If I’ve properly read between the lines, and my guesses are accurate, there are a few things to remember. IF, then:

-This product has been rushed to production. I wouldn’t touch it within the first 6-9 months, or until it’s been upgraded at least once.
-Let’s face it, some of Cisco’s best new to market products were bought, not built internally. This was built internally. Enough said.
-Prime’s feature set will explode over the next few years, to make it better compete with the full OpenFlow feature set.
-The next version of IOS, ASA, WCS, etc. will have new hooks for this software to continue it’s feature expansion. Use caution with new versions of code for any devices. New hooks in the software = new security vulnerabilities and new bugs
-We may actually see a great security/network management product from Cisco in the next couple of years!

Texas Hold’em and the IETF – Did Brocade bet against TRILL?

For the last two post, which you can find HERE and HERE, I’ve knocked Cisco around. For those who don’t know me, I should warn that I am an equal opportunity offender. With that in mind, let’s take a look at Brocade’s implementation of TRILL.

As most of you should know, TRILL uses IS-IS on Layer 2 to identify the shortest path between switches, and load balance across those paths. Since this is happening at layer 2, not layer 3, it does away with Spanning Tree, which means more bandwidth and faster fail-over using the same number of ports, fiber paths, cables, and switches.

Of course, despite the fact that we all understand the above to be true, Brocade decided to go their own way and replace IS-IS with FSPF or Fabric Shortest Path First.

If you haven’t done much work in SAN environments, you may not be familiar with FSPF. Brocade created FSPF in 1997 to answer bandwidth concerns in Fiber Channel SANS. It has since become the standard path selection protocol in Fiber Channel fabrics.

With that understanding, let me back up and rephrase. As TRILL utilizing IS-IS was being developed by the IETF, Brocade a member of the IETF, decided to implement their own version of TRILL utilizing FSPF.

Brocade along with Cisco are both offenders. They both claim to be working with the IETF, yet at the same time both have released competitors to TRILL. Are we to believe that Brocade worked to make TRILL the best possible solution at the same time that they were creating a competitor to it? What about Cisco and FabricPath?

Both companies claim that their solution “extends” TRILL with additional features.

Were those “extended” features brought up in meetings when the TRILL standard was being discussed? Did the IETF choose to ignore those suggestions? I doubt it.

Cisco, Brocade, and most like every other vendor sat at the table the same way a poker player does during a game of texas hold ’em. No one showed their cards, but everyone watched the flop, river, and turn cards, to see what they could create with their own hands to drive the other players off the table.

Make no mistake, TRILL did not benefit from Brocade, Cisco, or any other vendor’s presence on the committee. Their involvement was for their own purposes, not the benefit of customers.