VIAVI Observer Apex- Finding the needle faster

I participated in the Tech Field Day Extra events at Cisco Live. One of the presenters, VIAVI has been floating near the edge of my awareness for a while, so it was great to see their presentation and get a better understanding of the VIAVI Observer Platform.

Anytime I see a presentation from a monitoring solution there are three questions that I ask:

“How useful would this be for tier one technicians?”

I usually consider that question from both the perspective of a NOC and also a helpdesk technician. If a monitoring tool isn’t practical for those roles, I am the one who gets stuck using it all of the time, and therefore, it has no place in my environment.

“How useful would this tool be for me?”

If the tool can’t offer enough information to be useful for a senior engineer, I don’t want to pay for it. It also increases the complexity of passing trouble tickets up the chain as each person has to start back at zero in their own tool.

“Does this make it easier to find the problem, or just add another step?”

Monitoring tools which only show up/down status and system logs have very little use for me. I can easily find those by other means, or on the device itself, faster than I can fire up a browser, click on a bookmark, log in, navigate through a device tree, etc.

VIAVI has provided the right answers to all three questions.

product-obsever-apex-welcome

The starting page for Observer is simple. It doesn’t take forever to load as it attempts to pull data from many different sources to provide a general health overview that rarely has anything to do with the reason you opened the application. Instead, Observer’s search box is ready for any relevant text the technician may know about the problem. If you have an IP, MAC address, VLAN, or hostname, those are all great places to start. You can also choose to push into a more generalized monitoring view like Application Performance, Network Performance, etc.

The search box is the beauty of the application for me. VIAVI indexes all of the monitoring sources for things like MAC addresses, IP addresses, interfaces, usernames, and other metadata and then correlates that information together. A technician doesn’t need to look up an IP address in the ARP table, get the MAC address, look up the MAC address in the MAC address table to get the port, then check the port for errors. A search on the IP address will provide all of that information, quickly! Since VIAVI also knows the assigned VLAN, it quickly displays “Here’s a bad actor on the same VLAN that is flooding the VLAN with bad frames.” The technicians can find problems without looking directly for them. That’s a huge win. This is not looking for a needle in a haystack. This is turning on an extremely powerful magnet and letting the needle come to you.

Another great feature is that Observer creates a baseline from the information that it acquires. With that baseline that understands system X typically runs at 75 percent utilization, but is now running at 90 percent, more problems quickly float to the surface. Additionally, the baseline filters out the normal abnormal. Is it “normal” for that system to run at 75 percent utilization all of the time? Maybe so. If it is, a technician doesn’t need a warning about it. It might be operating as designed.

If a technician can’t find a solution through the dashboard, the next engineer who picks up the problem will want to dig deeper. Thanks to the stored packet traces which provided all of the metadata the technician used, the engineer can take a look at the actual packets. Aside from the standard fields like source and destination, IP’s and ports, Observer also includes a patent-pending User Experience Score which is a 1-10 scale to aid in finding problems faster within the trace files.

Taking the click-through troubleshooting one step further, Observer creates Application Dependency Maps which aid an engineer to understand all of the dependent systems quickly and which are affecting performance.

When considering my initial three questions I proposed, I feel VIAVI’s Observer is providing pretty compelling answers for each. I look forward to learning more.

In many ways, Tech Field Day offers a similar solution to VIAVI Observer. TFD allows me to filter through the marketing hype, and get to the bottom of a product or solution and whether it will be useful to me. Don’t forget to check out the many other videos and content created by Tech Field Day at Cisco Live.

Cisco Live US 2014 – Engage Now!

Last year, I attended Cisco Live for the first time in my career. I went expecting to learn a lot, and I was not disappointed. You can read about my experiences here and here. If you haven’t read them, you should read them now. No, really, go read them. 

Now that you have read them, you know that you need to begin planning your social experience now. The scheduler will soon be available, and while you are considering the need of various classes, be certain that you create time slots to meet people. There is an incredible braintrust available in the social media hub. If you take the time to mingle and discuss you will be surprised at what you will learn.

I have long been a proponent of Twitter for IT professionals. If you and I have met over the last few years, and I haven’t asked about your social media interaction, I would question whether you actually met me and not a doppelgänger. If you have actually met me, I hope that my influence, no matter how small, pushed you to engage.

If you are new to social media and planning on hanging out in the Social Media Hub, let me offer a few suggestions.

  • Engage now. Don’t expect to show up to the social media hub without ever talking to any other engineer on twitter and expect to enjoy your experience. We like our jokes, our running discussions (arguments), and interacting. The social media hub is our opportunity to continue our online discussions in person. If you want a great list of engineers to follow, just check out who I follow.
  • Don’t be afraid to ask questions. We all come from a different background. Some of us are jack-of-all-trades, some of us specialize. We don’t expect anyone to be an expert in everything. We enjoy learning from each other. If you listen, and ask questions, you will learn.
  • Leave the oversized ego at home. Most of us have bigger personalities than egos. There are people in this group who know more than you. Trust me on this! If you show up with the goal of proving how smart you are, you’re going to have a bad time.
  • Don’t worship at the feet of your favorite author/personality. Yes, they will hang out with us and yes, they know an incredible amount about certain topics. Without exception though, they don’t want to be placed on a juvenile pedestal. They want to engage with other engineers. Story time:

Last year, I started a conversation with a well known author. We talked about our careers, about IT in general and the direction of technology. During these conversations, no less than 15 people approached to tell the author how great he was. The author was very happy to talk with them, and many times tried to draw the individual into our conversation. He would introduce me, mention the topic we were discussing at the moment, and made a genuine attempt to engage them in the discussion. Without fail, they thanked the author for his work, and then shyly withdrew. They were worshiping, not engaging.

  • Finally, register NOW! Register now to be certain you can attend the session that you want or need. This will also ensure that you can get an exam registered before all of the slots are filled. You can register here:

Cisco Live Registration

Like Swiss Cheese – The road to being certifiable – Part 2

I tested, and received my first certification in 2000. I had been in IT for only 6 months, and I passed the Windows NT Server exam, which gave me the title of Microsoft Certified Professional. I did so after spending $7000 on a 6 month MCSE course. Finishing the course, just so happened to coincide with Microsoft announcing the end of the 4.0 track, and the beginning of the 2000 track. I didn’t have enough time to pass all 6 exams, to complete my MCSE, so I spent $7100, including the exam fee to attain my MCP. Needless to say, I wasn’t happy.

Moving forward over the next 5 years, I worked in every aspect of IT. I worked as help desk support, DBA, .Net programmer, and Web Developer. Eventually, I got sick of programming, and decided to plot my return to servers and networking. It was that or walk away from IT all together.

It took me a year, but I finally found a job that would trust me with their network, and I quickly made up for lost time. I fell in love with networking, and realized that I had finally found my niche in IT. Wired, wireless, firewalls, it all just made sense to me on a level that nothing else I had ever touched had.

Since that time, I have considered getting certified multiple times. In my opinion, the Cisco certifications are the most well respected vendor certifications available, and since I was working with about 90% Cisco equipment, there was no reason for me NOT to be certified. The only problem was, there didn’t seem to be any reason for me TO be certified either.

Salary surveys and employment studies seemed to indicate that certifications didn’t equal better pay, or higher level of employment. I have always been a busy guy, and passing certifications would require me to give up a lot of personal time that could be used to pursue other interest.

I was facing a motivation crisis. Couple that with my past experience in certifications, and the fear of, dare I say it, not passing an exam (also known as failure). I had plenty of reasons NOT to take a certification exam.

This all changed a couple of months ago. I made a couple of realizations that made getting certifications important to me, not for resume building, but for me as an individual.

I had just finished having a conversation with a junior level engineer over TRILL. I had explained in detail the finer points of TRILL vs. every vendors’ competitor. I discussed how it would most likely push L3 routing back into the Core and Distribution layers and out of the Access layer. I explained IS-IS.

Then, I was asked for help to setup a static Frame Relay map. My response was “Google for it” and I walked away quickly. I could discuss complex new technologies and yet somehow, a basic CCNA level task had escaped me. There were holes in my knowledge that I couldn’t escape.

I thought about that experience over the next couple of weeks. I realized that I was suddenly surrounded by real network experts through twitter: @etherealmind, @amyengineer, @matthewnorwood, @jtie_6ee7, @networkingnerd, @ecbanks, and many more. I liked the conversations that were taking place through blogs and other avenues. I also felt like I had a dirty little secret that would one day be discovered. I didn’t know (some) basic CCNA level stuff about networking.

It didn’t matter how well I could discuss PAGP vs. LACP, OSPF vs. EIGRP, IPv6, TRILL, or any other topic. It didn’t matter that my home network included an ASA and aironet AP. I could be easily stumped (without the internet) on basic topics that I never bothered to learn and memorize.

That was when I decided it was time to begin my certification journey. I would start with ICND1, taking no shortcuts. I wouldn’t take the CCNA composite test, in-case it didn’t cover a topic in-depth enough. I would become certified, and more importantly I would fill in the gaps, and know where I stood.

I easily passed the ICND1. According to Cisco, I have at least entry level experience and knowledge (surprising, right?). I quickly scheduled the ICND2, and there is where the holes appeared. On the portions of the test I knew, I didn’t miss any, or at least not more than one. ACL’s, OSPF, STP, and IP subnetting wasn’t a problem. There were problems though, and despite a few HORRIBLY worded questions, I can only blame myself. I missed passing by 21 points out of 1000.

Needless to say, I will be retaking the exam next week. I expect to pass, and more importantly, I will have filled in a few more holes.

ICND- Is Cisco oN Drugs – The road to being certifiable – Part 1

So I have a dirty little secret that I’m going to let you in on. Until recently the only IT certification that I held was an expired MCP certification dating back to the days of NT4.0. That’s right, I wasn’t a CCanything, and didn’t really see the need. I had years of experience on my resume, and didn’t want to put myself through the emotional distress caused by chasing certifications. There was also the question in the back of my mind: “if I begin taking exams, when do I stop, ccna, ccnp, ccie?”

So for reasons that will be explained later, I decided that it was time to begin the journey to become certifiable certified. Rather than jump directly into the Route, Switch, and Tshoot exams, which I really wanted to do, I instead decided to make myself step through it one step at a time, beginning with ICND1. I spent a week going over the material, just to be certain I knew what to expect, and scheduled the exam.

I wouldn’t say that I have “test anxiety” but anytime you spend $125 on an exam there are going to be strong emotions involved. I went into the exam a little nervous, but still expecting to pass easily.

This brings me to the reason that I HATE certification exams. I was shocked throughout the exam at how many poorly worded questions there were. I felt like I was arguing semantics with someone over whether or not “yes” means “yes” always or just on the odd and even numbered days. It finally led to the question that if I had been in an argument, I would have walked away before resorting to violence.

A loose paraphrase of the question was:

Which are swapped to change a straight-through cable to a crossover cable?
1 and 2
2 and 4
1 and 3
etc.
etc.

Now, the very first answer was “1 and 2” , which I understood to mean “the orange pair that includes wires 1 & 2” so I clicked the check box and then began looking for “3 and 6” to indicate the green pair. The only problem was, there was no “3 and 6” as an answer. I re-read the question, and all of the answers, still no “3 and 6”, I re-re-read, and still no dice.

At this point, I had seen a couple of poor questions or examples, and I was about to chalk this up as “another screw up on this stupid exam” and just click a box so that I could move on. But, I couldn’t stand to be beaten by such a simple question. I re-re-re-read the question, and finally figured out what they were really asking.

The question was really asking which STRANDS, WIRES, or PINS are swapped, not which PAIRS. Why one of those simple words was not used, I cannot tell you. It would have made the 4 minutes I spent on that question less than 20 seconds. At this point I was so frustrated with the many poorly worded questions, I spent 5 minutes writing a comment on this question before I moved on.

By the end of the exam I was sure that I had passed, and when my last two questions were “complex” subnetting questions, I guessed because I didn’t feel like doing the math, and wanted to be done with the exam. I passed with a great score, and ~20 minutes left.

I don’t mind challenging questions. I like to think, and want to know that when I complete an exam I have accomplished something. At the end of the ICND1 exam,  I had figured out what Cisco was TRYING TO ASK enough times to pass. That is all that I feel I accomplished.

Making people decipher and decode poorly written questions does not vet them as a capable certification candidate.

I now know that ICND really stands for “Is Cisco oN Drugs”

Google warns of World IPv6 Day

Google is warning users of tomorrow’s test of IPv6, and more importantly of the fact that current IPv4 addresses have been depleted. I was only able to see the yellow banner in Linux running Firefox4, it never appeared on my Windows 7 machine. Google warning of IPv6 testing on June 8th.

While the banner is sure to cause some discussion among the non-networking crowd, I wish Google had included a link to more information. Instead they only include a link to test a users internet connection for IPv6 readiness. I don’t think the average user understands that their ISP is responsible for providing IPv6 connectivity, or of the problems that currently face IPv4.

I will give Google credit for starting the conversation. Hopefully, tomorrow there will be a lot of companies asking themselves what they must do to be ready for IPv6. Enterprise must lead IPv6 adoption, because as we all know, carriers are more than happy to sit on their butts as long as no one complains. The fact that so many ISP are considering CGN is a perfect example of that.

The velociraptor died after choking on a rib bone, so creating IPv7 is out of the question

OK, I admit it. I’ve had my head stuck firmly in the sand for almost 11 years. 11 years ago, to the month, I was sitting in my first TCP/IP class. I had fought through the first two days of class feeling mentally exhausted. I was finally beginning to wrap my head around IPv4 and variable length subnet mask. In fact, I was understanding IPv4 well enough that I could help my fellow students decipher the statements coming from our newly minted (and very proud of it) CCIE.
I was feeling pretty good about myself, and may have started to strut, just a little, as I moved from desk to desk, helping other students.
I should mention now, that I’m fairly quick on the up-take. I’m not bragging, simply stating that I meet the minimal requirements to be a geek. For some reason, I had really struggled with IPv4, so once I felt like I had a firm grasp of the concept, I was feeling pretty good.
My CCIE instructor, from his seat of power, saw a little pride develop in his class as more people caught the basics of VLSM. He, in the ultimate wisdom which comes with that coveted CCIE number, decided it was time to strangle those good feelings until they were most certainly dead. He did so, by launching into a 30 minute diatribe of how IPv4 would die a “quick death” and how IPv6 would take its place.
I’m sure you can imagine the look of horror on the faces of the students in the room. He certainly saw it, and fed off the fear as he blew through the broad topic that is IPv6. He delighted in mentioning that every device would have multiple IP’s, that each IP would be part of a different subnet. He threw out new words like anycast to a group of people who barely understood muilticast and unicast.
Wait, what?
In 30 minutes, he convinced three students that IT was not really the field they wanted to pursue, and the rest that IPv6 was EVIL. I was so affected and confused by that 30 minute rant, it took me five years before I had a practical understanding of subnetting IPv4 networks again.
Since that time, I have done my best to ignore the existence of IPv6. I used the fact that vendors were still releasing new products without IPv6 support as a reason to keep my eyes and ears firmly closed.
<My IPv6 Rant>
I believe that when IPv6 was being created someone said, “Yes, we COULD do that, but SHOULD we do that”. The rest of the attendees sat silently as he was taken from the room, and forced to watch his organs being fed to a genetically engineered, but very bored, velociraptor. The group then hired a soothsayer to read the velociraptor droppings, which gave us IPv6, reality TV, and the song “Friday”. The velociraptor died after choking on a rib bone, so creating IPv7 is out of the question.
</My IPv6 Rant>
With that said, IPv6 is here to stay, and it’s time for us, as Network Engineers, to get on board. We can’t complain about NAT64, without being willing to make the commitment to IPv6. When new protocols like TRILL are brought up for discussion, it’s easy to get excited. TRILL takes something that we already know (IS-IS, L2, etc) and simply builds on it. It is also transparent to layers 4-7, so it doesn’t affect non-network types.
IPv6, causes us to backtrack. It changes all of the rules. It’s not just IPv6, it’s new routing protocols, DNS, application stacks, etc. We have to forget what we learned in IPv4, and relearn it for IPv6. Server admins and developers will also have to update their skills. It’s painful.
With that acknowledged, we can’t put off learning to subnet, route, and filter IPv6. It’s time to begin examining IPv6 routing protocols, and buying equipment or ordering circuits which don’t support IPv6 should be out of the question. Yes, it does feel like starting from scratch. Yes, you will have to learn every protocol that you thought you knew all over again. Yes, IPv6 makes everything more complicated.
System Admins and developers can’t support IPv6 until we do. We must move forward, so that they can move forward.
Most network engineers agree that NAT is a poor solution to the problem staring us down. There are only a few other options. We can upgrade our skills, beginning the long arduous task of becoming experts in IPv6. We can ignore the change, until we are required to upgrade; then deal with entire IT teams being unprepared, learning on the fly, while implementing poor solutions in the near-term. Finally, we can make the same choice that those three classmates of mine did. “Maybe networking isn’t for me, I’ll go do something easier, like lion taming.”

Could Cisco Prime be the first step towards OpenFlow competition?

As it has been clear from my previous post, I have a love/hate relationship with Cisco. I love some of their products and I love working in IOS. There are also things that I hate: Cisco’s management platform and the lack of consistency between product lines; subnet mask vs. wildcard mask being a great example. Another thing I hate, Cisco’s management tools. CiscoWorks is a joke, and in smaller environments, where CiscoWorks would be overkill, companies are left with Cisco Network Assistant(CNA).

<RANT>
I realize that CNA is free, that Cisco doesn’t make any money on it, and that it was never meant for large enterprise. However, if there has ever been a product deserving of a “Beta” tag, I’m not sure what it is. What a piece of junk!
</RANT>

Now Cisco has released Cisco Prime. In all of the articles that I have read, the primary function is listed as “unified access across wired and wireless networks”. Clearly Cisco intends this to be a security solution. However, as you read further, things get a little more interesting. Here are the features as per a Cisco Blog post: http://bit.ly/gWBijM

Centralized Policy. Support any user on any device and provide secure access across the entire network by setting a single set of policies that can be distributed and enforced across the entire network.

Network Management. Unified management via Cisco Prime for wired and wireless networks helps increase IT efficiency, reduce IT training, and decrease time to resolve IT issues by providing a converged service-centric management platform.

Automation for Voice and Video. Ensure consistent high-quality user experience on any end-point. The latest innovations using Cisco Medianet enhancements provide automation and troubleshooting in the network to deliver application quality of experience, particularly video. Plus, organizations can reduce cost and time when resolving application choke points in the network, and scale applications to any endpoint with greater speed and efficiency.

The last two items are what piqued my interest. Unified network management, bandwidth control and shaping for audio and video; aren’t these features discussed when OpenFlow comes up? Is it possible that Cisco has recognized the need to address OpenFlow now, before it gets a stronger foothold in the market?

If I’ve properly read between the lines, and my guesses are accurate, there are a few things to remember. IF, then:

-This product has been rushed to production. I wouldn’t touch it within the first 6-9 months, or until it’s been upgraded at least once.
-Let’s face it, some of Cisco’s best new to market products were bought, not built internally. This was built internally. Enough said.
-Prime’s feature set will explode over the next few years, to make it better compete with the full OpenFlow feature set.
-The next version of IOS, ASA, WCS, etc. will have new hooks for this software to continue it’s feature expansion. Use caution with new versions of code for any devices. New hooks in the software = new security vulnerabilities and new bugs
-We may actually see a great security/network management product from Cisco in the next couple of years!

Texas Hold’em and the IETF – Did Brocade bet against TRILL?

For the last two post, which you can find HERE and HERE, I’ve knocked Cisco around. For those who don’t know me, I should warn that I am an equal opportunity offender. With that in mind, let’s take a look at Brocade’s implementation of TRILL.

As most of you should know, TRILL uses IS-IS on Layer 2 to identify the shortest path between switches, and load balance across those paths. Since this is happening at layer 2, not layer 3, it does away with Spanning Tree, which means more bandwidth and faster fail-over using the same number of ports, fiber paths, cables, and switches.

Of course, despite the fact that we all understand the above to be true, Brocade decided to go their own way and replace IS-IS with FSPF or Fabric Shortest Path First.

If you haven’t done much work in SAN environments, you may not be familiar with FSPF. Brocade created FSPF in 1997 to answer bandwidth concerns in Fiber Channel SANS. It has since become the standard path selection protocol in Fiber Channel fabrics.

With that understanding, let me back up and rephrase. As TRILL utilizing IS-IS was being developed by the IETF, Brocade a member of the IETF, decided to implement their own version of TRILL utilizing FSPF.

Brocade along with Cisco are both offenders. They both claim to be working with the IETF, yet at the same time both have released competitors to TRILL. Are we to believe that Brocade worked to make TRILL the best possible solution at the same time that they were creating a competitor to it? What about Cisco and FabricPath?

Both companies claim that their solution “extends” TRILL with additional features.

Were those “extended” features brought up in meetings when the TRILL standard was being discussed? Did the IETF choose to ignore those suggestions? I doubt it.

Cisco, Brocade, and most like every other vendor sat at the table the same way a poker player does during a game of texas hold ’em. No one showed their cards, but everyone watched the flop, river, and turn cards, to see what they could create with their own hands to drive the other players off the table.

Make no mistake, TRILL did not benefit from Brocade, Cisco, or any other vendor’s presence on the committee. Their involvement was for their own purposes, not the benefit of customers.

Cisco is SCARED! Why Cisco won’t release an emulator.

Greg Ferro posted on his blog another plea to Cisco to play nice and give network engineers tools for testing, verifying, and learning new technology. If you’ve missed the recent debate on the matter, it’s OK. Crawl back under that rock, you won’t miss a thing.

I generally read Greg’s posts while nodding my head like some sick bobble headed doll, with an occasional grunt in agreement. However today, my head stopped bobbing when I realized something…

Cisco is AFRAID of the virtual switch/router.

Let that sink in for a minute.

I know what you’re thinking. “They don’t have anything to be afraid of. That’s crazy talk.”  I’m sure that people said the same about Dell and HP when ESX was first announced. “They don’t have anything to worry about. No data center could ever virtualize all of their servers. That’s just crazy.” Only, it did happen. Right now I am sitting just a few hundred feet from 100 servers that would be over 500 servers if it wasn’t for vmWare. Think of the lost revenue to Dell and HP.

But, you say, “what about the Nexus 1000v”. What about it? Cisco had already lost sales because all of those virtual servers didn’t need individual switchports. That was Cisco’s way of getting some of that revenue back. It wasn’t about extending network engineer’s control into the virtual environment. It was about lost port revenue.

Imagine with me for a moment. What would happen if you could virtualize the Edge and Core layers of your network all onto a single HA cluster. (Maybe a couple of Dell or HP servers.)

Firewalls, Check
Routing, Check
IDS, Check
VPN, Check

Where is the need for 10GB, 40GB, 100GB, TRILL, or Fabric Path? What about all of the other technologies that Cisco will sell us over the next 10 years, forcing us to replace existing hardware?

Outside of the HA cluster, you would need a couple of switches for Distribution, and you would need your normal Access layer switches, but how many components of the network would be cut? Not only routers, firewalls, and switches, but adapters, redundant power supplies, wireless controllers.

It’s already been done. Look at Cisco Call Manager. A router, switch, and server that do the work of racks and racks of PBX equipment.

“But, I just want them to release it so that I can test.”

Cisco has three choices: 1. Stick fingers in their ears and hum loudly. (Current tactic) 2. Release a good virtual network platform, and wait for everyone to ask, “wait…why can’t we virtualize this for real?” 3. Release a crippled, barely working virtual platform, and then get derided for their poor product.

No matter how Cisco looks at it, they lose.

Suddenly I am asking myself. After IPv6, what is the next big thing to happen in networking? Could virtualization change networking the way it changed servers?

Is Cisco getting back on track?

Cisco’s big-man-in-charge, John Chambers, sent out an email to all employees this week, which outlined a few important things:

-Cisco has lost focus
-Cisco was caught off guard by certain movements within the Networking community (openflow, new products from other vendors, etc)
-Cisco makes it difficult for new product to make it to market
-Cisco has to focus on the core business components, rather than continuing to diversify into low margin consumer markets
-Most importantly, Cisco shareholders, employees, and customers are not happy with the current direction that Cisco has taken

The message is a great read, and gives me hope that Cisco can get back on the ball, and address some of it’s core issues. Kudos to the Cisco team for taking a hard look at where they are, and making decisions to correct their wandering trajectory. Here’s hoping they follow through!

http://blogs.cisco.com/news/message-from-john-chambers-where-cisco-is-taking-the-network/