KRACK Attack Mitigation – A Call to Arms!

Ask any wireless engineer about the relationship with vendors who make the non-standard clients on their network and you’ll likely get a range of responses from quiet sobs to yelled expletives.

Problems ranging from bad driver or firmware updates, KRACKdevices which don’t follow the 802.11 standard, and long delays in problem resolution are all part of the experience.

Often we may say to a customer “These clients are causing problems and here is proof. You should look at replacing them.” While the vendor of those products are telling that same customer “Your network sucks!”

With that in mind, I want to consider a few things as we begin the KRACK Attack mitigation.

  • Check CERT’s Vulnerability Notes Database for the status of vendor updates. This is a pretty extensive list, and is worth following:
    CERT’s Vulnerability Database
  • Some vendors will be VERY slow to issue patches. It is absolutely essential that we as wireless engineers who have the ability to approve devices refuse any new client deployments without the appropriate patches.
    Bring the security team into the discussion, and ensure that as a united front, unpatched clients are refused!
    Those who work in a sales role should warn all customers away from vendors who are not actively communicating their patch strategy, with clearly defined release dates. We should not send money to any company that doesn’t see resolving this as one of their highest priorities. Those companies should wither and die.
  • Many large enterprises have specific budgets for IT security related expenditures. If the budget isn’t available from teams responsible for the devices, check with the security team. They may have a budget that can be utilized.
  • Communicate to the vendors this week. Ask about patching schedules for KRACK. Ask to be included in weekly updates on the status until patches are released. Make it very clear that you see this as a high priority and are not willing to accept a “Maybe, eventually” patch schedule.

As a group of wireless engineers, we cannot accept anything less than appropriate patches which clearly mitigate KRACK.

Advertisements

Geek Tools – Cape Networks for more than just wireless

In case you missed it, a couple of weeks ago I wrote about my experience testing Cape Networks solution for wireless monitoring. You can find that post here. I first learned about Cape Networks at WLPC, and was able to have a conversation with them at Mobility Field Day 2 that you can watch here.

One point that continues to impress me about Cape Networks is the ability to test much more than WiFi.

It really comes down to the strength of the dashboard and the various tests that each sensor can run. The ability to test against internal and external systems is one example.

Screen Shot 2017-09-15 at 11.57.05 AM

Each sensor can test against web servers, iperf, or custom ports of your choosing.

Users can configure a test to run against predefined external websites like Adobe Creative Cloud, Microsoft Office 365, Dropbox, and others. But, the sensor can also test against custom websites, checking not just “Is it up?” but HTTP status codes and latency as well.

I’ve used this recently to help an outside vendor truly understand that “No, our network is not to blame” for the high latency their users are complaining about.

When all other external websites are seeing ~20ms latency, and your web application is averaging ~90ms over a period of weeks, guess what? YOU have a problem!

Screen Shot 2017-09-15 at 12.06.02 PM

Averaging 96ms of latency. Maybe that’s why the application is always slow?

Obviously, due to the nature of these tests being performed over WiFi, latency, jitter, and packet loss are all expected to be a bit higher, especially if they are performed during times of peak WiFi utilization. However, when you have tests to compare across multiple online services, it’s easy to notice standout patterns.

One feature request I would make to Cape Networks is this: Allow test to be ran across both the LAN and WiFi connections. If we can compare across these two mediums, we may also see additional information useful in diagnosing wireless issues.

Have you found a non-WiFi use for the Cape Networks sensors? If so, tell me about them in the comments.

As a MFD2 delegate, I did receive a free sensor from Cape Networks and various stickers and other low value (but tasty) snacks. All other expenses for MFD were covered by Tech Field Day. I was not compelled to write about Cape Networks in any way other than personal user experience. My employers decision to purchase sensors was based solely on the user experience and ease of problem resolution.

A Public Apology to Fluke Networks

In February I traveled to Phoenix to attend the Wireless LAN Professionals Conference (WLPC). It was an excellent conference with a ton of useful information and resources. One of the remarkable aspects of WLPC is that there are no corporate sponsors. All conference expenses are covered by attendees, and while vendors are encouraged to include items in the conference attendee bag, they are no booths, booth babes or trolls. I am certain some attendees would rather run booth to booth grabbing tchotchkes and attempting to avoid getting their badges scanned. I find the WLPC model refreshing.

At this years conference the organizers tried something new. Once the conference was done for the day, they opened the conference rooms for vendors to host attendees. Dinner or drinks were usually provided.

It was during one of these events that I overstepped an invisible, but clearly present line of professionalism, and I recognize that I owe a public apology to Fluke Networks. During their evening session, when things became slow for a moment, I took the opportunity to ask a question. I don’t remember the conversation verbatim, but my question was something like: “When will the Mac client be released?”

A simple question right? Only, the answer I got somehow exposed some raw emotions, and those emotions fueled my responses. I managed to completely side-track their session by asking for attendee participation in straw poles:
“Raise your hand if you want a Mac version.”

I mocked their walking man pointer used during surveys as a waste of CPU resources, when all I needed was crosshairs, and I continue on ranting and raving for a few more minutes. I acted like a drunk heckler, only I can’t blame alcohol.

As soon as my rant slowed, I realized I had fueled the crowd, and as other people began to chime in, I watched them reinforce my points and I sat there feeling vindicated; feeling great about delivering a bit of honesty and a big dose of reality. Their session never got back on track, but I will say the Fluke Networks team handled it with aplomb.

I now recognize that I needed a big dose of humility in that moment, not vindication. 

I’ve thought about that discussion a lot since it happened, which is what led to this blog post. Ultimately, that was the wrong venue for the conversation that I forced on them. I sat there with a belly full of food that they had graciously provided, and I completely derailed their conversation. My apologies to the Fluke Team in attendance. My apologies to the other attendees who might have been sitting there hoping for the very product walkthrough that Fluke was providing.

My blog IS the correct venue for the discussion. My passion for technology, networking, and specifically wireless fueled the rant and I plan to outline some of my frustrations in an upcoming blog post. The response I received that fueled my rant was one of disconnect. “Why would you want that?” Again, certainly not verbatim, but that was the message. I hope to start a conversation rather than rant into the void. With that in mind, I will lay out the case, and then I will put this to rest.

GeekTools: SolarWinds Wireless Heat Maps

Ever changing environments are the biggest problem that wireless engineers face. A new site can be surveyed, and based on that, an ideal wireless design can be created for the space providing perfect signal, overlap, and SNR; the wireless engineer leaves the site, SolarWinds NPM Wireless Heat Mapsmoving on to the next assignment, and that perfect design last through the weekend. Now the engineer is located in a different state, working on a different project, and is getting calls from the customer.

“Hey, we have problems, and I need you to fix them.” the customer says.
“Ok, can you describe the problem for me?” the engineer ask. Secretly, the engineer is shocked the customer is calling for any reason other than to laude the engineers talent, foresight, and general awesomeness.
“None of our customers can connect in the waiting area.” states the customer with disapproval.

Generally, troubleshooting this type of problem is straight forward. A quick look for interferers, a check to ensure all equipment is still functioning, and a general eye for anything that has changed. As a nod to the possibility of a changed environment, a simple question is asked.

“Has anything at the site changed? the engineer queries.
“Of course not” is the answer the customer provides, voice now dripping with disappointment.

Thus, the engineer continues to dig further.

Large enterprise organizations who deploy Cisco hardware generally keep maps for each facility in Cisco Prime. Smaller organizations without the budget or time to assign to Prime can find themselves looking for a different solution.

SolarWinds has a new solution that is part of NPM 11.5 and it is worth investigating. They now offer wireless heat maps. The simplicity of setting up the heat maps makes it easy for under-staffed shops to use the tool effectively. Import the floor plan, set scale, and then drag the AP’s supplied by the Cisco WLAN Controller onto the map into the correct locations. Once the AP’s are placed, the software makes a best-guess of wireless coverage. This is a standard but flawed practice.

The issue lies in physical placement of the AP’s within their environment. The AP’s are all at ceiling height, above cube walls, water features, whiteboards, and many other sources of signal degradation. The clients are on a much lower plane, and therefor see a different footprint.

SolarWinds solves this issue by allowing an engineer to place known clients on the map, and then use those to further improve the heat map. This provides a tool that can be used to understand what is happening at standard client heights, where signal matters.

“Oh look, now I see a huge null in the coverage.” the engineer says. “Are you sure there haven’t been any changes near the AP I placed by the receptionist desk?
“Oh, that’s right. We hung the sign this weekend” says the customer.
“That large metal sign that was in the shop area last week?” ask the engineer.
“Yeah, that’s the one, We suspended it from the ceiling right over the receptionist. It looks awesome. That wouldn’t cause this problem would it?”

The engineer proceeds to bang his head against the desk with a dull thud, thud, thud.
*Names have been changed to protect the identities of those responsible*

Watch SolarWinds discussing their wireless heat maps at Network Field Day 9 here: (Heat map discussion starts at 19:40)

-I participated in Network Field Day 9 as a delegate. As part of that participation, the cost of all travel and accommodations were covered. Additionally, some companies chose to give delegates small gifts for their participation. These accommodations do not in any way constitute a requirement for coverage, good or bad. In short, I am an opinionated jerk,  I was invited despite that, and anything I write is purely my own opinion. Special thanks to Tech Field Day, for the service they provide to engineers and vendors. If you would like to be a delegate at a future event, you can learn more here.  

Geek Tools – SSH and Telnet on OS X

Since I made the switch to a Mac in my day job, I’ve had two major frustrations. The first is the lack of Visio for OS X. The second one, was a little more major. I needed a replacement for MRemoteNG. I’ve searched for options and grown weary of reading the general post of “why would you need a specialized SSH tool, when it is built into the terminal of OS X?”

That statement is usually offered by a web developer who might have SSH connections to 3-5 servers on a daily basis. They live in a very specific world, and have a hard time understanding anything outside of that world. Feel bad for them; don’t hate them.

In the world of network engineers however, we may connect to 50 or more devices in a day, and may have logins to thousands of devices over an enterprise network. In that environment, there is a real need for the ability to bookmark devices.

After searching for options, I found one option that worked to some extent. This SSH workflow for Alfred is excellent. However, since I use a hosts file from someonewhocares.org to block a lot of advertisers and trackers, the index was never very useful.

After considering this problem from all angles, I finally had an “AH HA!” moment, and the simplicity of the solution made me equal parts giddy and disappointed that it took me so long to resolve. I created a file with a similar layout to a hosts file, in-fact I even named it hosts.txt. Each row of the file list a hostname, and an IP address. Since this file is purely text, you could add anything to each line that you wanted. 

#site1
device1 10.0.1.1 description
device2 10.0.1.2 unique protocol info
device3 10.0.1.3 more information
device4 10.0.1.4
#site2
device1 10.0.2.1
device2 10.0.2.2
device3 10.0.2.3
device4 10.0.2.4
device5 10.0.2.5
#site3
device1 10.0.3.1
#site4
device1 10.0.4.1
device2 10.0.4.2

But how does this help us manage thousands of devices you ask? It doesn’t, but grep does. If we pass a search string to grep along with the file name, all matching hosts show up. Yes it is simple, but it is useful because of that!

In my file, I created a site heading by starting the line with an octothorpe. I use this so that I can search for sites. This looks like:

grep ^# hosts.txt
#site1
#site2
#site3
#site4

I can also search for all devices at a location using a statement like:

grep ^#site2 -A6 hosts.txt
#site2
device1 10.0.2.1
device2 10.0.2.2
device3 10.0.2.3
device4 10.0.2.4
device5 10.0.2.5
#site3

In this case, I am telling it to start at “#site2” and show the next 6 lines. Since the 6th line is the next site, I know that I am seeing all of the devices from site 2.

Finally, if I know part of the hostname, I can simply search on it, and it will display.

Hopefully this gives you a better way of managing huge networks from terminal.

Cisco Live US 2014 – Engage Now!

Last year, I attended Cisco Live for the first time in my career. I went expecting to learn a lot, and I was not disappointed. You can read about my experiences here and here. If you haven’t read them, you should read them now. No, really, go read them. 

Now that you have read them, you know that you need to begin planning your social experience now. The scheduler will soon be available, and while you are considering the need of various classes, be certain that you create time slots to meet people. There is an incredible braintrust available in the social media hub. If you take the time to mingle and discuss you will be surprised at what you will learn.

I have long been a proponent of Twitter for IT professionals. If you and I have met over the last few years, and I haven’t asked about your social media interaction, I would question whether you actually met me and not a doppelgänger. If you have actually met me, I hope that my influence, no matter how small, pushed you to engage.

If you are new to social media and planning on hanging out in the Social Media Hub, let me offer a few suggestions.

  • Engage now. Don’t expect to show up to the social media hub without ever talking to any other engineer on twitter and expect to enjoy your experience. We like our jokes, our running discussions (arguments), and interacting. The social media hub is our opportunity to continue our online discussions in person. If you want a great list of engineers to follow, just check out who I follow.
  • Don’t be afraid to ask questions. We all come from a different background. Some of us are jack-of-all-trades, some of us specialize. We don’t expect anyone to be an expert in everything. We enjoy learning from each other. If you listen, and ask questions, you will learn.
  • Leave the oversized ego at home. Most of us have bigger personalities than egos. There are people in this group who know more than you. Trust me on this! If you show up with the goal of proving how smart you are, you’re going to have a bad time.
  • Don’t worship at the feet of your favorite author/personality. Yes, they will hang out with us and yes, they know an incredible amount about certain topics. Without exception though, they don’t want to be placed on a juvenile pedestal. They want to engage with other engineers. Story time:

Last year, I started a conversation with a well known author. We talked about our careers, about IT in general and the direction of technology. During these conversations, no less than 15 people approached to tell the author how great he was. The author was very happy to talk with them, and many times tried to draw the individual into our conversation. He would introduce me, mention the topic we were discussing at the moment, and made a genuine attempt to engage them in the discussion. Without fail, they thanked the author for his work, and then shyly withdrew. They were worshiping, not engaging.

  • Finally, register NOW! Register now to be certain you can attend the session that you want or need. This will also ensure that you can get an exam registered before all of the slots are filled. You can register here:

Cisco Live Registration

Cisco Live 2013 Final Thoughts

Image of JD on his bike in West Virginia

Image by Klaus Jones

I spent the last 5 days on the seat of my motorcycle driving hundreds of miles through the mountains of West Virginia. I do some of the best thinking on my motorcycle. The sound and vibrations of my pipes, driving with my whole body, leaning in and out of curves, the awareness of everything on, in, or around the road. Somehow, with all of that going on, I think A LOT.

As I continued to process everything I learned at Cisco Live, there were some thoughts that stuck out. These have very little to do with the social aspect, as I have already written about that here.

 Cisco Live Itself

1)   Why isn’t there a “lessons learned” document or post from the team who setup the wireless network? That was an incredible undertaking. I heard no complaints. I want to know what the Cisco Live Team has done over the last few years to scale the wireless network. Maybe the article is out there, but I haven’t seen it. This article wouldn’t be theory or sales, this is open communication about a real-life incredibly complex environment.

2)   Ditto on the WAN connection.

3)   As a first time, late registering attendee, I didn’t fully understand the Meet the Engineer, or the Table Topics at lunch. Now that I understand both, I will take better advantage of them next year.

4)   There is a special program for Netvets. There is a special party for CCIE’s. Why isn’t there a session on Sunday or early Monday for first timers? Make it a welcome party, initiation, meet and greet, and Q&A. I would have felt overwhelmed if it wasn’t for the great group of engineers that I hung out with at the Social Media Hub.  It would have also answered #3 above.

World of Solutions

I was surprised by the number of engineers running through the WoS chasing cheap plastic swords and other bits of junk. I liked a few of the T-Shirts, and grabbed a few of those, I picked up some buttons from Solarwinds, who clearly understands geek humor, and I avoided the rest. I realized on my ride this week, that the attendees were following the design. Run from booth to booth conquering and claiming prizes. Vendors, can I make a few suggestions?

1)   If you plan to give away shirts, make it a good design. If I like the design, I will wear it. Other engineers will see it. Conversations will be started about your company. Isn’t that the goal? If the design is bad, it will end up in the “donate” pile, as the yard work t-shirt, or used to wash cars. None of those are good for brand recognition. Special points given to geek humor and high quality shirts. If you want to guarantee that it sees the office, make it a polo shirt.

2)   Stop trying to win customers with a 5-minute pitch thrown out at the speed of sound by a mouthpiece that can’t answer questions. Your audience is technical. Do you think the audience can’t tell when the speaker is reciting words that they don’t understand?

3)   Find a way to engage potential customers. Make it easy for them to talk with a technical person, who can answer technical questions, and provide technical solutions. (Noticing a theme?)

4)   Don’t scoff at me when I refuse to provide my information for your cheap junk.

5)   Most importantly, don’t scoff when I sit through your presentation, give you my information, and then refuse your cheap junk. I am the person you are trying to reach, someone who is genuinely interested in your product, and who could easily be convinced to become a customer. I’m not there for the cheap junk, I’m there for more information about your product. If you could only answer my technical questions…

Now is a great time to register for next year!