KRACK Attack Mitigation – A Call to Arms!

Ask any wireless engineer about the relationship with vendors who make the non-standard clients on their network and you’ll likely get a range of responses from quiet sobs to yelled expletives.

Problems ranging from bad driver or firmware updates, KRACKdevices which don’t follow the 802.11 standard, and long delays in problem resolution are all part of the experience.

Often we may say to a customer “These clients are causing problems and here is proof. You should look at replacing them.” While the vendor of those products are telling that same customer “Your network sucks!”

With that in mind, I want to consider a few things as we begin the KRACK Attack mitigation.

  • Check CERT’s Vulnerability Notes Database for the status of vendor updates. This is a pretty extensive list, and is worth following:
    CERT’s Vulnerability Database
  • Some vendors will be VERY slow to issue patches. It is absolutely essential that we as wireless engineers who have the ability to approve devices refuse any new client deployments without the appropriate patches.
    Bring the security team into the discussion, and ensure that as a united front, unpatched clients are refused!
    Those who work in a sales role should warn all customers away from vendors who are not actively communicating their patch strategy, with clearly defined release dates. We should not send money to any company that doesn’t see resolving this as one of their highest priorities. Those companies should wither and die.
  • Many large enterprises have specific budgets for IT security related expenditures. If the budget isn’t available from teams responsible for the devices, check with the security team. They may have a budget that can be utilized.
  • Communicate to the vendors this week. Ask about patching schedules for KRACK. Ask to be included in weekly updates on the status until patches are released. Make it very clear that you see this as a high priority and are not willing to accept a “Maybe, eventually” patch schedule.

As a group of wireless engineers, we cannot accept anything less than appropriate patches which clearly mitigate KRACK.

Advertisements

Geek Tools – Cape Networks for more than just wireless

In case you missed it, a couple of weeks ago I wrote about my experience testing Cape Networks solution for wireless monitoring. You can find that post here. I first learned about Cape Networks at WLPC, and was able to have a conversation with them at Mobility Field Day 2 that you can watch here.

One point that continues to impress me about Cape Networks is the ability to test much more than WiFi.

It really comes down to the strength of the dashboard and the various tests that each sensor can run. The ability to test against internal and external systems is one example.

Screen Shot 2017-09-15 at 11.57.05 AM

Each sensor can test against web servers, iperf, or custom ports of your choosing.

Users can configure a test to run against predefined external websites like Adobe Creative Cloud, Microsoft Office 365, Dropbox, and others. But, the sensor can also test against custom websites, checking not just “Is it up?” but HTTP status codes and latency as well.

I’ve used this recently to help an outside vendor truly understand that “No, our network is not to blame” for the high latency their users are complaining about.

When all other external websites are seeing ~20ms latency, and your web application is averaging ~90ms over a period of weeks, guess what? YOU have a problem!

Screen Shot 2017-09-15 at 12.06.02 PM

Averaging 96ms of latency. Maybe that’s why the application is always slow?

Obviously, due to the nature of these tests being performed over WiFi, latency, jitter, and packet loss are all expected to be a bit higher, especially if they are performed during times of peak WiFi utilization. However, when you have tests to compare across multiple online services, it’s easy to notice standout patterns.

One feature request I would make to Cape Networks is this: Allow test to be ran across both the LAN and WiFi connections. If we can compare across these two mediums, we may also see additional information useful in diagnosing wireless issues.

Have you found a non-WiFi use for the Cape Networks sensors? If so, tell me about them in the comments.

As a MFD2 delegate, I did receive a free sensor from Cape Networks and various stickers and other low value (but tasty) snacks. All other expenses for MFD were covered by Tech Field Day. I was not compelled to write about Cape Networks in any way other than personal user experience. My employers decision to purchase sensors was based solely on the user experience and ease of problem resolution.

Geek Tools: Ekahau’s New Sidekick

Today Ekahau went public with a new device that moves them from just another industry player to leading the wireless survey tool industry.

The Ekahau Sidekick answers a lot of questions that have been plaguing those of us who regularly do wireless surveys. To understand why it is such an important move for the company, you first need to understand a few of the pain points that comes with being a wireless engineer.

  • Some USB3 hubs create significant RF noise that can affect wireless survey results.
  • Many 802.11ac USB adapters have very poor consistency between devices.
  • Laptop batteries are often non-removable, requiring regular recharging when used for survey work.
  • How reliable is a 3×3:3 USB adapter when all antennas are internal without physical separation? (Short answer, they aren’t)
  • A laptop with 4-5 dongles connected to it is difficult to manage. I know many people who have snapped off a USB dongle in their career.

So, how does the Sidekick do it different?

  • It is a dual 802.11ac radio system with 3×3:3 radio chains and the appropriate antennas.
  • It has a very fast dual band spectrum analyzer with incredibly high resolution.
  • The device has it’s own 8 hour battery, and does not draw from the laptop battery.
  • Nothing to snap off and break. This thing is very rugged, and easily hangs from the hip as the engineer moves around.

So the big question is cost. The Ekahau Sidekick cost $2995US. The question of annual support was not answered during the presentation. This assumes the user already has an active license for ESS or ESS Pro.

Most importantly, by separating the hardware and drivers into a unique specialized unit, the Ekahau Sidekick can be used by those of us who use MacOS just as easily as those crazy few still using Windows. Drivers and firmware are no longer a concern.

Well done Ekahau!

You can find out more at: https://www.ekahau.com/sidekick

 

I received nothing to write this post. I am an active Ekahau user, and purchase licenses and support just like any other user. Hopefully, I will be able to convince my manager that he should fund one more purchase…

Geek Tools – Cape Networks for Wireless Monitoring

“Wireless isn’t working!” – everyone

How many times have we heard that mantra? As wireless engineers we know all of the intricate details that are required to be in place before wireless “just works.” We often find ourselves trying to explain this to people who see wireless as magic, and us as the magicians. They don’t care about the intricacies of roaming. They don’t care about the underlying systems, many of which we rarely control. DHCP, DNS, RADIUS, and ultimately the services they are trying to connect to.

Assuming a medium to large sized corporate environment, there is likely someone (a team) responsible for the DNS, DHCP, and Radius, and that is not likely to be the same team responsible for wireless. In very large environments, the LAN team that even provides the network cable for the AP may be a different team.

Further increasing the confusion is that problems can often appear isolated, with only a small group of users experiencing the issue.

Then the troubleshooting must begin. Is it a client issue? Were drivers or firmware recently updated on the users systems? Is there a common location, time, or AP that the experience is related to? The list of questions begin to build.

I ran into this in my own network recently. Users were complaining of being unable to connect to wireless. The problems were reported from various locations slowly over a number of days. No particular client was having consistent issues, and I never saw the problem on a customers computer while they were having it. I began looking through logs and following pretty standard troubleshooting steps. Nothing came up. It was as if the problem didn’t exist, yet I was hearing about it often enough to believe that it did.

Considering that I had just returned from Mobility Field Day 2 and participated in the Cape Networks session, I had an idea. Cape Networks provided delegates with a sensor to test. I spoke with my manager at work, a very smart guy (he hired me, right?) who agreed it would be OK to test the sensor in our environment.

The secret to the Cape Networks sensor is that it IS a client. It sees what a client experiences and its entire function is to report on the user experience. It is cloud connected, with an intuitive dashboard, that makes setup and management easy, and remote troubleshooting painless. You really should watch the Cape Networks presentation!

After installing the sensor and configuring the device for our wireless network and the internal services that I wanted it to test, I walked away and forgot about it for a day.
The next morning, I logged back in, and my issue was staring me in the face. DHCP

The time to get a DHCP address was all over the map, peaking as high as 11 seconds. Problem found! Users who experienced those peaks would clearly have issues connecting; add in their own impatience, maybe turning off and on wireless, and of course they couldn’t connect.

Before and after changes were implemented to resolve the DHCP issues users experienced.

Before and after changes were made to DHCP.

What was even more important was that I now had clear metrics that I could take to my team that manages DHCP. I could point to the problem, and then after we developed and implemented a resolution, I was able to point to the same metrics as proof that our plan worked.

As you might have guessed, complaints and rumors of complaints quickly died away.

My organization has since made the decision to invest in sensors from Cape Networks. Their reporting and ease of use make for an unmistakable value. I strongly recommend that you also check them out.

As a MFD2 delegate, I did receive a free sensor from Cape Networks and various stickers and other low value (but tasty) snacks. All other expenses for MFD were covered by Tech Field Day. I was not compelled to write about Cape Networks in any way other than personal user experience. My employers decision to purchase sensors was based solely on the user experience and ease of problem resolution.

Geek Tools – Installing Spectools on WLPC Odroid

spectoolsscreenOne of the maker sessions from WLPC was setting up an Odroid for use as a network tool. It was a great session and I hope to see more of these at future WLPC’s. Once the videos are posted, you will be able to find the link here.

The first thing I wanted to try was installing Spectools on my Odroid to use with my Metageek Spectrum Analyzers. I have two Metageek Wi-Spy DBx’s and thanks to the 2017 WLPC bag, one Wi-Spy 2.4x.

The Wi-Spy 2.4x analyzer is supported in a much older version of Spectools. If you only own that one analyzer, simply run the following from the CLI:

sudo apt-get install spectools

On the other hand, if you want to use Spectools with a DBx, you must compile from the latest version. This takes a bit more work as it must be compiled from the source code. After fumbling around with it along with Jerry Olla we were both able to get it successfully installed and working.

Here are the directions which worked for both Jerry and I.

  1. Install the required prereqs:
sudo apt-get install libgtk2.0-dev libusb-dev build-essential

2. Clone the Spectools package:

git clone https://www.kismetwireless.net/spectools.git

3. Change to the Spectools directory:

cd spectools

4. Now, the fun part. The included config.guess will not recognize the Odroid. However, the distribution installed on the Odroid includes a MUCH newer version that will, so we need to copy it to the spectools directory:

cp /usr/share/misc/config.guess config.guess

5. Now we can follow the standard process to compile Spectools to operate on the Odroid.

./configure
make
make install

And with that, Spectools should now support the Metageek dbx. Install VNC, and you have an easily deployable sensor.

Up next, installing Websockets for wi-spy

A Public Apology to Fluke Networks

In February I traveled to Phoenix to attend the Wireless LAN Professionals Conference (WLPC). It was an excellent conference with a ton of useful information and resources. One of the remarkable aspects of WLPC is that there are no corporate sponsors. All conference expenses are covered by attendees, and while vendors are encouraged to include items in the conference attendee bag, they are no booths, booth babes or trolls. I am certain some attendees would rather run booth to booth grabbing tchotchkes and attempting to avoid getting their badges scanned. I find the WLPC model refreshing.

At this years conference the organizers tried something new. Once the conference was done for the day, they opened the conference rooms for vendors to host attendees. Dinner or drinks were usually provided.

It was during one of these events that I overstepped an invisible, but clearly present line of professionalism, and I recognize that I owe a public apology to Fluke Networks. During their evening session, when things became slow for a moment, I took the opportunity to ask a question. I don’t remember the conversation verbatim, but my question was something like: “When will the Mac client be released?”

A simple question right? Only, the answer I got somehow exposed some raw emotions, and those emotions fueled my responses. I managed to completely side-track their session by asking for attendee participation in straw poles:
“Raise your hand if you want a Mac version.”

I mocked their walking man pointer used during surveys as a waste of CPU resources, when all I needed was crosshairs, and I continue on ranting and raving for a few more minutes. I acted like a drunk heckler, only I can’t blame alcohol.

As soon as my rant slowed, I realized I had fueled the crowd, and as other people began to chime in, I watched them reinforce my points and I sat there feeling vindicated; feeling great about delivering a bit of honesty and a big dose of reality. Their session never got back on track, but I will say the Fluke Networks team handled it with aplomb.

I now recognize that I needed a big dose of humility in that moment, not vindication. 

I’ve thought about that discussion a lot since it happened, which is what led to this blog post. Ultimately, that was the wrong venue for the conversation that I forced on them. I sat there with a belly full of food that they had graciously provided, and I completely derailed their conversation. My apologies to the Fluke Team in attendance. My apologies to the other attendees who might have been sitting there hoping for the very product walkthrough that Fluke was providing.

My blog IS the correct venue for the discussion. My passion for technology, networking, and specifically wireless fueled the rant and I plan to outline some of my frustrations in an upcoming blog post. The response I received that fueled my rant was one of disconnect. “Why would you want that?” Again, certainly not verbatim, but that was the message. I hope to start a conversation rather than rant into the void. With that in mind, I will lay out the case, and then I will put this to rest.

GeekTools: SolarWinds Wireless Heat Maps

Ever changing environments are the biggest problem that wireless engineers face. A new site can be surveyed, and based on that, an ideal wireless design can be created for the space providing perfect signal, overlap, and SNR; the wireless engineer leaves the site, SolarWinds NPM Wireless Heat Mapsmoving on to the next assignment, and that perfect design last through the weekend. Now the engineer is located in a different state, working on a different project, and is getting calls from the customer.

“Hey, we have problems, and I need you to fix them.” the customer says.
“Ok, can you describe the problem for me?” the engineer ask. Secretly, the engineer is shocked the customer is calling for any reason other than to laude the engineers talent, foresight, and general awesomeness.
“None of our customers can connect in the waiting area.” states the customer with disapproval.

Generally, troubleshooting this type of problem is straight forward. A quick look for interferers, a check to ensure all equipment is still functioning, and a general eye for anything that has changed. As a nod to the possibility of a changed environment, a simple question is asked.

“Has anything at the site changed? the engineer queries.
“Of course not” is the answer the customer provides, voice now dripping with disappointment.

Thus, the engineer continues to dig further.

Large enterprise organizations who deploy Cisco hardware generally keep maps for each facility in Cisco Prime. Smaller organizations without the budget or time to assign to Prime can find themselves looking for a different solution.

SolarWinds has a new solution that is part of NPM 11.5 and it is worth investigating. They now offer wireless heat maps. The simplicity of setting up the heat maps makes it easy for under-staffed shops to use the tool effectively. Import the floor plan, set scale, and then drag the AP’s supplied by the Cisco WLAN Controller onto the map into the correct locations. Once the AP’s are placed, the software makes a best-guess of wireless coverage. This is a standard but flawed practice.

The issue lies in physical placement of the AP’s within their environment. The AP’s are all at ceiling height, above cube walls, water features, whiteboards, and many other sources of signal degradation. The clients are on a much lower plane, and therefor see a different footprint.

SolarWinds solves this issue by allowing an engineer to place known clients on the map, and then use those to further improve the heat map. This provides a tool that can be used to understand what is happening at standard client heights, where signal matters.

“Oh look, now I see a huge null in the coverage.” the engineer says. “Are you sure there haven’t been any changes near the AP I placed by the receptionist desk?
“Oh, that’s right. We hung the sign this weekend” says the customer.
“That large metal sign that was in the shop area last week?” ask the engineer.
“Yeah, that’s the one, We suspended it from the ceiling right over the receptionist. It looks awesome. That wouldn’t cause this problem would it?”

The engineer proceeds to bang his head against the desk with a dull thud, thud, thud.
*Names have been changed to protect the identities of those responsible*

Watch SolarWinds discussing their wireless heat maps at Network Field Day 9 here: (Heat map discussion starts at 19:40)

-I participated in Network Field Day 9 as a delegate. As part of that participation, the cost of all travel and accommodations were covered. Additionally, some companies chose to give delegates small gifts for their participation. These accommodations do not in any way constitute a requirement for coverage, good or bad. In short, I am an opinionated jerk,  I was invited despite that, and anything I write is purely my own opinion. Special thanks to Tech Field Day, for the service they provide to engineers and vendors. If you would like to be a delegate at a future event, you can learn more here.