Today after the markets closed, Arista announced the acquisition of Mojo Networks. This is a very interesting development, and I am curious to see what Arista does with the technology.
Ask any wireless engineer about the relationship with vendors who make the non-standard clients on their network and you’ll likely get a range of responses from quiet sobs to yelled expletives.
Problems ranging from bad driver or firmware updates, devices which don’t follow the 802.11 standard, and long delays in problem resolution are all part of the experience.
Often we may say to a customer “These clients are causing problems and here is proof. You should look at replacing them.” While the vendor of those products are telling that same customer “Your network sucks!”
With that in mind, I want to consider a few things as we begin the KRACK Attack mitigation.
Check CERT’s Vulnerability Notes Database for the status of vendor updates. This is a pretty extensive list, and is worth following:
CERT’s Vulnerability Database
Some vendors will be VERY slow to issue patches. It is absolutely essential that we as wireless engineers who have the ability to approve devices refuse any new client deployments without the appropriate patches.
Bring the security team into the discussion, and ensure that as a united front, unpatched clients are refused!
Those who work in a sales role should warn all customers away from vendors who are not actively communicating their patch strategy, with clearly defined release dates. We should not send money to any company that doesn’t see resolving this as one of their highest priorities. Those companies should wither and die.
Many large enterprises have specific budgets for IT security related expenditures. If the budget isn’t available from teams responsible for the devices, check with the security team. They may have a budget that can be utilized.
Communicate to the vendors this week. Ask about patching schedules for KRACK. Ask to be included in weekly updates on the status until patches are released. Make it very clear that you see this as a high priority and are not willing to accept a “Maybe, eventually” patch schedule.
As a group of wireless engineers, we cannot accept anything less than appropriate patches which clearly mitigate KRACK.
In case you missed it, a couple of weeks ago I wrote about my experience testing Cape Networks solution for wireless monitoring. You can find that post here. I first learned about Cape Networks at WLPC, and was able to have a conversation with them at Mobility Field Day 2 that you can watch here.
One point that continues to impress me about Cape Networks is the ability to test much more than WiFi.
It really comes down to the strength of the dashboard and the various tests that each sensor can run. The ability to test against internal and external systems is one example.
Each sensor can test against web servers, iperf, or custom ports of your choosing.
Users can configure a test to run against predefined external websites like Adobe Creative Cloud, Microsoft Office 365, Dropbox, and others. But, the sensor can also test against custom websites, checking not just “Is it up?” but HTTP status codes and latency as well.
I’ve used this recently to help an outside vendor truly understand that “No, our network is not to blame” for the high latency their users are complaining about.
When all other external websites are seeing ~20ms latency, and your web application is averaging ~90ms over a period of weeks, guess what? YOU have a problem!
Averaging 96ms of latency. Maybe that’s why the application is always slow?
Obviously, due to the nature of these tests being performed over WiFi, latency, jitter, and packet loss are all expected to be a bit higher, especially if they are performed during times of peak WiFi utilization. However, when you have tests to compare across multiple online services, it’s easy to notice standout patterns.
One feature request I would make to Cape Networks is this: Allow test to be ran across both the LAN and WiFi connections. If we can compare across these two mediums, we may also see additional information useful in diagnosing wireless issues.
Have you found a non-WiFi use for the Cape Networks sensors? If so, tell me about them in the comments.
As a MFD2 delegate, I did receive a free sensor from Cape Networks and various stickers and other low value (but tasty) snacks. All other expenses for MFD were covered by Tech Field Day. I was not compelled to write about Cape Networks in any way other than personal user experience. My employers decision to purchase sensors was based solely on the user experience and ease of problem resolution.
Today Ekahau went public with a new device that moves them from just another industry player to leading the wireless survey tool industry.
The Ekahau Sidekick answers a lot of questions that have been plaguing those of us who regularly do wireless surveys. To understand why it is such an important move for the company, you first need to understand a few of the pain points that comes with being a wireless engineer.
Some USB3 hubs create significant RF noise that can affect wireless survey results.
Many 802.11ac USB adapters have very poor consistency between devices.
Laptop batteries are often non-removable, requiring regular recharging when used for survey work.
How reliable is a 3×3:3 USB adapter when all antennas are internal without physical separation? (Short answer, they aren’t)
A laptop with 4-5 dongles connected to it is difficult to manage. I know many people who have snapped off a USB dongle in their career.
So, how does the Sidekick do it different?
It is a dual 802.11ac radio system with 3×3:3 radio chains and the appropriate antennas.
It has a very fast dual band spectrum analyzer with incredibly high resolution.
The device has it’s own 8 hour battery, and does not draw from the laptop battery.
Nothing to snap off and break. This thing is very rugged, and easily hangs from the hip as the engineer moves around.
So the big question is cost. The Ekahau Sidekick cost $2995US. The question of annual support was not answered during the presentation. This assumes the user already has an active license for ESS or ESS Pro.
Most importantly, by separating the hardware and drivers into a unique specialized unit, the Ekahau Sidekick can be used by those of us who use MacOS just as easily as those crazy few still using Windows. Drivers and firmware are no longer a concern.
I received nothing to write this post. I am an active Ekahau user, and purchase licenses and support just like any other user. Hopefully, I will be able to convince my manager that he should fund one more purchase…
How many times have we heard that mantra? As wireless engineers we know all of the intricate details that are required to be in place before wireless “just works.” We often find ourselves trying to explain this to people who see wireless as magic, and us as the magicians. They don’t care about the intricacies of roaming. They don’t care about the underlying systems, many of which we rarely control. DHCP, DNS, RADIUS, and ultimately the services they are trying to connect to.
Assuming a medium to large sized corporate environment, there is likely someone (a team) responsible for the DNS, DHCP, and Radius, and that is not likely to be the same team responsible for wireless. In very large environments, the LAN team that even provides the network cable for the AP may be a different team.
Further increasing the confusion is that problems can often appear isolated, with only a small group of users experiencing the issue.
Then the troubleshooting must begin. Is it a client issue? Were drivers or firmware recently updated on the users systems? Is there a common location, time, or AP that the experience is related to? The list of questions begin to build.
I ran into this in my own network recently. Users were complaining of being unable to connect to wireless. The problems were reported from various locations slowly over a number of days. No particular client was having consistent issues, and I never saw the problem on a customers computer while they were having it. I began looking through logs and following pretty standard troubleshooting steps. Nothing came up. It was as if the problem didn’t exist, yet I was hearing about it often enough to believe that it did.
Considering that I had just returned from Mobility Field Day 2 and participated in the Cape Networks session, I had an idea. Cape Networks provided delegates with a sensor to test. I spoke with my manager at work, a very smart guy (he hired me, right?) who agreed it would be OK to test the sensor in our environment.
The secret to the Cape Networks sensor is that it IS a client. It sees what a client experiences and its entire function is to report on the user experience. It is cloud connected, with an intuitive dashboard, that makes setup and management easy, and remote troubleshooting painless. You really should watch the Cape Networks presentation!
After installing the sensor and configuring the device for our wireless network and the internal services that I wanted it to test, I walked away and forgot about it for a day.
The next morning, I logged back in, and my issue was staring me in the face. DHCP
The time to get a DHCP address was all over the map, peaking as high as 11 seconds. Problem found! Users who experienced those peaks would clearly have issues connecting; add in their own impatience, maybe turning off and on wireless, and of course they couldn’t connect.
Before and after changes were made to DHCP.
What was even more important was that I now had clear metrics that I could take to my team that manages DHCP. I could point to the problem, and then after we developed and implemented a resolution, I was able to point to the same metrics as proof that our plan worked.
As you might have guessed, complaints and rumors of complaints quickly died away.
As a MFD2 delegate, I did receive a free sensor from Cape Networks and various stickers and other low value (but tasty) snacks. All other expenses for MFD were covered by Tech Field Day. I was not compelled to write about Cape Networks in any way other than personal user experience. My employers decision to purchase sensors was based solely on the user experience and ease of problem resolution.
One of the maker sessions from WLPC was setting up an Odroid for use as a network tool. It was a great session and I hope to see more of these at future WLPC’s. Once the videos are posted, you will be able to find the link here.
The first thing I wanted to try was installing Spectools on my Odroid to use with my Metageek Spectrum Analyzers. I have two Metageek Wi-Spy DBx’s and thanks to the 2017 WLPC bag, one Wi-Spy 2.4x.
The Wi-Spy 2.4x analyzer is supported in a much older version of Spectools. If you only own that one analyzer, simply run the following from the CLI:
sudo apt-get install spectools
On the other hand, if you want to use Spectools with a DBx, you must compile from the latest version. This takes a bit more work as it must be compiled from the source code. After fumbling around with it along with Jerry Olla we were both able to get it successfully installed and working.
Here are the directions which worked for both Jerry and I.
4. Now, the fun part. The included config.guess will not recognize the Odroid. However, the distribution installed on the Odroid includes a MUCH newer version that will, so we need to copy it to the spectools directory:
cp /usr/share/misc/config.guess config.guess
5. Now we can follow the standard process to compile Spectools to operate on the Odroid.
./configure
make
make install
And with that, Spectools should now support the Metageek dbx. Install VNC, and you have an easily deployable sensor.
In February I traveled to Phoenix to attend the Wireless LAN Professionals Conference (WLPC). It was an excellent conference with a ton of useful information and resources. One of the remarkable aspects of WLPC is that there are no corporate sponsors. All conference expenses are covered by attendees, and while vendors are encouraged to include items in the conference attendee bag, they are no booths, booth babes or trolls. I am certain some attendees would rather run booth to booth grabbing tchotchkes and attempting to avoid getting their badges scanned. I find the WLPC model refreshing.
At this years conference the organizers tried something new. Once the conference was done for the day, they opened the conference rooms for vendors to host attendees. Dinner or drinks were usually provided.
It was during one of these events that I overstepped an invisible, but clearly present line of professionalism, and I recognize that I owe a public apology to Fluke Networks. During their evening session, when things became slow for a moment, I took the opportunity to ask a question. I don’t remember the conversation verbatim, but my question was something like: “When will the Mac client be released?”
A simple question right? Only, the answer I got somehow exposed some raw emotions, and those emotions fueled my responses. I managed to completely side-track their session by asking for attendee participation in straw poles: “Raise your hand if you want a Mac version.”
I mocked their walking man pointer used during surveys as a waste of CPU resources, when all I needed was crosshairs, and I continue on ranting and raving for a few more minutes. I acted like a drunk heckler, only I can’t blame alcohol.
As soon as my rant slowed, I realized I had fueled the crowd, and as other people began to chime in, I watched them reinforce my points and I sat there feeling vindicated; feeling great about delivering a bit of honesty and a big dose of reality. Their session never got back on track, but I will say the Fluke Networks team handled it with aplomb.
I now recognize that I needed a big dose of humility in that moment, not vindication.
I’ve thought about that discussion a lot since it happened, which is what led to this blog post. Ultimately, that was the wrong venue for the conversation that I forced on them. I sat there with a belly full of food that they had graciously provided, and I completely derailed their conversation. My apologies to the Fluke Team in attendance.My apologies to the other attendees who might have been sitting there hoping for the very product walkthrough that Fluke was providing.
My blog IS the correct venue for the discussion. My passion for technology, networking, and specifically wireless fueled the rant and I plan to outline some of my frustrations in an upcoming blog post. The response I received that fueled my rant was one of disconnect. “Why would you want that?” Again, certainly not verbatim, but that was the message. I hope to start a conversation rather than rant into the void. With that in mind, I will lay out the case, and then I will put this to rest.
Ever changing environments are the biggest problem that wireless engineers face. A new site can be surveyed, and based on that, an ideal wireless design can be created for the space providing perfect signal, overlap, and SNR; the wireless engineer leaves the site, moving on to the next assignment, and that perfect design last through the weekend. Now the engineer is located in a different state, working on a different project, and is getting calls from the customer.
“Hey, we have problems, and I need you to fix them.” the customer says.
“Ok, can you describe the problem for me?” the engineer ask. Secretly, the engineer is shocked the customer is calling for any reason other than to laude the engineers talent, foresight, and general awesomeness.
“None of our customers can connect in the waiting area.” states the customer with disapproval.
Generally, troubleshooting this type of problem is straight forward. A quick look for interferers, a check to ensure all equipment is still functioning, and a general eye for anything that has changed. As a nod to the possibility of a changed environment, a simple question is asked.
“Has anything at the site changed? the engineer queries.
“Of course not” is the answer the customer provides, voice now dripping with disappointment.
Thus, the engineer continues to dig further.
Large enterprise organizations who deploy Cisco hardware generally keep maps for each facility in Cisco Prime. Smaller organizations without the budget or time to assign to Prime can find themselves looking for a different solution.
SolarWinds has a new solution that is part of NPM 11.5 and it is worth investigating. They now offer wireless heat maps. The simplicity of setting up the heat maps makes it easy for under-staffed shops to use the tool effectively. Import the floor plan, set scale, and then drag the AP’s supplied by the Cisco WLAN Controller onto the map into the correct locations. Once the AP’s are placed, the software makes a best-guess of wireless coverage. This is a standard but flawed practice.
The issue lies in physical placement of the AP’s within their environment. The AP’s are all at ceiling height, above cube walls, water features, whiteboards, and many other sources of signal degradation. The clients are on a much lower plane, and therefor see a different footprint.
SolarWinds solves this issue by allowing an engineer to place known clients on the map, and then use those to further improve the heat map. This provides a tool that can be used to understand what is happening at standard client heights, where signal matters.
“Oh look, now I see a huge null in the coverage.” the engineer says. “Are you sure there haven’t been any changes near the AP I placed by the receptionist desk?
“Oh, that’s right. We hung the sign this weekend” says the customer.
“That large metal sign that was in the shop area last week?” ask the engineer.
“Yeah, that’s the one, We suspended it from the ceiling right over the receptionist. It looks awesome. That wouldn’t cause this problem would it?”
The engineer proceeds to bang his head against the desk with a dull thud, thud, thud.
*Names have been changed to protect the identities of those responsible*
Watch SolarWinds discussing their wireless heat maps at Network Field Day 9 here: (Heat map discussion starts at 19:40)
-I participated in Network Field Day 9 as a delegate. As part of that participation, the cost of all travel and accommodations were covered. Additionally, some companies chose to give delegates small gifts for their participation. These accommodations do not in any way constitute a requirement for coverage, good or bad. In short, I am an opinionated jerk, I was invited despite that, and anything I write is purely my own opinion. Special thanks to Tech Field Day, for the service they provide to engineers and vendors. If you would like to be a delegate at a future event, you can learn more here.
When the iPhone debuted on the AT&T network, AT&T was clearly not expecting the demand that was created. They were caught off-guard by the influx of customers, and more importantly they were surprised by the data consumption of users, who had purchased a device created to consume data. Problems were reported at a ridiculous rate, and rumors abounded everywhere within the Tech blogs that Apple was threatening to take their ball phone and go home to Verizon if AT&T didn’t do something fast.
In the mean time, Apple began working on ways of optimizing the iPhones use of the carriers network, and kept pushing AT&T for improvements. It took AT&T a couple years, and a LOT of money to build their network up. Some people will argue that if the iPhone had not been made available on other carriers that AT&T would still be having issues.
Apple studies, lives and dies by user experience. They knew that a poorly performing network would reflect on their device. It was not enough to simply blame the network. If the network wasn’t available, then features of their phone weren’t available either.
With that in mind… Apple DOES NOT provide developer access to wireless API’s in IOS. Troubleshooting WLAN issues for IOS devices can only be accomplished from the infrastructure side. Without jailbreaking an iPhone, there is no way to access RSSI, SNR, or other WLAN statistics.
Which device is best for troubleshooting iPad connectivity issues on a WLAN? If you answer anything other than “another iPad”, go directly to jail, do not pass go, and do not collect $200. This is an oversight decision that Apple needs to quickly reconsider.
Apple, we are the network. Without WLAN Engineers, iPads and iPhones won’t function correctly on corporate networks. Without the proper tools, WLAN engineers cannot support IOS devices when there are issues on the WLAN. Without tools, our network problems reflect on your devices. Help US, help YOU.
Yesterday I was called about a problem in a new warehouse where I had recently rolled out wireless. I knew what the problem was before I ever logged into the wireless LAN controller. My organization leases approximate two thirds of a large warehouse, and the remaining space is occupied by various organizations. Those various organizations are broadcasting from 29 unique AP’s all crowded into the 2.4Ghz space.
I knew the issue, because I had raised the red flag before the project had even begun. I explained the problems that would be experienced, due to the other networks, and that there was little I could do to mitigate the problem. I was able to work with the building owner to disable AP’s that existed on our side of the warehouse.
Since I had already explained the problem once, I thought I would take a different tack. I typed out a quick short story that explains the overlap problem, and sent it off. It seems the story made a positive impact and helped the manager understand the root of the problem. I thought I would share this to help bridge the gap between engineers and business managers, that need to understand wireless problems.
Bob is excited to finally be going to the XYZ annual conference in Podunck, Al. This year, the conference is bigger than ever, and he was lucky to even get a ticket. When he arrives, he learns that all sessions will be taking place in room 1, room 6, or room 11. Since he paid extra, he has two days of additional classes which he can choose to attend, and quickly fills his schedule.
On the first day, each session is taught from the stage, with the latest in PA equipment. The speaker is easily heard, and the presentation is clear and effective. Bob ask a few questions, and gets answers he both understands, and appreciates. He leaves feeling like he has learned an incredible amount in a very short period of time.
On the second day, more people have arrived at the conference, and he is surprised to find that each room has two classes going on at the same time. There is now a stage at each end of the room labeled A and B. Also, since the focus of the second day is Q&A, audience participation is paramount for the day to be effective. After breakfast, Bob gets a seat near Stage A, and while Stage B is distracting at times, he is still able to understand things that are being said. After lunch, however he isn’t so lucky. Near the middle of the auditorium noise from the Stage B often overwhelms the sound from stage A. Also, when the Stage B audience participates, he gets distracted, and forgets the question he wanted to ask the presenter on Stage A. Once he finally remembers, and gets the attention of Stage A, it is clear that they can’t understand him, so he repeats his question multiple times. Finally the presenter understands the question, but Stage B creates so much noise that Bob never hears the answer. Bob leaves that day feeling frustrated.
On the third day, everyone has arrived. Bob is horrified to learn that each room will have 4 sessions running simultaneously. The scene is pure pandemonium, and Bob does something smart…he spends the day playing golf.
devices which don’t follow the 802.11 standard, and long delays in problem resolution are all part of the experience.
One of the maker sessions from 
moving on to the next assignment, and that perfect design last through the weekend. Now the engineer is located in a different state, working on a different project, and is getting calls from the customer.
It must be the network... 