Geek Tools – Cape Networks for Wireless Monitoring

“Wireless isn’t working!” – everyone

How many times have we heard that mantra? As wireless engineers we know all of the intricate details that are required to be in place before wireless “just works.” We often find ourselves trying to explain this to people who see wireless as magic, and us as the magicians. They don’t care about the intricacies of roaming. They don’t care about the underlying systems, many of which we rarely control. DHCP, DNS, RADIUS, and ultimately the services they are trying to connect to.

Assuming a medium to large sized corporate environment, there is likely someone (a team) responsible for the DNS, DHCP, and Radius, and that is not likely to be the same team responsible for wireless. In very large environments, the LAN team that even provides the network cable for the AP may be a different team.

Further increasing the confusion is that problems can often appear isolated, with only a small group of users experiencing the issue.

Then the troubleshooting must begin. Is it a client issue? Were drivers or firmware recently updated on the users systems? Is there a common location, time, or AP that the experience is related to? The list of questions begin to build.

I ran into this in my own network recently. Users were complaining of being unable to connect to wireless. The problems were reported from various locations slowly over a number of days. No particular client was having consistent issues, and I never saw the problem on a customers computer while they were having it. I began looking through logs and following pretty standard troubleshooting steps. Nothing came up. It was as if the problem didn’t exist, yet I was hearing about it often enough to believe that it did.

Considering that I had just returned from Mobility Field Day 2 and participated in the Cape Networks session, I had an idea. Cape Networks provided delegates with a sensor to test. I spoke with my manager at work, a very smart guy (he hired me, right?) who agreed it would be OK to test the sensor in our environment.

The secret to the Cape Networks sensor is that it IS a client. It sees what a client experiences and its entire function is to report on the user experience. It is cloud connected, with an intuitive dashboard, that makes setup and management easy, and remote troubleshooting painless. You really should watch the Cape Networks presentation!

After installing the sensor and configuring the device for our wireless network and the internal services that I wanted it to test, I walked away and forgot about it for a day.
The next morning, I logged back in, and my issue was staring me in the face. DHCP

The time to get a DHCP address was all over the map, peaking as high as 11 seconds. Problem found! Users who experienced those peaks would clearly have issues connecting; add in their own impatience, maybe turning off and on wireless, and of course they couldn’t connect.

Before and after changes were implemented to resolve the DHCP issues users experienced.

Before and after changes were made to DHCP.

What was even more important was that I now had clear metrics that I could take to my team that manages DHCP. I could point to the problem, and then after we developed and implemented a resolution, I was able to point to the same metrics as proof that our plan worked.

As you might have guessed, complaints and rumors of complaints quickly died away.

My organization has since made the decision to invest in sensors from Cape Networks. Their reporting and ease of use make for an unmistakable value. I strongly recommend that you also check them out.

As a MFD2 delegate, I did receive a free sensor from Cape Networks and various stickers and other low value (but tasty) snacks. All other expenses for MFD were covered by Tech Field Day. I was not compelled to write about Cape Networks in any way other than personal user experience. My employers decision to purchase sensors was based solely on the user experience and ease of problem resolution.

Geek Tools – Installing Spectools on WLPC Odroid

spectoolsscreenOne of the maker sessions from WLPC was setting up an Odroid for use as a network tool. It was a great session and I hope to see more of these at future WLPC’s. Once the videos are posted, you will be able to find the link here.

The first thing I wanted to try was installing Spectools on my Odroid to use with my Metageek Spectrum Analyzers. I have two Metageek Wi-Spy DBx’s and thanks to the 2017 WLPC bag, one Wi-Spy 2.4x.

The Wi-Spy 2.4x analyzer is supported in a much older version of Spectools. If you only own that one analyzer, simply run the following from the CLI:

sudo apt-get install spectools

On the other hand, if you want to use Spectools with a DBx, you must compile from the latest version. This takes a bit more work as it must be compiled from the source code. After fumbling around with it along with Jerry Olla we were both able to get it successfully installed and working.

Here are the directions which worked for both Jerry and I.

  1. Install the required prereqs:
sudo apt-get install libgtk2.0-dev libusb-dev build-essential

2. Clone the Spectools package:

git clone https://www.kismetwireless.net/spectools.git

3. Change to the Spectools directory:

cd spectools

4. Now, the fun part. The included config.guess will not recognize the Odroid. However, the distribution installed on the Odroid includes a MUCH newer version that will, so we need to copy it to the spectools directory:

cp /usr/share/misc/config.guess config.guess

5. Now we can follow the standard process to compile Spectools to operate on the Odroid.

./configure
make
make install

And with that, Spectools should now support the Metageek dbx. Install VNC, and you have an easily deployable sensor.

Up next, installing Websockets for wi-spy

A Public Apology to Fluke Networks

In February I traveled to Phoenix to attend the Wireless LAN Professionals Conference (WLPC). It was an excellent conference with a ton of useful information and resources. One of the remarkable aspects of WLPC is that there are no corporate sponsors. All conference expenses are covered by attendees, and while vendors are encouraged to include items in the conference attendee bag, they are no booths, booth babes or trolls. I am certain some attendees would rather run booth to booth grabbing tchotchkes and attempting to avoid getting their badges scanned. I find the WLPC model refreshing.

At this years conference the organizers tried something new. Once the conference was done for the day, they opened the conference rooms for vendors to host attendees. Dinner or drinks were usually provided.

It was during one of these events that I overstepped an invisible, but clearly present line of professionalism, and I recognize that I owe a public apology to Fluke Networks. During their evening session, when things became slow for a moment, I took the opportunity to ask a question. I don’t remember the conversation verbatim, but my question was something like: “When will the Mac client be released?”

A simple question right? Only, the answer I got somehow exposed some raw emotions, and those emotions fueled my responses. I managed to completely side-track their session by asking for attendee participation in straw poles:
“Raise your hand if you want a Mac version.”

I mocked their walking man pointer used during surveys as a waste of CPU resources, when all I needed was crosshairs, and I continue on ranting and raving for a few more minutes. I acted like a drunk heckler, only I can’t blame alcohol.

As soon as my rant slowed, I realized I had fueled the crowd, and as other people began to chime in, I watched them reinforce my points and I sat there feeling vindicated; feeling great about delivering a bit of honesty and a big dose of reality. Their session never got back on track, but I will say the Fluke Networks team handled it with aplomb.

I now recognize that I needed a big dose of humility in that moment, not vindication. 

I’ve thought about that discussion a lot since it happened, which is what led to this blog post. Ultimately, that was the wrong venue for the conversation that I forced on them. I sat there with a belly full of food that they had graciously provided, and I completely derailed their conversation. My apologies to the Fluke Team in attendance. My apologies to the other attendees who might have been sitting there hoping for the very product walkthrough that Fluke was providing.

My blog IS the correct venue for the discussion. My passion for technology, networking, and specifically wireless fueled the rant and I plan to outline some of my frustrations in an upcoming blog post. The response I received that fueled my rant was one of disconnect. “Why would you want that?” Again, certainly not verbatim, but that was the message. I hope to start a conversation rather than rant into the void. With that in mind, I will lay out the case, and then I will put this to rest.

GeekTools: SolarWinds Wireless Heat Maps

Ever changing environments are the biggest problem that wireless engineers face. A new site can be surveyed, and based on that, an ideal wireless design can be created for the space providing perfect signal, overlap, and SNR; the wireless engineer leaves the site, SolarWinds NPM Wireless Heat Mapsmoving on to the next assignment, and that perfect design last through the weekend. Now the engineer is located in a different state, working on a different project, and is getting calls from the customer.

“Hey, we have problems, and I need you to fix them.” the customer says.
“Ok, can you describe the problem for me?” the engineer ask. Secretly, the engineer is shocked the customer is calling for any reason other than to laude the engineers talent, foresight, and general awesomeness.
“None of our customers can connect in the waiting area.” states the customer with disapproval.

Generally, troubleshooting this type of problem is straight forward. A quick look for interferers, a check to ensure all equipment is still functioning, and a general eye for anything that has changed. As a nod to the possibility of a changed environment, a simple question is asked.

“Has anything at the site changed? the engineer queries.
“Of course not” is the answer the customer provides, voice now dripping with disappointment.

Thus, the engineer continues to dig further.

Large enterprise organizations who deploy Cisco hardware generally keep maps for each facility in Cisco Prime. Smaller organizations without the budget or time to assign to Prime can find themselves looking for a different solution.

SolarWinds has a new solution that is part of NPM 11.5 and it is worth investigating. They now offer wireless heat maps. The simplicity of setting up the heat maps makes it easy for under-staffed shops to use the tool effectively. Import the floor plan, set scale, and then drag the AP’s supplied by the Cisco WLAN Controller onto the map into the correct locations. Once the AP’s are placed, the software makes a best-guess of wireless coverage. This is a standard but flawed practice.

The issue lies in physical placement of the AP’s within their environment. The AP’s are all at ceiling height, above cube walls, water features, whiteboards, and many other sources of signal degradation. The clients are on a much lower plane, and therefor see a different footprint.

SolarWinds solves this issue by allowing an engineer to place known clients on the map, and then use those to further improve the heat map. This provides a tool that can be used to understand what is happening at standard client heights, where signal matters.

“Oh look, now I see a huge null in the coverage.” the engineer says. “Are you sure there haven’t been any changes near the AP I placed by the receptionist desk?
“Oh, that’s right. We hung the sign this weekend” says the customer.
“That large metal sign that was in the shop area last week?” ask the engineer.
“Yeah, that’s the one, We suspended it from the ceiling right over the receptionist. It looks awesome. That wouldn’t cause this problem would it?”

The engineer proceeds to bang his head against the desk with a dull thud, thud, thud.
*Names have been changed to protect the identities of those responsible*

Watch SolarWinds discussing their wireless heat maps at Network Field Day 9 here: (Heat map discussion starts at 19:40)

-I participated in Network Field Day 9 as a delegate. As part of that participation, the cost of all travel and accommodations were covered. Additionally, some companies chose to give delegates small gifts for their participation. These accommodations do not in any way constitute a requirement for coverage, good or bad. In short, I am an opinionated jerk,  I was invited despite that, and anything I write is purely my own opinion. Special thanks to Tech Field Day, for the service they provide to engineers and vendors. If you would like to be a delegate at a future event, you can learn more here.  

Hey Apple, Help Us, Help You!

When the iPhone debuted on the AT&T network, AT&T was clearly not expecting the demand that was created. They were caught off-guard by the influx of customers, and more importantly they were surprised by the data consumption of users, who had purchased a device created to consume data. Problems were reported at a ridiculous rate, and rumors abounded everywhere within the Tech blogs that Apple was threatening to take their ball phone and go home to Verizon if AT&T didn’t do something fast.

In the mean time, Apple began working on ways of optimizing the iPhones use of the carriers network, and kept pushing AT&T for improvements. It took AT&T a couple years, and a LOT of money to build their network up. Some people will argue that if the iPhone had not been made available on other carriers that AT&T would still be having issues.

Apple studies, lives and dies by user experience. They knew that a poorly performing network would reflect on their device. It was not enough to simply blame the network. If the network wasn’t available, then features of their phone weren’t available either.

With that in mind… Apple DOES NOT provide developer access to wireless API’s in IOS. Troubleshooting WLAN issues for IOS devices can only be accomplished from the infrastructure side. Without jailbreaking an iPhone, there is no way to access RSSI, SNR, or other WLAN statistics.

Which device is best for troubleshooting iPad connectivity issues on a WLAN? If you answer anything other than “another iPad”, go directly to jail, do not pass go, and do not collect $200. This is an oversight decision that Apple needs to quickly reconsider.

Apple, we are the network. Without WLAN Engineers, iPads and iPhones won’t function correctly on corporate networks. Without the proper tools, WLAN engineers cannot support IOS devices when there are issues on the WLAN. Without tools, our network problems reflect on your devices. Help US, help YOU.

Supporting Apple devices on the WLAN

Since the iPad was released, it has received a mixed welcome within Enterprise environments. While a lot companies have at least some plan to move forward with iPads, these drivers are usually coming from the business side, instead of IT. In-fact, most IT shops are being dragged into IOS support with strong reluctance.

The broad questions which are causing resistance can be summed up in one word: SUPPORT. IT departments must figure out how to support the device in multiple areas. Information integrity and control, end-user support, and connectivity support all must be dealt with. Since this is a networking blog, I want to deal with the last one; and will do so in the next two articles.

Supporting iPads on the network is more complex than connecting them to an SSID and providing login credentials. If we look at the standard iPad user in most organizations, we see a highly mobile user, users who also have laptops. Most of these users requested an iPad after having a positive experience with their company issued iPhones. That translates to a user having three wireless devices at there desk at any given time: their laptop, their iPhone and their iPad.

To understand the problem this creates, let’s look at how we survey for a wireless network. There are two considerations: coverage and capacity.

Wireless Coverage
A survey can be  based on square footage, and provide a certain RSSI from wall-to-wall. This is a perfectly acceptable way to survey if everyone has their own office. However in Cube-ville, a single AP may cover 100 desk or more. If each desk has one wireless device, you now have a physical medium (the channel or airspace) that is incapable of supporting all of the connected clients.

Wireless Capacity
The other way to perform a wireless survey is based on capacity. In a high capacity environment, the wireless spectrum, not the AP is the bottleneck. More on this later…

In a capacity based scenario, a number of desk are chosen, lets say 25. For every 25 desk, there is an AP. Those AP’s are placed based on coverage area, and in to minimize channel overlap. For the same 100 desk in Cube-ville, you now have 4 AP’s. Since there will be channel overlap, the radios are turned way down, and in general, the physical medium is now capable of handling the number of clients.

Taking this environment to the next step, each desk gets an iPhone, and a few months later, 1 in 4 request an iPad. We can safely assume that complaints will begin coming into IT about the wireless network. The AP airspace that was previously servicing 25 clients now contends with 62 per AP. Time for another wireless survey and at least twice as many AP’s!

Now we can see the problem that many companies are facing. The i-devices are here, and businesses seem to love them. The network team must begin planning and building now. I would like to make a few suggestions which might keep network teams from finding themselves behind the eight  ball.

  • Budget to begin surveying your high density environments now.
  • Develop a plan for support, complete with timelines and cost. Present this to the highest management level you can reach, so that it can be considered as the business begins planning device deployments.
  • If your company has a charge-back system for devices, be certain a cost is associated with each IOS device to support the wireless network going forward.
  • Be certain to include a survey and additional equipment as a cost of any iPad rollout projects, make certain the business can see the total cost of deploying iPads and iPhones.
  • Finally, be first in line to get an iPad if you don’t already have one. You can’t support what you don’t understand; besides, it really is a great device.

I realize that there are other options out there other than the “i” devices. However, I haven’t heard of, or seen, a single enterprise level roll out. However, these rules apply to the world of Android and Windows too. More devices per square foot equals more demand on the wireless network.