During the recent RIPE 65 meeting, some attendees experienced problems with the wireless network. Together with these attendees we investigated the problem. Find a detailed report below.
The new wireless network SSID we used for the recent RIPE 65 meeting in Amsterdam was configured to use band steering . This is used to encourage users to prefer a 5GHz network, which has many more channels to play with. This seems to work fine with both Windows and Mac clients; unfortunately, wpa_supplicant (Linux & FreeBSD) does not support band steering. There is a Redhat bug report and a post discussing the issue on a Debian platform .
Band steering causes an access point to silently drop probe requests on the 2.4GHz network, if it knows a client is capable of 5GHz. If wpa_supplicant decides that a 2.4GHz network BSSID is the best thing to connect to, it will try to connect to it. When it times out, it retries 3 times, blacklists the Access Point (AP) and tries to connect to another one. It does not try the same BSSID over 5GHz, whereas the band steering setup assumes it will.
During the RIPE 65 meeting, there were 468 attendees and we had a peak of 500 wireless clients on the wireless network. 7% of these were Linux users who could potentially have had problems. In the end only 7 attendees reported problems with their wireless connectivity.
SSID without band steering
In an effort to solve these issues, we set up a second SSID named "ripe65 no bandsteering" (sorry for the whitespace). This was a copy of the RIPE meeting SSID (ripemtg) with band steering turned off. We did not announce the configuration of this new SSID, because the attendee assisting us in trouble shooting hit another problem: association was much quicker but handing clients over to different APs seemed to take a significant amount of time. When a client lost its connection to an AP, it first tried to connect to the lost AP three times before re-electing. The timeout between each attempt was significant. The total handover period took about 40-50 seconds.
We were unable to recreate the issue on our own hardware using live CDs of Ubuntu LTS 10.04.4 and 12.04.1. However, it would appear that the advice described in the Debian post would allow users to use band steering. Specifically using nl80211 with a recent version of wpa_suplicant and modifying the necessary dbus rule . This is not ideal as it means building external packages and updating standard configurations. So we still wanted to get the "ripe65 no bandsteering" SSID to work.
We have details of the hardware, kernel and driver used in the machine which had the issues described above. We will try and recreate this environment in the hope that we can recreate the issues observed, and have them fixed before the RIPE 66 meeting in May 2013.
While troubleshooting this issue, we also noticed other strange behaviour. One Linux machine sent a probe request for the ripemtg SSID to a specific BSSID. However, it did not receive a response for that probe; at least it did not receive one visible to wpa_supplicant. When we looked at this in the AP management software, it seemed as though more than one response had been sent, but from different BSSIDs.
Unfortunately we did not have time to investigate this issue thoroughly during the meeting. We will attempt to recreate this issue and work with the vendor in an effort to understand if this is expected behaviour and whether it is standards-compliant. Once we have something more concrete, we will provide an update.
As mentioned in the closing plenary, we bought a new hardware platform, Aerohive . We also added 802.11n support to the wireless network. We are very happy with this choice. However, as with anything new, there is always a learning curve.
The new network has a very dense deployment of APs, with roughly 60% more APs deployed than at previous meetings. At the beginning of the week we configured all APs with both 2.4GHz and 5GHz networks. However due to the narrow channel distribution of the 2.4GHz network, we noticed signal interference, even with the radio power turned to its lowest setting. In order to rectify this, we disabled the 2.4GHz network on 50% of the APs. As we were using band steering, the 2.4GHz network was fairly underutilised so this did not cause any contention.
In previous meetings we created channel maps and set the channel of each AP manually to overcome interference. The Aerohive technology supports Dynamic Frequency Selection (DFS), allowing the APs to be automatically configured with a channel that has the least amount of interference. With 802.11n DFS also allows APs to utilise the Unlicensed National Information Infrastructure (U-NII) radio band. At some point during the meeting, DFS stopped working and we noticed a few areas where all the APs were on the same channel. This resulted in packet loss and latency.
We are unsure why DFS stopped working and will follow this up with the vendor. Our theory is that there may be problems with pushing configuration updates to a large number of clients. We resolved this issue by pushing the configuration out to the APs again, but in batches of 10.
Unfortunately we were not aware of the issue with band steering until the middle of the RIPE meeting week. For future meetings, we intend to bring multiple laptops with different wireless chip-sets. We will have these machines set up to dual boot the current versions of Windows and Linux (probably Ubuntu LTS). If desired, we can also set up FreeBSD.
We noticed that attendees seemed reluctant to report issues to the RIPE NCC operations team during the RIPE meeting. We would like to encourage you all to come and talk to us if you experience any problem. We run a busy network and try to implement state-of-the-art technology. At the same time, we need to ensure everything works on a wide range of devices and operating systems. Unfortunately we do not have the resources to test all of these combinations before the meeting starts. We rely on reports provided by our users so we can try to resolve any issues. We also recognise that we have experts in our audience and we would welcome your advice. If you have any comments, advice, recommendations or criticism about this article or how we could have handled things better, please let us know.
Thank you to everyone who reported the issue. We would like to extend a special thanks to Lorenzo Colitti who reported the issue and also assisted us with trying new configurations, gathering information and general troubleshooting.
Comments are disabled on articles published more than a year ago. If you'd like to inform us of any issues, please reach out to us via the contact form here.
Chris Hills •
"Airohive" should be spelled "Aerohive". I shall look forward to watching the videos from the meeting soon.
Hide one reply
Mirjam Kühne •
Hi Chris, thanks for pointing out the typo. It is fixed now. Which meeting videos are you looking for?
Gert Doering •
The issues you describe do not match the problems I have encountered - I did successfully associate to APs (in 2.4GHz band, as my laptop's broadcom chip cannot do 5GHz) and was not able to send or receive anything to the network. Changing the AP (BSSID) made it work, roaming back to known-problematic APs made it fail again. So there was something else going on as well.
Hide one reply
john bond •
Hello Gert, Thanks for letting us know. I'll contact you for more information by email john
Sebastian Wiesinger •
Hi, as Gert said, there were other things involved too. I hat problems with my iPad/iPhone. It was connected but couldn't send/receive data (at least it seemed that way). We also use Aerohive and my colleague has this information: When using band steering you should not have multiple SSIDs configured IF one of them is not broadcasted. This seems to cause multiple issues with band steering.
Hide one reply
Menno Schepers •
Hi Sebastian, we broadcast all our SSID's during the RIPE meeting, there where no hidden SSID's. Is that what your colleague is referring to?
Ondřej Caletka •
Just for the record, I used IWL4965AGN chipset on a Gentoo Linux laptop with wpa-supplicant 1.0 and no Network Manager. I did not observe any Wi-Fi issues during the meeting. So the bandsteering issue may be connected to an older version of wpa_supplicant or to the Network Manager.
Hide one reply
john bond •
Hi Ondrej, Thanks, The issues we found where related to 0.7.3. Current versions of wpa do work however im not sure which version it was fixed in. Network manager can make this issue difficult to trouble shoot as it keeps restarting wpa_suplicant so its difficult to start a manual session. With a recent version of wpa_suplicant and NM configured correctly it should work
You should use Cisco gear. Aerohive and others is a bad choice... especially for these high-density environments.
Hide one reply
john bond •
Hello Johnny, We have used Cisco devices in the past, and we have also had issues with them. The specific problem in this case was caused by a feature called "band steering". We would have used the same feature with Cisco devices, and users would have experienced the same connectivity issues. As a general device supporting a high density environment such as the RIPE meeting, Aerohive has been more than acceptable and provided excellent service at RIPE 66. I remain unconvinced that using devices from a different vendor would have helped with this issue. However, if you have specific case studies or side-by-side comparisons of Cisco and Aerohive with regards to band steering, please do share them with us, so that we can learn from them.