Real-world feedback with a real product can often be critical to understanding how people are using your product.
Wireless devices and networks need to function properly upon deployment, even at times of high traffic and limited network resources. Testing Wi-Fi devices in a lab under controlled conditions lets you see if a device conforms to standards and lets you measure physical properties such as transmit power and bit-error-ratio. Lab testing also lets you examine higher-level communications protocols and lets you see how a device or network recovers from errors.
Lab testing can, to some degree, emulate how a device or system will behave once deployed. Best of all, lab testing is repeatable and can be automated. But, it’s no substitute for live testing at the network level. Unfortunately, live testing such as in a stadium can be difficult and expensive, but it provides valuable data on how a connected device will function and can reveal network cold spots that need better service.
Lab vs. live
Live implementations can have many moving pieces. High costs can occur because you need access to test venues or you need to procure many test samples. Plus, live testing can be time consuming, but it has benefits. In-lab testing generally requires fewer people and can be repeated as many times as needed once developed. Table 1 highlights the pros and cons of both approaches.
|Streamlined process for time-consuming tasks||Difficult to implement human interaction feedback|
|Handles repetitive tasks easily||Ineffective at smaller single test passes|
|Eliminates many common manual testing errors||Doesn’t ensure quality even for passing sections|
|Can be used for load and performance testing||Requires maintenance and development effort for every change|
|Excels in getting as close to release results||Slower than lab testing|
|On-site or real-world testing can find many “uncommon” bugs||Difficult to re-test or confirm some results|
|UI testing of states and interactive elements||Often repetitive and can allow for human errors|
The first step in making the decision about which method to use requires an understanding of the functionality needed. When developing an automated test process, we’ve often found need to develop a simulated environment to complement the automation process.
For these tests, we set up lab test for testing networks that uses an automation framework running on a dedicated laptop under Linux. We used Selenium to run automated data scripts. This flow, as shown in Figure 1, lets us test devices directly connected to a tester and as well as over a wireless network link.
The automated setup used some data gained from an on-site test. After setup, we included the access points that were used on-site along with our inventory of devices and the help of Selenium and Jmeter—an open-source app for network load testing—we tested bandwidth limits to develop a general idea on the capability of each access point, such as the number of simultaneous connections and general performance under heavy load. While the data gathered in the lab was useful, they did not factor in the uncertainty and various building materials that we encountered in the field.
In this situation, our internal lab tests led us to live testing due to the size of the venue. Attempting to validate the size requirements of a 15-acre location with more than 15,000 access points. With the knowledge of each access point, we were able to build lab capability that provided an easier path to make recommendations on the final requirements. We did so by identifying how effectively each device performed in an environment that was completely controlled versus the data we collected during a real world evaluation.
[Continue reading on EDN US: Live environments]