One of the most interesting areas in mobile has been the fast evolution of Augmented Reality (AR) platforms and applications utilizing mobile device’s camera in overlaying UI elements over the live video stream in real time. These applications and games create very engaging user experiences but are very difficult to test for following reasons:

1. The input for the apps come from the live video feed generated by the device’s camera and as well as from the information generated by the inbuilt accelerometer. Instead of generating clicks on the screen, the device needs to be moved around to provide input to the application under test.
2. The user experience is very sensitive to any delay, distortion and noise in the input and, as such, testing involves significant qualitative component in addition to more traditional quantitative fail/pass test results.
3. Real-time processing of moving video and overlaying 3D object on top of the video stream is very resource intensive and the majority of the processing is done on device’s dedicated hardware accelerated chips. Therefore using real devices is a must and the test results vary significantly from one device (or hardware platform) to another.

When Google approached Bitbar with their requirements of automating the AR application Testing process it was clear from the beginning that this is not going to be a standard test automation project and therefore the solution had to be a heavily customized private device cloud that takes these requirements into account. It was also clear that all this cannot be achieved by custom software alone but requires also innovative hardware arrangements to do the job.

The first task was to create a physical arrangement where all devices are positioned on a plane with the same distance from a three-dimensional ‘table’ about 2 feet (60cm) from the devices. Additionally the table needed to have different textures so that the Augmented Reality platform is able to align the superimposed 3D objects above the table surface. Finally we needed to ensure even lighting conditions on and around the table so that shadows or dark areas do not create interference to the testing process. This was achieved by attaching wide angle LED-lights pointing the table from four directions creating an even light blanket.

Second requirement was much more challenging: To be able to calibrate the AR application the automation has to move the devices for some time to calibrate the video and accelerometer input together. This has to be done every time when the application is re-installed (in this case every time). To make the setup more complex, the devices have to be stationary when the actual tests are executed. To solve this we mounted the devices on oscillating planes and used a custom built controller to oscillate the planes over API calls. This way the customer was able to start and stop the oscillation directly from their test scripts and have full control of the whole automated test flow.

The tests themselves are standard Server Side Appium test executed simultaneously on all (Android and iOS) devices to ensure comparable test results. The output of the tests is traditional pass/fail information, detailed video of the execution as well as detailed metrics of the usage profile of critical hardware metrics (CPU, GPU, Memory).

As it can be seen from above only the sky is limit when it comes to the customization on Bitbar Private Cloud setups. We have helped to automate testing of a mobile application that interacts with wearables, wifi access points, fire alarms, intelligent fridges, automotive systems etc and always we have been able to find creative solutions to bring our customers’ test automation to the next level.