Dominik Dary's Blog

Mobile WebDriver Selendroid

selendroid logo

In this blog post I would like to write again about mobile test automation. In the last months I spent a lot of my time working on a new test automation framework: Selendroid - a mobile WebDriver implementation for native and hybrid Android apps.

Selendroid is an implementation of the Selenium WebDriver JSON Wire protocol, which is about to become a W3C Standard.

Main features of selendroid are:

  • The only mobile WebDriver that supports native & hybrid apps on Android version (api 10 to 19)
  • The app under test must not be modified in order to automate it
  • Different locator types are supported to find elements
  • Gestures are supported: Advandced User Interactions API
  • Selendroid can interact with multiple Android devices (emulators or hardware devices) at the same time
  • Existing Android emulators are started automatically
  • Selendroid supports hotplugging for hardware devices
  • It can be integrated as a node into the Selenium Grid for scaling and parallel testing
  • Build in Inspector to simplify test case development.
  • Full compatibility to the WebDriver JSON Wire protocol.

In Action

I order to keep that example as simple as possible, I have created a small demo project in which the selendroid-test-app is used, the app that we use to verify that selendroid itself works fine.

More Details?

The source is hosted at github: http://github.com/selendroid.

The Documenation is available at http://selendroid.io.

Happy mobile testing!

GDG DevFest in Zürich

Yesterday and today I attended the Google Developer Group (GDG) DevFest in Zürich and I really enjoyed it! I have seen good presentations, met really nice people and I had great discussions. This afternoon I did a presentation about how to automate native Android apps using Calabash-Driver and how to leverage an existing Selenium Grid2 architecture to allow scaling and parallel testing.

Here you find my slides of today’s presentation:

For the GDG DevFest presentation I used the Greenhouse application to showcase about how to write an automated Android end-to-end test. Please keep in mind that this showcase was created mainly to demonstrate, how Android app automation looks like in practice. This means the super class of the test case is starting the Calabash-Driver Server automatically. In a scenario where the Selenium Grid would be used, this component is typically started per shell script.

The actual code of the showcase is located at Github. To get started please follow the steps described in the Greenhouse application blog post. For executing the scripted test you need first to install both apk files, that are located in the github project in the download section. After starting the emulator, the test can be executed by executing the TestNG test.

[Update 1. November 2012] The full talk was recorded and is now available on YouTube.

My colleague Tim Messerschmidt ( Developer evangelist at PayPal) attended the DevFest as well and introduced PayPal Access. If you are interested in this topic, I recommend to checkout his blog post.

How to Inspect Android Activities

For writing good automated mobile end-to-end tests I personally prefer to use e.g. the IDs of the elements. For doing this, you can have a look at the application’s source code and analyze the corresponding layout.xml file.

But sometimes, if e.g. on the screen are dynamic lists like search results displayed, it is quite helpful to inspect the dialog on the mobile device that is currently displayed. The Android platform has a tool called hierarchy viewer - which is working quite well. The full UI element tree is displayed and the properties of each element can be displayed. The drawback of this tool is, that is quite slow and on some machines with less than 4GB RAM it is unusable.

For Calabash-Android I implemented a command that inspects the current dialog and returns about each UI view component the corresponding class name, the Android ID (name, defined in the layout.xml file) and for some elements the text:

In combination using the interactive Ruby shell, which I have described in my last blog post , this is quite powerful and is speeding up the test development a lot. The command for the shell is:

performAction('inspect_current_dialog')

Speed Up the Development of Calabash-Android Tests

In one of my previous blog posts I described about how to use Calabash-Android in general. In today’s post I’m describing about how to speed up the development of automated mobile tests using Calabash-Android using and mostly configuring the interactive Ruby shell (IRB).

IRB is a shell allows the execution of Ruby commands with immediate response, experimenting in real-time. This is pretty useful if you are about to automate an Android application screen that is part of an flow and you are in the process of finding for the UI elements the right commands e.g. where sometimes it is easier to use find Elements by text and sometimes you can use the Android element names. Especially if you are using customized UI components, then sometimes this can be tricky.

To speed this process I’m using the IRB shell, that is already preconfigured in Calabash-Android. Important are the files

  • .irbrc (which configures the IRB shell) and
  • irb_android.sh (the script that is starting the irb shell)

which can be found here.

I started with this original templates and customized them:

In the file .irbrc the main change is in the last line, the command, to connect to the test server, which is running on the device / emulator. In the file irb_android.sh I added the command to start the Android instrumentation, which will start the calabash-android test server on the phone / emulator and as well the application under test (AUT). The port and as well the package of the AUT can be configured, especially making the package configurable makes it easier if you are writing tests for different apps.

The IRB shell is started using the command:

./irb_android.sh com.springsource.greenhouse.test

Now you can send calabash commands e.g. like press a button with the text “Sign In”:

performAction('press_button_with_text', 'Sign In')

If you would like to get an overview what commands are by default available, you find them on github

The above described approach is working under the assumption, that the Test.apk (which contains the Calabash-Server) and your application to test are already installed on your device/emulator. The Test.apk file is created dynamically at runtime while executing the calabash-android tests. The generated file is then located under the folder features/support/.

Calabash-ios

In today’s post I’m describing about how to setup calabash-ios to automate your functional end-to-end test for iOS applications. In my previous post I described about how to setup the Greenhouse application.If you are interested in how to automate an Android app, checkout this blogpost.

For using Calabash-ios, you need to have access to the app source code. The XCode project must modified, which means in detail, that a http server is added through linking with the the calabash.framework. The Calabash client library is sending HTTP requests to this server which is executing those commands.

Calabash-ios Architecture

To get startet github contains a very detailed description, about howto setup calabash-ios in general.

Currently there are two ways about how the setup can be done:

  • Automated: using the command calabash-ios setup
  • Manual: detailed steps are described on the wiki

I made good experience using the automated setup, but if you identify weird things in your app during test execution, I recommend to do a manual setup. For the Greenhouse iOS app I have used the manual setup, because I have seen some weird behavior.

In the first screencast I’m showing you, about how to do the automated calabash setup with the provided script:

In the second screencast, I describe all the detailed steps that needs to be done for manual the setup:

The the new target has been added to the app, you can verify in the logs that after starting the app the http server on the port 37265 is bound to. The default feature folder can be created with running the command ”calabash-ios gen”, that is containing the feature file and all the other Ruby related Cucumber classes.

The third screencast is showing the actual test execution:

Karl Krukow, the creator of Calabash-ios, has given at the CukeUp! 2012 a pretty nice presentation and live demo about advanced features (like gesture (touch) recording and how these events can be played back).

Calabash-Android

In my previous post I blogged about open source frameworks for mobile end-to-end testing and in the other past post I described about how to setup the Greenhouse sample app environment.

In this post I would like to describe how to test an Android app with Calabash-Android, which was open sourced in the beginning of March.

The basic architecture of Calabash is described in this figure, which I have taken from the Less Painful homepage: Calabash-Android Architecture From an architectural perspective there is a client, that it using Cucumber by default, that is sending command to a server, that is running on the device/emulator, and that is used to communicate with the application under test (AUT) using the Robotium framework. The communication between the client and the server is done using a socket connection sending Json commands. Going with this approach the AUT doesn’t need to be modified (e.g. server to be added) because in the Android test project this remote server is added. Using the Robotium framework is a good choice, because it is the defacto standard for automated Android end-to-end testing. From my perspective adding this extra remote component makes sense, because in the real world you probably have to create before running the test some test data that you will use in your test. Without having this remote architecture you need to add all the helpers that are communicating with the test system (using various protocols) to you test apk.

To get started to test the Greenhouse Android app, you need first to clone the Calabash-Android project from Github:

git clone git://github.com/calabash/calabash-android.git
git submodule init
git submodule update

Now you need to compile the Greenhouse Android app using the ”mvn clean install” command and copy the greenhouse.apk file from the target folder into the base folder of the Calabash Android project. If you would like to run the Greenhouse app on a device, you need to modify the the property base.url in the file dev.properties to the IP where your Greenhouse server app is running.

To configure Calabash please update the build.properties file like this:

tested.package_name=com.springsource.greenhouse
tested.main_activity=com.springsource.greenhouse.MainActivity
tested.project.apk=greenhouse.apk

Now you can execute the tests on an emulator (please make sure this is started before) running the command:

ant clean test

All details about this setup you find as well in the first screencast:

Writing tests is pretty simple, there are predefined steps. My added steps can be found in the ddary_steps.rb file. The interesting part about the test is, that the first part of the test is testing a webview (authorization of the app) and the second part is using Android native UI elements. The elements in the webview are identified using css locators.

All details about the test execution can be found in the second part of the screencast:

Adding new command to Calabash-android is pretty easy. All commands, that you can use in your Ruby code, are mapped to Actions:

sh.calaba.instrumentationbackend.actions

The command ”press_button_with_text” is implemented in the class PressButtonText. The method key() is used for mapping the class to the command name and the execute(..) method is used to talk to the Robotium solo object. New written actions are only available if they are located in this package:

sh.calaba.instrumentationbackend.actions

Test Automation - 10 (Sometimes Painful) Lessons Learned

Last Wednesday Michael Palotas and I have given together a presentation at the Swiss Testing Day 2012 in Zürich about the 10 (sometimes painful) Lessons we have learned in test automation and the presentation is about:

eBay’s European quality engineering team has broad experience implementing end-to-end test automation in different software development environments (agile, waterfall, co located , outsourced, distributed). This presentation illustrates the key lessons learned from a technical and business perspective:

  1. Flip the testing triangle
  2. Everyone knows what’s automated
  3. A tool is not a strategy
  4. Write the right tests
  5. Instant Feedback is essential
  6. Automation is software development itself
  7. Maintainability is king
  8. Speak the same language as the developers
  9. Invest into the test Infrastructure
  10. Manual Testing is still very important

The video is now in Youtube:

Setup the Greenhouse Application

In my past blog entry I listed different Open Source Frameworks that can be used for mobile testing on Android and iOS. For showcasing the different frameworks, I will use the Open Source project Greenhouse that SpringSource has created some while ago to:

Serve as a open-source, public-facing reference and driver for Spring technology, including Spring MVC, Security, Integration, and the newSpring Social and Mobile projects.

For making the demonstration easier, I have forked the corresponding projects on Github and in the following lines you find all the details you need to follow to setup everything:

SpringSource has created a guide about how to setup the development environment. The main reason for creating the fork was the data initialization, for doing this you need

  • to clone the greenhouse project and

  • to update the test-data.sql.

Running the webapp The webapp can be started using Tomcat from the command line by invoking the Tomcat 7 Maven plugin: mvn clean install t7:run The webapp can be found under: http://localhost:8080/greenhouse where you can login with the following details (in the above mentioned sql file you find other credentials as well): Username: craig@habuma.com Password: freebird

Running the Android app (Details are taken from the greenhouse-android wiki) Build and Run the Greenhouse for Android app: $ cd greenhouse-android $ mvn clean install If your favorite Android emulator is started: $ mvn android:deploy

In this screencast I describe how the greenhouse environment is setup:

Open Source Mobile Test Automation Frameworks

The mobile market is changing. In the past a lot of mobile solutions has been created and a lot of them has been used to gather experience in the mobile field. Now more and more companies like car manufacturers or banks (PayPal, Barclays)are creating mobile solutions that really have an impact on their main business. If you look e.g. at eBay’s mobile facts, the numbers are just impressing:

  • 2011 there was a transation volumne of 5 billion USD worldwide
  • 1 million Listings every weeks are done via mobile
  • 176 Number of dollars spent every second through mobile purchases

From a test automation perspective, I’m looking into this area from a functional end-to-end testing perspective, there are different open source tools available to write automated tests for iOS and Android:

Overview about Open Source Test Automation Frameworks:

Framework Native Platforms Mobile Web Homepage Emulator Device
Nativedriver iOS and Android - http://goo.gl/ZnhnN X X
Calabash iOS and Android - http://goo.gl/F52W4 X X
MonkeyTalk iOS and Android - http://goo.gl/JSrIt X X
Frank iOS - http://goo.gl/1dmkA X
Robotium Android - http://goo.gl/OiAc9 X X
KIF iOS - http://goo.gl/oBTHH X
Zuchini iOS - http://goo.gl/umvnY X X
Selenium - iOS and Android http://goo.gl/RIqj6 X X

The list shows how many frameworks are available and all of them have their benefits. I think it is interesting that currently it looks like that there is no ‘killer framework’ out there like ‘Selenium’ for the web with its infrastructure component Grid2. In the upcoming months I will write some detailed blog posts about how to use the different frameworks in practice.

In the list above I probably have overseen a framework, so please, add the missing one to the comments and I will add it to the list.

Camholder 0 2 1

2012-10-20-gdg-devfest-in-zurich.markdown— date: ‘2012-03-10 13:00:04’ layout: post slug: camholder-0-2-1 status: publish title: Camholder-0.2.1 wordpress_id: ‘32’ categories:

- mobile testing

Via Twitter I found the camholder (http://camholder.org), which can be used to

… present your mobile content right on your device – with all it’s sensors! The HD camera and the (optional) Mac Software are proven to work on stage for your best live performance.

So I ordered one and some days ago the Camholder was delivered and I’m really happy that I ordered one. I’m planning to use the CamHolder in presentation and as well during the day to day work because the team I’m part of is distributed around Europe.

Picture of the CamHolder.

In this screencast is Heiko Behrens explaining the details about the CamHolder:

The CamHolder is working great, each detail like the “mounting” of the mobile device to the CamHolder is done in an awesome way. The mobile devices are fixed on the sticky ground. The clue is that the device can be easily removed and a new one can be attached easily during a live session. The free available CamHolder software is working great and has exactly the feature that I have expected.