Functional testing Python PySide PyQt applications with TDriver (Qt Testability framework)

About

You’ve written an application in Python using PySide and Qt.  You want to automate testing.  For example, you are using agile development and want to test all features of your app after you make any change to the app.  You want to test at the functional level, that is, from a user’s perspective.  What tools do you use?

This blog posting is a true log: a sequence of actions that I took in exploring the above question. You probably don’t want to follow it exactly; there is much thrashing.  It is a ramble in dependency weeds.

Exploring frameworks for functional testing

A little search finds the usual suspects:

• Storytext
• Squish
• TestComplete
• hooq
• TDriver

Criteria for selecting a framework for functional testing

• cost
• adaptability (open sourcedness)
• script writing versus recording
• cross platform support

My evaluation of frameworks for functional testing Qt apps

Storytext is open source and free but doesn’t support Qt.  It is a recorder, and does have a novel idea of adding a layer of abstraction between tests and the SUT widgets: if your GUI changes, you may not need to re-record  tests.  (I actually started to adapt StoryText to Qt, but didn’t get very far. I decided that an off-the-shelf framework might do 90% of what I needed.)

Squish is probably the best bet (if you can afford it), since it is commercially supported.

TestComplete seems to only run on Windows.

hooq is open source, but seems quiescent and has little documentation.  A few blogs  document experience with it.  It is a recorder, so you don’t need to write test scripts.

TDriver is open source and was originally supported by Nokia and partners, but is now more or less orphaned.  It does have some old documentation.  It is not a recorder, but uses test scripts written in Ruby.  There are a few blogs, such as this one about its use at Canonical to automate testing of Ubuntu apps.  However, it is not apparent that Canonical is still using it.   That seems strange; it seems like Ubuntu still lets contributors and (early adopting users) do most functional testing, at least for applications.

Testing whether a Python app  can use the Qt testability plugin

Again, note that this is not necessarily doing things in the preferred order.

When a Qt app starts, you pass it command line args (also called options).  If one of the args is ‘-testability’ then Qt tries to load a plugin (a dynamically loaded library) to support testing.

>./pensool.py -testability

yielded the error message ‘Library qttestability load failed’ on the console.

Building the agent

I read these instructions for installing TDriver for Meego.   There, you find instructions for installing TDriver from repositories.  However, it points to repositories for Ubuntu 10.10.  These are probably dated.  Meego itself seems to be an orphaned, unsupported project.  So I decide to follow the instructions for “building on other platforms”, that is, build from source for newer platforms.  Alternatively, there are other instructions.

I think the missing library is part of the so-called agent.  The architecture of the Qt testability framework allows for the test runner to be on a host and for the system under test (the SUT) to be for example a cell phone, and the two commuicate via an agent.  Here I am running everything on the same machine, but I still need the agent.  To build the latest version of the agent:

git clone http://git.gitorious.org/tdriver/agent_qt.git
cd agent_qt 
qmake -r CONFIG+=no_mobility 
make
sudo make install

(Without the ‘no_mobility’ you will get an error about ‘mobility API.’)

I also got an error about an include file ‘X11/extensions/XTest.h’.

Getting the XTest package

This is about the X11 windowing system on Linux.  One extension called XTest supports sending X events to an app for testing.  The extension is in a package ‘libxtst-dev’.  Note the spelling is different.  Note that ‘-dev’ as always means it include source and header files for compiling other programs with it.   I found and installed the package using the Ubuntu Software Center app (or > sudo apt-get install libxtst-dev’ would work.)

Now, the build of the agent works, and starting my SUT works without an error about missing library qttestability.

Architecture of TDriver

You start three components:

• your app with the -testability option
• the agent ( a server to a test runner or the Visualizer) : > qttasserver
• a test runner or Visualizer: > tdriver_visualizer

You can start them in separate consoles (or all in the same console with the ‘&’ suffix to start them in the background, but then their printed output to the console, if any, is intermixed.)

Building the Visualizer app

The Visualizer is a graphical app that shows you the widget structure of a running Qt app.  It is not necessary for testing (TDriver test cases are written in Ruby scripts.) But I wanted to test whether the Visualizer would connect to my app.

git clone http://git.gitorious.org/tdriver/visualizer.git
cd visualizer
qmake -r 
make 
sudo make install

Trying the Visualizer app

The Visualizer is a graphical app that shows you the widget structure of a running Qt app.  It is not necessary for testing (TDriver test cases are written in Ruby scripts.) But I wanted to test whether the Visualizer would connect to my app.

The app has a different command line name:

>tdriver_visualizer

I got an error message: Could not locate TDriver parameters file: /etc/tdriver//tdriver_parameters.xml

A little research shows that probably I first need to install the Ruby components, which probably will do the configuration.

I read these instructions for installing TDriver for Meego

Installing needed Ubuntu packages

sudo apt-get install rubygems ruby1.8-dev libxslt-dev libxml2-dev

What is rubygems?

rubygems is a package that lets you download and install packages written in the Ruby language.  It is similar to Python’s disutils package, but it seems to download packages, not just install them (like >python setup.py install.)

You could also install it:

Start  the ‘Ubuntu Software Center’ application (click on the icon in the launcher at the left side of your display.)

Search for ‘rubygems’.  You should find a package named ‘rubygems’.  Select it and click on the ‘Install’ button.

Now you should have the command line application: gem.

Installing Ruby libraries

sudo gem install testability-driver testability-driver-qt-sut-plugin

This downloads, builds, and installs the libraries, written in Ruby, for TDriver.  I don’t know where gem looks for these, I assume it is not in the gitorious repository for the rest of TDriver.  I’m not sure whether these libraries are more general purpose, that is, whether they are useful for other frameworks than Qt.

If you failed to install the package libxslt-dev, you will get errors.

I also got the error “ERROR:  While generating documentation for builder-3.2.0” but I assume this is not critical, it is the documentation.

Finally a modicum of success

Now I started my app, the agent, and the visualizer, without error messages.  I futzed with the visualizer until it showed my app.  I discovered what I feared: it doesn’t show much.  It shows some of my Qt widgets and even QGraphicItems, but most of them are unnamed.   That is probably something I can change.

Also, my app implements some of its own widgets (controls) that don’t inherit from QWidget, and they are missing from the visualizer.  Maybe I need to rethink that design.

Finally, I used QContextMenus, but they are hidden and don’t show up in the visualizer.  Maybe I can still send them events.  But their action depends on the QGraphicItems they hover/pop-up on, and I don’t want to hardcode coordinates into my tests.

At best, I may be able to test part of my app: all the menubar items (e.g. File>Open).   Possibly I can test all the context menu items too.  I probably won’t be able to test some of the esoteric GUI techniques, such as hover tools.

Now to writing some test scripts…

A test script in Ruby

I hacked an example from the links above:

#!/usr/bin/env ruby
require 'tdriver'
include TDriverVerify

sut = TDriver.sut(:sut_qt)

pensool_app = sut.run(:name => '<fullpath>/pensoolQt/Foo')

pensool_app.Button(:text => '1').tap

verify { pensool_app.QLineEdit(:text => '3') }

pensool_app.close

The first goal is just to start the app successfully.  When I executed the above file, I got a warning similar “required file not found: tdriver”.  This means that Ruby is not finding its libraries (gems).  To solve this, define an environment variable in your shell session before executing the test script (or define it permanently in your .profile):

>export RUBYOPT=rubygems

(Coming from the Python world, I am surprised.  A default Ruby installation needs to be told to use rubygems?  I was under the impression that Ruby had a better library management system than Python.)

Now when I run the script, it waits a minute then I get:

/var/lib/gems/1.8/gems/testability-driver-1.5.4/lib/tdriver/base/sut/generic/behaviours/sut.rb:986:in `__wrappee_316__run’: Run failed. Failed to launch application. Exception: MobyBase::ApplicationNotAvailableError (MobyBase::BehaviourError)

So I open that file and start reading.  Apparently the test framework is trying to communicate with the SUT and matches it by application name.  I start the visualizer and see that the SUT has various attributes: applicationName=Pensool but fullName=/usr/bin/python2.7.  So I suspect that the framework is not ready for an interpreted SUT, where the ‘name’ of the executable is the interpreter (python2.7) which doesn’t match the name of the script file (Foo) which doesn’t match the name that my app shows to users (Pensool.)

Aaaargh, more later…..

 

Foo

Experimental results for tiny, geared, brushed, DC motors and small, thinfilm solar cells

TT Motor makes a tiny, brushed, geared, DC motor: TGPP06.  Firgelli Automation will sell you one model for $15.

Powerfilm makes a credit card sized, thinfilm(plastic) solar cell SP3-37.  Jameco sells them for about $4.

What happens when you connect them and expose to direct, full sun?

The TGPP06 is a family.  All models are about the size of ‘pager’ motors.  The family uses the same 3V nominal motor.  Family members all have a plastic, planetary gear set on the end of the motor.  The length of the gear set differs among the family members ( one extra planetary gear group over the preceding family member.)

• TGPP06-C:  1/26.5 gear redution, 980 rpm loaded, 1250 rpm no load, stall torque 120 g.cm
• TGPP06-D-136: 1/136 gear reduction, 190 rpm loaded, 240 rpm no load, stall torque 400 g.cm
• TGPP06-D-700: 1/700 gear reduction, 37 rpm loaded, 47 rpm no load, stall torque 800 g.cm

That’s from their data sheets.  (Their data sheets don’t  always seem consistent. I’m wondering about the torque specs, which don’t seem correct for the gear ratios.)

The Powerfilm SP3-37 is a panel (solar cells in series) putting out 3V at 22 mA at ‘operating conditions’ which is approximately full, direct sun with a suitable load (approximately at the maximum power point on the current to voltage curve?)  See their data sheet.

Experimentally when connected to a SP3-37, the TGPP06-D-700 turns at about 15 rpm in full, direct (perpendicular to the panel) sun.  Under bright overcast (the sun behind a rare cumulus cloud) the motor stopped turning.

Experimentally when connected to a SP3-37, the TGPP06-D-136  turns at about 120 rpm in full, direct (perpendicular to the panel) sun.  In full sun, I could not stop the rotation by pinching the rotor shaft with my fingers (but it is very small, and hard to pinch with your fingers.)   When held at 45 degrees to full sun, it turns more slowly, a few tens of rpm.

Flickering lights, squirrels, open neutral

This is about a household electrical problem I had.  The bottom line: if your lights flicker when your refrigerator comes on, call the power company.

The symptoms included (over the course of several weeks:)

• flickering lights when the refrigerator came on
• refrigerator PTC burned up
• microwave sounding different; microwave LED panel dim; microwave diode failed open
• cable modem died
• computer and game console crashes

At first I thought the refrigerator was failing and putting too much load on its branch.  My computer was plugged into the same branch circuit (bad idea) and so it sometimes crashed when the fridge started.  I tried plugging the computer into a different branch circuit (an outlet on a different circuit breaker.)  That didn’t help.

I talked to my neighbours since they are on the same transformer on the street. But they were not experiencing problems.

I was about to start analyzing which leg of the 220V supply my refrigerator is on.  But I also called the power company, and they promptly came.

The lineman said a squirrel had chewed through the bare neutral leg of my drop.  The drop is the wires from the transformer on the street to your house.  Usually it is three wires, two insulated and one bare aluminum.  The neutral wire carries the weight, but also is a grounded part of the electrical path.

Squirrels and other rodents like to chew on aluminum foil.  Aluminum does rust or oxidize.  Aluminum oxide is another word for carborundum or sapphire.  After a layer forms, the layer protects from further corrosion (unlike iron oxide, which doesn’t protect from deeper corrosion, forming pits.)  I am wondering now whether rodents chew aluminum oxide because it is a salt, or rodents like bare aluminum.

When the neutral is open (chewed through), you still have power.  The end of the neutral at your house is also connected to a grounding rod at your house, forming a path to the earth back to the power company (to the nearest other grounding rod on the power companies’ equipment.)  But that path might be higher resistance than the neutral wire in your drop (if your grounding rod is corroded and the soil is dry?)  As the lineman explained it, when the neutral wire of the drop is open, when a largish load (a refrigerator is a large load when it starts, but not after it is running) comes on one leg (one 110V portion of the two in a center tapped 220V transformer, that the neutral wire taps the center of), the voltage on that leg (and all the branch circuits on that leg) temporarily drops, and the voltage on the other leg (and all the other branch circuits on it) increases.  This is a problem for electronics.

The chewed open I think was also intermittent.  The lineman said it was sparking.  When the wind blew, my symptoms were worse.  And it must have been progressive: the squirrel chewed until the original half inch wire was only a thread, then finally it was just two pointed wire ends in contact unless the wind blew.

The lineman said I could file a claim for damaged equipment.  This gets into politics.  Is a squirrel an act of dog?  Should society insure me?   If there is universal health insurance, should there be universal  squirrel-damage-to-electronics insurance?

The power company has tried to sell me insurance before.  It probably covers everything including squirrels.  I think they are emphasizing protection against lightning strike damage, which surely is an act  and which they probably don’t indemnify against otherwise.

Finally, I say we don’t need the squirrel insurance, but we do need to upgrade our electrical system.  The meter is smart and communicates with the power company for billing, it should be able to detect this problem and report it to the power company proactively.  Its mostly software which means it would cost almost nothing.

 

 

Python PySide Qt Pyinstaller Apps on OSX

Ongoing notes about my experience.

Bottom line: don’t use Pyinstaller on OSX with PySide yet (but you can use it with OSX and PyQt, or Windows and PySide?)  As of this writing, Pyinstaller suffers bugs with PySide on OSX that it doesn’t suffer with PyQt.  Specifically, because PySide installs itself in /usr/lib on OSX, Pyinstaller fails to find, or finds too many, dependencies.

You’ve written an app in Python language using PySide bindings which uses Qt framework.  You develop on a Linux machine.  You want to port to other desktop platforms, including Max OSX.

Pyinstaller is a tool that bundles your app.  (There are alternative packaging tools like py2app, py2exe, macdeployqt, freeze, pyside-assistant etc. but AFAIK Pyinstaller has traction and covers more platforms.)

Bundles means puts everything needed into one file.  Also called packaging.  Note the difference in philosophy on different platforms: on Linux, a package does NOT include everything, but knows what other packages it depends on, and downloads and installs them as necessary.  On OSX, and Windows, an app bundle (package) is totally self contained (except for standard platform, system libraries?), and won’t download and install other packages.

Pyinstaller for Linux is strange in that it uses the non-Linux philosophy: it bundles everything into one package, having no dependencies.  It doesn’t make it any easier for users,  since they don’t care which philosophy is used (either SHOULD work reliably.) But Pyinstaller makes it easier for the developer, because they don’t need to worry about the two philosophies on different platforms.

Cross packaging

Pyinstaller is not a cross packager: you must be on the OS for which you are packaging, and the development environment must include all the libraries and Python packages that your app needs.  (It can be a virtual machine.  For example, you could be on a Linux box running Windows in a VirtualBox.  But your Windows virtual machine must have all the libraries and Python packages installed that your app needs.)  Pyinstaller will find those packages and bundle them up; the user will not need to recreate your steps in setting up the development environment.

Using Pyinstaller in the primary development environment

If you are developing on Linux (and your program is working) then presumably everything you need is installed.  You should be able to install and use Pyinstaller.

Except! If you are using an IDE (integrated development environment) like PyDev plugin in Eclipse IDE.  In that case, the IDE may play tricks with the PYTHON_PATH to help your app find dependent packages.  For example, my app used other Python packages that I had written that were separate linked projects in Eclipse.  Pyinstaller was unable to find them.  So I needed to package up (Python packaging: > python setup.py sdist)  those projects and install them (>python setup.py install) on my machine in the standard place.  In other words, make your machine look like a user’s machine without an IDE who was figuring out  the dependencies themselves, and downloading Python packages in order.

Then Pyinstaller worked fine.

Moving to OSX

On another platform, you first need to reconstruct your primary development environment. For example, download and install:

• (Python comes with OSX)
• Qt
• PySide
• your own dependent Python packages (that you wrote)
• any other Python packages not included with OSX
• Pyinstaller (the very latest, dev version since bugs for OSX are more likely fixed there.  I encountered a Pyinstaller bug with “qt_menu.nib not found” that was fixed in the dev version of Pyinstaller)

Then Pyinstaller SHOULD work (and it seems to for me, the jury is still out.)

Pyinstaller works but your app may not.  Especially for apps that use low-level details of the framework, like handling events in the event loop, your app may not work on other platforms.

Crashes on OSX

My app crashes with a seg fault.

A dialog appears titled “Problem Report for myApp”.   It says it quit unexpectedly (user friendly terms for crashed.)  Choose the “Details” button.  It will show you a stack trace.  Note the app is multithreaded.

My program also gets many warnings on the console: “..you may have two sets of Qt binaries..”

Enabling more meaningful stack trace from Python

Your stack trace likely includes many calls inside the Python interpreter.  They won’t be very meaningful unless you are using a Python interpreter built for debugging.  How to build a debugging version of Python is documented on the web.  But I decided to go back to basics.

Hello world

“Hello world” is a tiny program used for teaching and testing.   It is a minimal test.  When you report a bug, or are exploring a bug, you often reduce your program to the smallest program that exhibits the buggy behaviour.   I hacked together this hello world:

import sys
from PySide import QtCore, QtGui
from PySide.QtGui import QApplication, QMessageBox

def main():
    a = QApplication(sys.argv)
    m = QMessageBox(QMessageBox.Information, "Hello", "World")
    m.exec_()

if __name__=="__main__":
    main()

It opens a dialog box, using Qt and PySide.

This program does not crash or give warnings about “two copies”.  However, I found that if you omit the second import, it does crash when you exit the program.  This suggests that Pyinstaller is not quite figuring out the dependencies reliably.

At this point, I decided that the OSX platform is not receiving all the love it should:

• from Apple for not making it easier for open source developers
• from PySide for not installing itself so  that Pyinstaller could determine dependencies as on other platforms
• from Pyinstaller for not hard-coding exceptions on the OSX platform for its dependency determination algorithm

I decided to move on to test Pyinstaller and PySide on a more popular platform (Windows.)  In other words, pick low hanging fruit, least effort for the most reward.

To install gdb on OSX

If you can’t see your problem in the stack trace,  you may need a debugger, like gdb, to examine the stack trace.

• get an Apple ID so you can use the App Store
• install Xcode (the official Apple IDE) free, in the store
• start Xcode and choose “Xcode>Preferences>Downloads”, and choose the “Install” button next to  “Command Line Tools”

(gdb is apparently not part of OSX, and not part of Xcode until you perform this secondary install step from within Xcode.)

(I started to download gdb from the gnu site, and quickly ran into dependency hell: .configure told me gcc was not installed. I figure the Apple Xcode package is best.)

(You DON’T want to learn Xcode.  The learning curve is steep.  There is a reason you are writing in Python.  You want to think about objects and not the tools.  You DO want to learn Xcode.  It is pretty and comprehensive.  The Objective-C language derives from an object-oriented, high-level language SmallTalk but also derives from low-level C.  It supports the mobile platform iOS.)

To be continued…

Testing

You should test your bundle on a clean machine.  It may work on your development environment only because your app finds packages there that were not properly bundled.  Another place where a virtual machine is handy.

Final packaging

Pyinstaller only creates a single executable (or a directory), but does not create an archive suitable for distribution (compressed, signed, with the proper suffix, etc.) through official, one-click-to-install, channels.  (You could distribute the single executable to trusting and computer knowledgeable users.)