Switching to QML from QWidget using Qt and PyQt

This is high-level, loose assemblage of discussion and links.

Context: you have a cross-platform app written using Qt and PyQt.  It is written using QWidgets.  You want to switch from QWidgets to QML.

The discussion is not exclusive to PyQt.  Much of the discussion also applies to apps writtent in C++, but mentions PyQt also.

This might also have some relevance for apps written purely in QML/Qt Quick.

Why switch to QML from QWidget?

These are possible reasons:

  • QML uses native platform style whereas QWidget fails to have native look and feel on some platforms (iOS)
  • QML more strongly follows the ModelViewController paradigm
  • QML is declarative (more compact and dense)
  • a QML GUI is easier to change (to provide multiple or improved GUI’s for the same business model)

To me, the first reason is the most compelling.  The last reason is least compelling: you probably won’t be changing the GUI often (but then here we are talking about changing at least the implementation of a GUI.)

(Qt apps are excluded from the Windows Store because Qt did not support Windows8/WinRt but Qt5.4 and QML may remedy that?)

Model View Controller paradigm

Qt documentation: Model – View Options in Qt

Wiki: Model–view–controller

It is quite likely that your existing QWidget apps does not strictly adhere to the MVC paradigm.  Thus you will be probably need to refactor your code: separate model code from view code.

A strategy for  developing (code changes) to switch to QML from QWidget

Assuming your app using QWidgets already runs and has tests, a good strategy is to keep the QWidget implementation of the View working, while you develop the QML implementation of the View.   In other words, use test driven development: make small changes, then test the QWidget implementation AND the QML implementation.  The QWidget implementation should always remain fully functional after each change.  The QML implementation will also always work, but in a stubbed-out, minimal fashion.

Changing the existing QWidget app will largely entail:

  • splitting classes along model/view lines
  • converting procedure calls to signals, connections, and properties

Swapping in the QML implementation of the view will be just a small change in the startup code: creating QML instead of QWidgets.  (Its a little more complicated than that, you might be making connections in the QML instead of in the main app.)

Embedding QML in a QWidget app

You can implement the main app window (and all other GUI) in QML.  But you can also embed QML GUI inside a QWidget app.

Briefly, it entails using QQuickView.  But a QQuickView not is-a QWidget.  But you can wrap a QQuickView in a QWidget:  Qt documentation: Introducing QWidget::createWindowContainer() discusses how to wrap a QQuickView in a QWidget.

Why retain a QWidget shell embedding QML?  I suspect there are few reasons to do so.  One reason might be just the strategy discussed above; get your feet wet by embedding QML and make the last step converting the main window from QWidget to QML.

My reason was different: my app uses elaborate UI  picking of objects from a QGraphicsScene.  The QML scene graph doesn’t expose the same API for picking that the QGraphicsScene does.

More on the interface between model and view using QML

Qt documentation:

Where ‘from C++’  more generally means ‘from model objects’ .  And the discussion applies to model objects written in PyQt.

The interaction is two-way, bidirectional:

  •  Control (in MVC) is from the view to the model
  • View is from the model to the view

Your interaction may be one direction or both:

  • a weather app is just a viewer of the weather
  • a styling app is just a controller of a document

Much of the logic in the QML is not business logic, just GUI logic.  For example, a button may switch pages and have no effect on the business model.  The button ‘controls’ the view, but not the business model.

Not all the interaction need be between the model and the view.  For example, if you are embedding QML in a QWidget app, the QWidget app may need to simply activate (show, make visible) a QML view component.  In other words, the interactions may be between the part of the view implemented in QWidgets and the part of the view implemented in QML.

PyQt and QML

PyQt fully supports QML.  You can use Python and PyQt for your model.

PyQt documentation:  Integrating Python and QML

I have shared a hacky Github project where I learn and test PyQt and QML.

For the same reason you use QML for the view, you should consider Python for the model (more expressive in fewer lines of code.)

The QML tradeoff: more portable but less native?

Typically, a cross-platform solution makes certain compromises.  Sometimes its discussed using terms such as “least common denominator” or “adaption.”

I can’t fully discuss this here.  (And probably the set of compromises that have been made are not fully documented in one place.)  But you should consider whether you will be able to live with any such compromises.

For example, I found that a QML Dialog on a desktop (Linux, I haven’t tested OSX) opens in a separate window, not on top of (transient to) the main window.  That probably doesn’t meet the HIG’s for desktops.  Possibly there is a solution that I haven’t yet found.  Or possibly I should look at that as a feature to embrace: I should make my dialogs non-modal and treat them as tear-offs that should not obscure the main window anyway.  But its an example of the compromises you might need to make.

There is much more to discuss here.  Briefly, Qt Quick Controls does use native widgets where possible, then falls back to QWidget implementations if possible, and then to a pure-QML implementation.

But again, its hard to know in advance without trying it.

Structuring your QML source code

In many examples, you see long QML files.  I  think that is an artifact of being an example: put everything in one file for ease of reading.  You can structure you QML source files into smaller files in more directories, using imports.

Qt documentation: Import Statements | QtQml 5.3

Packaging your app

I don’t foresee any show stoppers here.  Just more to learn.  If you are already packaging a PyQt app, you probably are already packaging resources using pyrcc.  The QML is just another resource to package.

But, I’m not sure how third-party QML plugins get packaged?

A strange architecture: PyQt and QML uses two interpreters

A PyQt app using QML uses two interpreters: the Python interpreter and QML’s Javascript interpreter.  (And on Android, there is a third Java interpreter.)  There is much discussion about the safety of, and constraints applied by app stores, re interpreters.   But this is just a curiousity, there are already shipping apps, using interpreters, in the stores .

A template for QML using models

Qt’s example Weather App is a starting place for reading code: Cross-Platform Applications in iOS and Android Stores with Qt

(I hope to reduce it to a simple template.)

Desktop to Mobile GUI difference: sharing

The main problem that I have found is that for certain apps (document editing or creation apps as opposed to reading or browings apps) the desktop:

  • exposes the file system to the user (in the File menu)
  • uses drag and drop

whereas mobile platforms use the notion of ‘sharing’ to the file system and other apps

Notably lacking in Qt Quick Control is a ‘share sheet’.  A github project QtSharingKit seeks to remedy that, with a QML plugin.

Pragmatically, that means more work for you:

More links about QML, Qt Quick, and Quick Controls.

You use these technology pieces:

  • QML a language.
  • Qt Quick: the standard library of types and functionality for QML
  • Qt Quick Controls: canned (pre-built) QUI controls built using QML
  • QML Applications

Using pyqtdeploy on MacOS to cross compile a PyQt app for iOS: part 3

Rehash

This is part three of three parts:

  1. install Qt and Xcode, then test build and deploy an example Qt app for iOS
  2. cross compile static libraries for iOS (using pyqtdeploy.)
  3. use pyqtdeploy to cross compile and package your app.

Overview

As discussed in Choosing a Toolchain, there are many possible toolchains.  In previous parts, we only compiled static libraries for the ARM architecture of real iOS devices.  In this example, we will use the pyqtdeploy/Xcode/real iOS device toolchain.

The steps are:

  • use pyqtdeploy GUI
  • sign up and pay for an iOS developer
  • connect and configure your real iOS device for development
  • use Xcode to deploy
  • run app on real device

Pyqtdeploying your app for target iOS

Here you use the pyqtdeploy GUI app to prepare projects, makefiles, and source code.  This is very similar as for any target.

Connecting and configuring your real iOS device

Hook up your device with a USB cable.

What follows is highly automated.  Xcode should start and display the Organizer, featuring your device

Briefly, wait for it to process a while, then choose the ‘Configure for Development’ button, and so forth.

(Dated, confusing, and slightly conflicting instructions are given here.)

Compiling and packaging your app for target iOS using Xcode

Assuming you ran qmake step in pyqtdeploy, it created an xcodeproj.

Open that project in Xcode.

Configure Xcode to deploy to your real device: click in the left side of the title bar where it may say ‘Foo project>iosSimulator’ ??.  A list of devices (real and simulator) should appear.  Choose a real device.

Choose Product>Run.  A few dialogs may appear:

  • allowing codesigning to occur
  • to tell you to unlock your sleeping device using its four character passcode.

This deploys your app, updating it if it was deployed previously.

This also runs your app, displaying stdout and stderr within Xcode.  Your app will become active (displaying its windows) on the device.  Touch the device to generate input events and debug.

Run your app later

On the real device, pan left and right in the display of apps to find your app.  (It probably has a default, nondescript icon.)  Tap to run it.

FAQ or Notes

No QPrintSupport module of Qt.  If you import PyQt5.QtPrintSupport, you get an error at link time.  Unlike on Android where I did not seem to get a link error (but only a crash?)  Apparently Qt does not yet support printing on mobile devices.

Debug on iOS first.  In my opinion, because Xcode generates better error messages at link time, and because Xcode displays your program’s stdout and stderr in a window (unlike using QtCreator on Android? I could not easily get it to work.)

If your app was designed for the desktop, it may work since the QtApplication flag about translating touch events to mouse event defaults to ‘translate touch to mouse events.’  If you change the default, it seems like some widgets e.g. menus and buttons will not work (they depend on mouse events?)

Notes on pyqtdeploy

You should read the official documentation, but it is brief.  Here, I elaborate:

  • discuss general concepts that seem obvious to experienced developers
  • discuss specific commands

About pyqtdeploy

(See my other platform specific posts, I plan to consolidate sections here.)

Installing and upgrading pyqtdeploy

The pyqtdeploy download page is not really a place to download.  It says that you can:

  • install the latest stable release using pip3
  • install the latest development version using mercurial (hg)

Managing pyqtdeploy using pip3

Installing pip3

pip3 is needed to install stable pyqtdeploy

  • OSX: pip3 is installed with Python3.
  • Ubuntu: if pip3 is not already installed:
sudo apt-get install python3-pip
Installing latest stable pyqtdeploy
pip3 install pyqtdeploy

(If that fails on permissions: sudo pip3…. )

Upgrading stable pyqtdeploy

Pyqtdeploy is released frequently.  To upgrade to the latest stable release.

>pip3 install pyqtdeploy --upgrade

Soft linking to pyqtdeploy

It seems that the above does not put pyqtdeploy and pyqtdeploycli executables in the path.  You might need to make soft links:

>cd /usr/local/bin
>sudo ln -s /Library/Frameworks/Python.framework/Versions/3.4/bin/pyqtdeploycli pyqtdeploycli
>sudo ln -s /Library/Frameworks/Python.framework/Versions/3.4/bin/pyqtdeploy pyqtdeploy

Managing your development version of pyqtdeploy using Mercurial

Note that mercurial uses Python2.7 but pyqtdeploy uses Python3.

Installing pyqtdeploy using hg

>hg clone http://www.riverbankcomputing.com/hg/pyqtdeploy
>cd pyqtdeploy          # to the directory you just cloned
>make VERSION           # ? not sure why next two steps are not dependencies in the Makefile
>make pyqtdeploy/version.py
>sudo python3 setup.py install

Again note that we install in python3 site packages.

Updating your development version of pyqtdeploy
>cd ~/Downloads/pyqtdeploy*
>hg pull
>hg update
> ??? make VERSION
>sudo python3 setup.py install
Installing Mercurial

You need Mercurial to download development version of pyqtdeploy.  Mercurial is a version control system.  Also known as the ‘hg’ command.

The Mercurial downloads page.

Note that Mercurial uses Python 2.7 and doesn’t support Python3 but that’s usually OK since Python-2.7 is usually installed.

Pyqtdeploy is now two tools

Starting with version 0.6, there are separate tools:

  • pyqtdeploycli: a command line tool for building static libraries
  • pyqtdeploy: a GUI tool for building your app

(I suppose you could also use pyqtdeploycli to build your app, passing it a configuration file.  But using the GUI pyqtdeploy is much more intuitive and easy.)

Disambiguating deploy

The pyqtdeploy tools don’t actually deploy.  They just prepare for deploy.  They may create or touch:

  • source files (.c, .cpp and .py)
  • qmake projects (.pro)
  • make files (Makefile)

Subsequently you must continue to build using qmake, make, and other tools such as Qt Creator and Xcode.  Then you have a binary, executable app.

Qt Creator does not deploy to the App Store

You must read QT Platform Notes – iOS very carefully.  It says that Qt Creator will do everything for building a Qt iOS app except:  deploy to the App Store.

You don’t need to use Qt Creator, you can use only command line tools (and then possibly the Xcode GUI.)

But this is related to the difference between the iosSimulator and a real device: they use different ABI architectures.  You should think early about whether you will use Qt Creator, and whether you will use the iosSimulator or a real iOS device.  You must fork the process of building early, when you are building static libraries (for either iosSimulator/ i386 architecture, or a real device/ ARM architecture.)

(Will Qt Creator deploy to Google Play store?)

Choosing a toolchain

This is related to choosing a machine for deployment.

Ideally, you learn and use one toolchain that works for every platform (avoid the learning curve for some tools.)

  • use pyqtdeploy/qmake/Xcode/real iOS device for iOS
  • use pyqtdeploy/qmake/Xcode/real Android device for Android

But for Android I have also used, with mixed results:

  • pyqtdeploy/Qt Creator/real Android device
  • pyqtdeploy/qmake/command line scripts/real Android device

I’m not sure whether I need to learn and use the emulator/simulator tools to do sanity testing across device models (e.g. iPhone vs. iPad) and OS versions (e.g. iOSv7 vs iOSv8.)

Cross platform software development

These are not entirely general notes, but often specific to using:

  • Qt
  • PyQt
  • pyqtdeploy

GUI considerations for crossing mobile and desktop platforms

This doesn’t make sense unless your app’s GUI is carefully designed to be portable across mobile and desktop platforms.  I call  this a ‘universal user interface.’

A desktop has, and a mobile platform doesn’t have (?):

  • multiple overlapping windows
  • menubar
  • a file system and file browser

A mobile platform has, and a desktop may not have:

  • multitouch and gestures

I’m still experimenting here (but this a principal reason for my app.)  I don’t think that Qt provides a magic bullet.  That is, everything you can do in Qt may  not be portable and you still might need to have platform specific (mobile versus desktop) code in your app.

(And Qt encompasses many tools, such as Qt Quick, which I am ignoring here, but which may be more portable.)

Qt considerations for crossing mobile and desktop platforms

Printing: Qt does not currently seem to compile the QtPrintSupport module for the mobile platforms.

See QTBUG ????

Early versions of Android and iOS also did not support printing.  Now they do:

  • iOS: AirPrint
  • Android: ???

I suppose it just takes time for Qt to adapt to the mobile print frameworks.  How do they differ from desktop print frameworks?

How many dev machines do you need for cross platform development?

Suppose your software development process includes these steps:

  • development (creating new features and fixing bugs)
  • deployment (packaging)
  • testing

You might want a machine dedicated to each step (so you don’t need to worry about cross contamination.)

Probably you want a Mac for your deployment machine.  A real Mac can deploy to iOS, Android, and MacOS.  You can also create virtual machines (Ubuntu and Windows) on that Mac so that it can also deploy to those platforms.  In other words, a Mac can deploy to all platforms supported by Qt.

The reverse is not true.  No other platform will let you deploy to iOS since you are not legally permitted to create a MacOS virtual machine on anything other than a real Mac machine, and since Xcode runs only on a Mac, and since only Xcode lets you develop for iOS (with exceptions such as HTML5/Javascript?)

Note that Qt does not currently support WindowsRT (Metro).  That’s of little use anyway, since that platform is trailing in the market?

The latest pyqtdeploy user guide has a discussion about directory structures that enable sharing of data across target deployment processes.

Favoring Qt over Python modules for crossing platforms

In your code,  you should favor using Qt classes over using Python modules.  For example, use Qt’s support for file system operations over Python’s support.  More specifically, try not to use the os module (which drags in many other Python modules) of Python.

The reason is, as of this writing, the Python organization does not support mobile platforms.  You may have difficulty cross compiling Python modules to mobile platforms (which you must do when you use pyqtdeploy.)  Also, you will package Qt libraries with your app, and also packaging Python modules that do the same thing is superflous.

Emulators and Simulators for Mobile Devices

To test a mobile app without a real device you can use:

  • Android emulator (AVD)
  • iOS simulator

Note the difference.

An AVD is an emulator, presenting the ARM ABI (the interface of the target hardware).  It executes the same binary that will run on a real device.  Pragmatically, you only need to compile your app once, for the ARM ABI.

The iOS simulator is a simulator, presenting the iOS API (the interface of the target OS.)  It executes a binary that is compiled for the i386 ABI, which is not the same ABI as real devices (ARM.)  Pragmatically, this means you need to compile your app for both ABI’s  (if you want to test on the simulator.)

Deploying PyQt Python Qt apps cross platform using pyqtdeploy

These are general notes and links for this subject.  In other words, this is a table of contents.  Many links are often specific to a target platform.

Some of the links to my posts are to incomplete drafts.   Also, I edit these blogs periodically, since pyqtdeploy is actively developed and I am learning too.  It is best to consult source documentation.

Cross-platform and Localization/Internationalization/i18n

You want the largest possible audience or market.  You are using Qt because it:

  •   is cross platform.
  •  is cross language: supports localization ( internationalization)

You increase your audience by porting to new platforms AND by localizing to new languages.

Contents

Original (source) documentation

pyqtdeploy v0.6 User Guide

Qt for iOS

Qt for Android

General

Cross platform software development

More notes about pyqtdeploy

Deciding between Xcode and QtCreator for packaging PyQt apps for OSX

Target Linux and MacOS

Using pyqtdeploy

MacOS bundling (packaging) a pyqtdeployed app using Xcode

 Target iOS

iOS part 1 (prepare tools)

iOS part 2 (cross compile static libraries)

iOS part 3 (cross compile app and deploy)

 Target Android

Android cross compiling Qt example C apps

Android part 1 (prepare tools and cross compile static libraries)

Android part 2  (cross compile app)

Android part 3 (deploy and debug) DRAFT

Localization

General notes

Localizing PyQt app for target OSX using Xcode

More localizing PyQt app for target OSX

Using Qt Linguist phrasebooks

Adventures in translating an app to Chinese

Universal user interface

Commands that support gestures

Using pyqtdeploy0.5 on Linux to cross compile a PyQt app for Android: Part 3

Rehash

You have:

  1.  set up your dev machine and built static libraries
  2. pyqtdeployed your app so it is ready to deploy

This part discusses:

  • deploying your app to a test target machine
  • debugging your app

More about pyqtdeploy work flow

The pyqtdeploy GUI lets you choose your work flow.  That is, in the ‘Build’ step, you can choose optional steps: qmake, make, ‘run app’.  For example, you can choose none of those, and Build will only create a .pro file.

Then you can switch to:

  • the command line
  • or Qt Creator (a GUI app)

and resume the work flow (qmake, make, deploying, and running.)

(I’m haven’t tried  ‘run’ in pyqtdeploy for target Android, I always switch to another tool.)

pyqtdeploy developer notes

The pyqtdeploy directory contains a file DeveloperNotes.txt.  These notes are by the developer of pyqtdeploy  but are useful hints about Android development.  The author of the notes has said the notes are NOT definitive and should be used with care.  I have found  typos there.

Deploying using the command line

Follow the instructions in the developer notes.   I.e. :

# Assert you ran qmake and make in pyqtdeploy, but not run

# cd to dir that pyqtdeploy created
cd ~/hellobuild/build
export ANDROID_NDK_ROOT=$HOME/android-ndk-r10
export ANDROID_SDK_ROOT=$HOME/android-sdk-linux
# configure install
make install INSTALL_ROOT=deploy
# make deployable
/home/bootch/Qt/5.3/android_armv7/bin/androiddeployqt \
 --input android-lib*.so-deployment-settings.json \
 --output deploy
# deploy
adb install -r deploy/bin/QtApp-debug.apk
echo Now go to AVD and run app

This only installs your app to a target device.  You subsequently must use the target device’s GUI to run your app.

If you get:

Failure [INSTALL_FAILED_ALREADY_EXISTS]

this means you omitted the -r flag, which means reinstall.  An alternative is to manually uninstall the app first (on the device, choose Settings>Apps, then select your app, then choose the Uninstall button.)

Deploying using Qt Creator

When you choose the ‘Run’ icon in Qt Creator, it builds, deploys, and runs your app.  So it is more automated, but harder to understand what Qt Creator is actually doing on your behalf.

Debugging pyqtdeployed apps on android

First, refresh the tools

The tools (especially pqytdeploy, SIP, and PyQt but also Qt and Python) are updated frequently.  You should use the latest versions, which may have fixed some Android crashes.

Remember that pyqtdeploy is the tool that helps build static Python, etc. libraries.   So if you update pyqtdeploy, you should probably rebuild static Python starting with invoking pyqtdeploy (so that any new patches known by pyqtdeploy are applied.)

Stderr and Android logcat

Unfortunately, Python prints exceptions to stdout/stderr.  These by default are dedirected to /dev/null on Android (which means they disappear.)

On android, the system log is called ‘logcat’.  To see it remotely:

>adb logcat > logcat.tmp
>Ctl-C
>vi logcat.tmp

Its a circular file.  Fileness means it persists (even across your debugging sessions.)  Circular means it wraps around, overwriting older data. Pragmatically, one debug session never seems to wrap around on itself.  The end of the file is always the latest log entries?  You can clear it by rebooting the device (powering it off.)

Redirecting stderr to logcat

You should be able to configure your target device to redirect stdout and stderr to logcat.  To do this requires:

  • a rooted real device
  • an AVD (virtual device)

An AVD is always rooted (you have superuser privileges to bypass all security and do what you want).

Rooting a real device is an adventure unto itself, so I chose not to do it yet.  But the advantage is that a real device eliminates the simulator layer, and is quicker to boot and so forth than an AVD.  You can still do certain debugging on an un-rooted real device, you just won’t be able to see Python exceptions directed to stderr.

!!! If a device is not rooted, the adb commands to redirect stdio fail quietly.  So you still won’t see stderr in the logcat.

To do it you can:

  • issue adb commands
  • create a persistent configuration file

adb commands to redirect stdout/stderr

>adb shell stop
>adb shell setprop log.redirect-stdio true
>adb shell start

Remember this doesn’t persist if you reboot the device (whether and AVD or a real device.)

If the device is rooted real device, you must precede that with >adb root.  (AVD’s are always rooted, so >adb root is not necessary.)

Debugging methods

Traditional choices:

  • use a debugger such as gdb (which is built into Qt Creator)
  • harness code with print statements
  • simple code analysis using crash dumps and Python exception tracebacks

It depends on where the bugs are:

  • in the Python interpreter
  • in your pyqtdeploy build process
  • in your app’s Python code

It is not uncommon to find bugs in the Python interpreter, since the Android platform is not supported by Python.org.  (the pyqtdeploy process patches the Python interpreter, and you might find that another patch is required.)

The pyqtdeploy build process is intricate.  It is not uncommmon to find that you have not frozen a needed Python module (and then you might get an ImportError exception.)

Once you get as far as running your own app’s Python code, you probably have installed an exception hook, and can display the exception and traceback on the GUI of the device. (But probably, you are only porting to Android, and have worked out most of your app’s bugs on another platform, except for any Android specific code.)

Harnessing pyqtdeploy apps on android

I experienced the need to harness (since my app seemed to crash before Qt Creator’s debugger could even get started properly and without any Python exceptions.)

This means writing C code.  You can use:

  • qWarning() C function.  It is properly redirected to Android logcat.
  • __android_log foo C function.

If you need to harness the Python interpreter, you might need to study how to write Python extensions, for example to convert a PyObject * to a representation that is a C string suitable for qWarning().

I harnessed pyqtdeploy_start.cpp and Python/import.c.

Here is the code I used:

TODO

Building a debug version of Python

reading an Android crash dump

packaging using Qt Creator

Using pyqtdeploy on MacOS to cross compile a PyQt app for iOS: part 2

This is work in progress that I will periodically update.  I have separate blogs about deploying on mobile platforms, Android and iOS.  These are related and you might read both.

History of this post:

  • originally written for pyqtdeploy version 0.5, Qt 5.3, PyQt5.3
  • Dec. 2014 updated for latest versions

Recent changes:

  • pyqtdeploy now uses a separate command pyqtdeploycli for use on a command line
  • PyQt when used with a QML program that uses QGraphicsItems might need to be patched

Rehash

This is part two of three parts:

  1. install Qt and Xcode (and Python?), then test build and deploy an example Qt app for iOS
  2. cross compile static libraries for iOS (using pyqtdeploy.)
  3. use pyqtdeploy to cross compile and package your app.

Using the same machine to package for MacOS and iOS

Can you do it?  You can, by segregating libraries for different targets. Ideally, I would not use the same machine to package for MacOS and iOS.  Instead, I would create a separate virtual machine in which to package for iOS.  Then I would not need to think or worry as much about the segregation.  When you use the same machine for many activities (development and packaging for many platforms) there is always the chance that you mess something up, halting another activity.  Virtual machines help you isolate the activities.  (For an example of a mess, you download a shared artifact such as Qt that introduces bugs on one target platform.  For another example, you inadvertently share an artifact that should not be shared across target platforms, say through improper PATH’s.)

I only have one physical Apple desktop (that is new enough to run MacOS OSX 10.9.) It is not where I do most of my programming (yet), only where I package my app for distribution.  I have already used it to package my app for MacOS.  Unfortunately, when I tried to install OSX 10.9 Mavericks to a VirtualBox VM, I failed.  I suspect my hardware is too old (one of the earliest Mac’s that supports OSX 10.9.)

Segregating an iOS build

Eventually create a directory structure like this (say in home):

ios
   iRoot
      python
   Downloads
      Python-3.4.0
      PyQt-gpl-5.3.1
      sip-4.16.1
   pensoolBuild
      pensool.pdy
      build
         pensool.pro

The ‘ios’ directory groups subdirectores related to cross compiling to iOS. iRoot is a stand-in for the root of the target device.  Many build objects are placed (installed) here during the cross build, read from here during the final packaging, and installed to similar places on the target device.  Also known as SYSROOT. In ios/Downloads:

  • Python-3.4.0 is a copy of the Python source distribution
  • PyQt-gpl-a123 is a copy of the PyQt source distribution
  • sip-4.16.1 is a copy of the SIP source distribution

All of these directories are downloaded.  Then building occurs in the directory.  Build install products go to subdirectories of ~/ios/iRoot pensoolBuild groups directories related to pyqtdeploying your app.  You run pyqtdeploy as a GUI here, passing pensool.pdy.  Pyqtdeploy creates the build subdirectory, populating it with a pensool.pro and source files. Then you run qmake.  It installs your app to iRoot, ready to download to the iosSimulator or a real device.

Downloading the source for the libraries to be built statically

Again, I am segregating these directories in ~/ios/Downloads to prevent any mixing. Download python, pyqt, and sip.  Download the ‘Gzipped source tarball’ for Python. If you already have built your app for MacOS desktop, you already have these directories.  You probably can just copy the directories, and rebuild (the makefiles should clean where necessary) but for safety you might want to download fresh copies. In my case, new releases of Python (3.4.1) and SIP (4.16.2) existed past my prior downloads.  I chose to stick with the versions I used to build for the desktop.

Refreshing Qt

If you already have an up-to-date Qt, including iOS stuff, skip this.  Suppose there is a new release of Qt.  Reinstall it using the online installer, say to ~/Qt.  (You may need to move the old installation directory to the trash.)

When you install, it should recognize your platform and offer to install iOS related stuff:

  • a kit for Qt Creator,
  • prebuilt frameworks (ready to bundle into an iOS package.)

Note that iOS does not allow dynamic libraries.  Hence Qt for iOS by default, as installed by the on-line installer, is built as static libraries. Thus, the Qt/5.4/ios directory is a read-only input for an iOS packaging (IOW, you should not need to build Qt statically for iOS.)

Updating PyQt for the host

To run pyqtdeploy requires Python3 and PyQt dynamic libraries on the host (regardless of target.)  Note that this is separate from statically building PyQt for target iOS.

If you there is a new version of Qt or PyQt, you should rebuild SIP and PyQt (the dynamic libraries.) While it is possible that the Qt API has not changed, and the old PyQt bindings will work, it is easier to rebuild than to determine whether the API has changed.  Here, PyQt installs to the native system’s python directory e.g. ~/python/    (which also should be in your PYTHONPATH env variable.)

TODO: patch sip/QtMultimediaWidgets/qgraphicsvideoitem.sip if you use QML.  Delete the code %ConvertToSubClassCode…….%End.  Do this for every copy of PyQt (the one for OSX and the one for iOS.)  You must do this before building PyQt, as the configure step is the one that generates C++ code from this patched .sip file.

These are the same commands used to build PyQt for OSX:

cd ~/Downloads/sip-4.16.5
python3 configure.py
make
make install

cd ~/Downloads/PyQt-gpl-5.4
python3 configure.py --qmake ~/Qt/5.4/clang-64/bin/qmake --sip ~/python/bin/sip
make -j4
make install

Expect tens of minutes of compiling for PyQt.  Seems like ‘sudo’ is not required.

Environment variables and paths

On OSX, I needed:

>export PYTHONPATH=/Users/bootch/python/lib/python3.4/site-packages

That points to the  installation directory for Python3.4 (which you must build from source since OSX does not ship with Python3).  If this env variable is missing, pyqtdeploy complains that it can’t find Qt modules. Also you will need to explicitly pass SYSROOT to many commands below.  I’m not sure if defining that in the environment will work instead.  SYSROOT tells many commands where to install the built products, in this case to ~/ios/iRoot Also, sip is a command that needs to be on the PATH.  Again, you need the same version of sip installed on your host as you intend to run on the target (iOS.)  I assume you have already built your app for the MacOS desktop and that sip is on PATH.  Otherwise, you can pass the –sip=/Users/bootch/python/bin/sip option to some commands below.

Updating pyqtdeploy

Probably there is a new version of pyqtdeploy since you last used it.  You always want to use the latest version, possibly even the unstable version. See about installing and upgrading pyqtdeploy in Notes on pyqtdeploy

Building Python statically for target iOS

>cd ~/ios/Down*/Pyth*
>pyqtdeploycli --package python --target ios-64 configure
>~/Qt/5.3/ios/bin/qmake sysroot=/Users/bootch/ios/iRoot
>make
>make install

(see more comments in my post about cross compiling to Android.) The result is a new directory ~/ios/iRoot/python populated with a library of the Python interpreter.

Building SIP statically for target iOS

(You should also have built SIP for the host.)

>cd ~/ios/Down*/sip*
>pyqtdeploycli --package sip --target ios-64 configure
>python3 configure.py --static --sysroot=/Users/bootch/ios/iRoot --no-tools --configuration=sip-ios.cfg
>make
>make install

(see more comments in my post about cross compiling to Android.)

(Edited: this not needed: >~/Qt/5.4/ios/bin/qmake sysroot=/Users/bootch/ios/iRoot)

Building PyQt statically for target iOS

>cd ~/ios/Down*/PyQt*
>pyqtdeploycli --package pyqt5 --target ios-64 configure
>python3 configure.py --static --verbose --sysroot=/Users/bootch/ios/iRoot --no-tools --no-qsci-api --no-designer-plugin --no-qml-plugin --configuration=pyqt5-ios.cfg --qmake=/Users/bootch/Qt/5.4/ios/bin/qmake --sip ~/python/bin/sip
>~/Qt/5.4/ios/bin/qmake PyQt5.pro sysroot=/Users/bootch/ios/iRoot 
>make # expect to wait tens of minutes
>make install

Note that you may need an intermediate step, to edit pyqt5-ios.cfg to remove extra Qt modules that don’t compile and that you don’t need. FAQ: if you get “You need a working sip on your PATH”, you can pass –sip=/Users/bootch/python/bin/sip to the configure step.

(Edits:

  • 5.3=>5.4
  • add: –sip ~/python/bin/sip
  • delete: –no-qml-plugin (I want to use QML on iOS now.)
  • add: PyQt5.pro

Continuing

Now you have static libraries for Python, PyQt, and SIP.  In the next part, we use pyqtdeploy to build a helloworld app and deploy it to an iOS simulator.

Using pyqtdeploy on MacOS to cross compile a PyQt app for iOS: part 1

This is part of a continuing series of blogs of my experience using pyqtdeploy.

You might want to start at the tutorial which I link at the bottom of the blog.  This blog really adds nothing except some discussion of why and how to get the required tools, and some context for pyqtdeploy.

Here, I have switched host development platform (now MacOS or OSX) and target (now iOS.)

I switched because I encountered difficulties porting to Android.  I decided to digress, in hopes that what I learn on a different target (iOS instead of Android) might teach me something.  Besides, MacOS has always been a highly regarded development platform, and many pyqtdeploy example screenshots seem to be from MacOS.  If you are using pyqtdeploy, then you probably want to port to all supported platforms, and you should choose the path of least resistance.  There is no use struggling with one platform (Android) especially if, like me, you are equally unfamiliar with all the mobile platforms or if one platform (iOS) might be better supported, more stable, etc.

The process has three parts:

  1. set up tools
  2. compile libraries statically
  3. compile your app

This is part one, setting up tools.

The goal is to get a Qt example app running on iOS.  Getting an example to run is not required to use pyqtdeploy.  It is just a milestone, that insures you have a proper build environment.  Hello world.

This blog assumes you want point-and-click.  You can use Xcode and Qt Creator tools from the command line, without using their GUI’s.  During the second part, you WILL be using the command line to build static libraries.  But you probably would not be reading this if you already know the commands.  And learning the GUI versions of the tools may guide you.

Required Tools

You need:

  • Xcode
  • Qt Creator

Pyqtdeploy uses qmake, which is packaged with Qt Creator.  (Again, later, you might be able to forego Qt Creator, and use the command line.)

Qt Creator and qmake invoke tools (compilers etc.) that are packaged with Xcode.

Which versions of tools

I used:

  • OSX 10.9.4 Mavericks
  • Qt5.3.0
  • Xcode 5.1.1
  • Qt Creator 3.1.1
I briefly tried to use OSX 10.7.5, Xcode 4.6.3, and Qt 5.2.  But it didn’t work and I decided there was no point not to use the latest tools. 
Everyone else is moving forward.  Qt on mobile platforms is bleeding edge.  In my experience, Qt 5.3 is much better than older versions.  Xcode 4.6.3 will actually be two major versions behind, now that Xcode 6 is imminently released.

Downloading the tools

The links are easy to find.

My advice is to download the entire Xcode (rather than say just the command line tools.)

My advice is to use the “online installer” to install Qt and its tools.  It is a small, smart installer that downloads and installs other components.  It will know your platform and offer you choices to download the proper kits for iOS.

Installing additional Qt components

I think I had installed Qt without using the online installer.  It did not include kits for iOS.  So I needed to install additional Qt components.

About Qt Creator kits

The Qt Creator documentation about Adding Kits tells you how to add kits in Qt Creator.  Restating….

Kits are groups of settings for building apps.  You usually don’t need to create or customize a kit, only use a kit that is distributed with Qt.

When you open a project, including the Qt examples, you are given the chance to configure the kit for the project.  But you must have previously downloaded the kit (which Qt Creator then detects.)

In this example, I wanted the kit for the ‘iOS Simulator.’

Downloading additional kits

You use the Qt ‘Maintenance Tool’  and choose ‘Package Manager’.  If you previously used the Qt ‘online installer’, then when you start the Maintenance Tool it remembers the repository used previously, and gives you choices of additional packages.

Otherwise, you might get an error ‘requires a valid repository’.  This is discussed in the Qt Forums.  To fix this:

and so forth.  After you do this, the Maintenance Tool should offer you the choice of downloading additional kits.

Building a Qt example

Follow this good, short video tutorial Qt5 Tutorial: Pushing Example App to iOS Simulator”.  The analog clock example (without menus) worked for me.

What’s next

The next part will build static libraries for Python, PyQt, and SIP, for the iOS platform.

 

Using pyqtdeploy0.5 on Linux to cross compile a PyQt app for Android: Part 2

Summary

In Part 1:

  • installed tools
  • built static libraries

In this part 2:

  • use pyqtdeploy to freeze Python code and generate a Qt Creator project
  • use QtCreator to compile and link the app

Refresh pyqtdeploy installation

You may need this since:

  •  pyqtdeploy is actively developed,
  •  the cross-compiling feature is the bleeding edge of pyqtdeploy development.

This assumes you cloned the repository (instead of installing a snapshot via pip3.)

>cd ~/Downloads/pyqtdeploy*
>hg pull
>hg update
>sudo python setup.py install

Use pyqtdeploy to generate Qt Creator project

Start pyqtdeploy

Open a terminal and enter:

>mkdir pensoolAndroidBuild
>cd pensoolAndroidBuild
>pyqtdeploy pensool.pdy

where pensool is the name of my/your app.  The pyqtdeploy GUI should open.  Here we made a new directory to build in.  Pyqtdeploy can create many files in that directory.

Configuring your pyqtdeploy project

Glossing over the details, this is similar to pyqtdeploy that is NOT cross-compiling.  (For example, nowhere do you see the word “Android”)  The main difference is that under the “Locations” tab, the “Target Python Locations” section points to a directory which contains cross-compiled artifacts (e.g. a SYSROOT of /home/bootch/aRoot.)

pyqtdeploy ‘Build’

After you have configured pyqtdeploy, choose the ‘Build’ tab.  Uncheck most of the options (i.e. ‘Run Qmake’ and ‘Run make’) since we will be doing that soon (I don’t know, maybe those would work, but I prefer to see any errors from inside QtCreator.)  Choose the ‘Build’ button.  This creates a qt creator project file (.pro)

Build using Qt Creator

Start Qt Creator

Navigate to the project created by pyqtdeploy (say ~/pensoolAndroidBuild/build/Pensool.pro) and choose the project.

The first time you open a project, Qt Creator tells you it has not been configured, and offers a choice of configurations.  Configure the project for Android.

Choose the ‘Build’ button.  (Now Qt Creator shows you a list of Android AVD’s.  You should have one if you have tested Qt Creator’s ability to cross-compile a C++ helloworld example.)

Choose an AVD.  (Now Qt Creator proceeds to compile and you may get errors.  See below, the FAQ. )

 FAQ Frequent errors

These are problems I encountered during the build phase in Qt Creator

Qt skips incompatible Qt libraries (built for the wrong architecture or word size)

This is because Qt Creator is building (linking) for a different architecture than the libraries.  It can be 32-bit versus 64-bit,  or Intel versus ARM architecture?  Typical error message from Qt Creator:

arm-linux-androideabi/bin/ld: warning: skipping incompatible /home/bootch/aRoot/lib/python3.4/site-packages/PyQt5/libQtGui.a while searching for QtGui
/home/bootch/android-ndk-r10/toolchains/arm-linux-androideabi-4.8/prebuilt/linux-x86/bin/../lib/gcc/arm-linux-androideabi/4.8/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lQtGui

How to know what architecture a library is built for (on Linux):

>objdump -f libfoo.a | grep ^architecture
>objdump -f libfoo.a       (a verbose alternative)

If that says ‘i386’, it was built  for the Intel architecture.  If it says architecture UNKNOWN, it is probably for the ARM architecture?  There is a different tool on the Mac.

Anyway, if the libraries differ in either 32-bit/62-bit or architecture, you probably built some of the static libraries incorrectly (without configuring Qt Creator to use the ARM kit by default?)

Optional Python modules are missing at link time

I got this:

main.o:main.cpp:extension_modules: error: undefined reference to 'PyInit__posixsubprocess'
main.o:main.cpp:extension_modules: error: undefined reference to 'PyInit_select'
mathmodule.o:mathmodule.c:function m_log2: error: undefined reference to 'log2'
collect2: error: ld returned 1 exit status

This might be wrong configuration of the Python build.  The Python build is configurable for optional modules (extension modules, written in C.)  This is briefly mentioned at the pyqtdeploy documents:

This will configure Python for a small sub-set of standard extension modules. Your application will probably require additional ones to be enabled. To do this you will need to make changes to the python.pro file and the config.c file (in the Modules directory). See the comments in those files for more information.

A minor quibble is that there don’t seem to be any pertinent comments in the python.pro file.  I think you just need to add additional lines in the body as in this example:

greaterThan(PY_MAJOR_VERSION, 2) {
    MOD_SOURCES = \
    ...
    Modules/_posixsubprocess.c \
    Modules/selectmodule.c \

Here, the names are names of source code files.  You can also view these names in Modules/Setup.dist.

To edit Modules/config.c, you add a line for SOME of the modules which you added a line to python.pro:  those modules whose error is like ‘PyInit…’

This process of figuring out what additional Python modules is similar to the process when you are not cross-compiling.  That is, when you built your app for the desktop, you determined what additional Python modules your app uses.  (The process there was iterative: build, see what Python complains about, add the module, and repeat.  Here, the link phase tells you all the missing modules at once, but it doesn’t tell you the names of the modules.  If you didn’t build your app for the desktop, you can determine the names of the missing modules by grepping the Python source code in the Pythonx.x/Modules directory for the function name given by the link error.

        extern PyObject* PyInit__symtable(void);
        extern PyObject* PyInit__posixsubprocess(void);
        extern PyObject* PyInit_select(void);
...
        {"_posixsubprocess", PyInit__posixsubprocess},
        {"select", PyInit_select},

In this example, _posixsubprocess is a key (and also the name of the source code file) but note that this results in two underscores in the value field.  In this example, ‘select’ is a key, but NOT the name of the source code file (which is ‘selectmodule.c’.  In other words, this key must be the same as the key defined in Modules/Setup.dist.

After you make these configurations, you should not run pyqtdeploy again (because it creates a new python.pro, overwriting your edits) but you should run qmake, make, make install again because qmake reads the python.pro file you just edited.

It seems that you can rebuild and reinstall a static Python without destroying the installed PyQt and SIP components (so you can proceed directly to Qt Creator and retry your build.)

More patches might be needed for Python modules

Again, pyqtdeploy, when helping you build Python, applies patches to the Python source code to account for differences on the Android platform (that Python.org does not support.)  If you configure Python with additional extension modules, you might need to make additional patches (until pyqtdeploy knows all the patches.)  Here are some ill-defined hacks:

SYS_getdents64

When building my app using Qt Creator I got this:

undefined SYS_getdents64

Apply this patch for Python issue 20307.

epoll_create1

If you get:

undefined EPOLL_CLOEXEC

(I tried a hack to define that variable, only to find another error:

undefined epoll_create1

so my eventual fix was to comment out a line in pyconfig.h:

/* #define HAVE_EPOLL_CREATE1 1 */

Apparently, the linux system call epoll_create1() is not implemented on Android?  And EPOLL_CLOEXEC is only needed if epool_create1() is used.

log2()

If you get

undefined log2

then in pyconfig.h make these changes:

/* #define HAVE_LOG2 1 */
#undef HAVE_LOG2

That controls whether m_log2() calls the system defined log2() or implements log2() de novo.  Apparently there is no system log2() implemented on Android?

 

Next:

Part 3 will discuss:

  • debugging your app on an AVD
  • packaging using Qt Creator

http://qt-project.org/doc/qtcreator-3.1/creator-deploying-android.html

The pyqtdeploy directory contains a file DeveloperNotes.txt.  This seems to be a discussion for a developer of pyqtdeploy (someone who is maintaining pyqtdeploy or wants to add a feature to it) but it also has useful hints about Android development.

Notes on i18n localization of Qt5 using PyQt

These are brief notes about localization of PyQt (or C++) programs using Qt5.  Use with caution, I’m learning too, and this could be wrong.

Localization is also known as:

  • i18n
  • internationalization
  • l10n (common on Qt sites)

The state of Qt translations

Briefly, they are haphazard.  Because of the transition from Qt4 to Qt5, many translations files are waiting to be ‘blessed’ by a third party translator.  You might need to use a mix of files from Qt4 and Qt5.

This page discusses how to contribute translations to Qt.  If you go there, this link takes you to the current translation files and shows you their percentage completion (whether they have been blessed by a person who knows the native language.)  That link also lets you download the current (or future) translation files.

Note that the ‘future’ translation files  (for Qt 5.4 as of the time of this writing) are full of translations, but all are marked ‘unfinished’  (until someone blesses them, sometime during the release of Qt5.4?)

Differences between Qt4 and Qt5 translation files

In Qt4, all translations were in one monolithic file but in Qt5 they are split up into a file per Qt module.  For example, in Qt4 translations were all in qt_xx.ts but in Qt5 translations for QtCore and QtGui are now in qtbase_xx.ts.  This applies both to the templates (.ts) and compiled translations (.qm).

In Qt5, there still exists a translation file having the same name as the Qt4 monolithic file (qt_xx.ts or qt_xx.qm) but it ‘includes’ (in other words references) the split files (e.g. qtbase_xx.ts and qtdeclarative_xx.ts).  In Qt5, that file is called a ‘meta catalog file.’  (Such a .ts file will not open in Linguist; Linguist says it is empty.)

So in your app code, you can still load a translator for the one name.  As discussed here, for C++ it recommends to concatenate .qm files into one file having that same name.

However, for a PyQt app that only uses one module, it might make sense to load a translator with only one file, e.g. qtbase_xx.ts.  But because of the transition from Qt4 to Qt5, a ‘blessed’ qtbase_xx.ts file may not exist.  You might just rename the Qt4 monolithic file qt_xx.ts to qtbase_xx.ts.

It is part of your build process to compile the template files (.ts => .qm) and to concatenate them into one file (or not.)  Your buld process may include a step of downloading the template files (.ts) for Qt library.

In the files at that link for Qt5.3, there are some files such as qt_pt.ts that are the ‘meta catalog file’ but the included split files are not present!  So you must be careful where you get your files: sometimes qt_xx.ts is a Qt4 file of translations, and sometimes it is a Qt5 ‘meta catlog’ that Linguist says is empty.

Understanding the gitorious Qt website

Warning: I don’t really understand it yet.  If you open the qt/qt5 repository, it doesn’t have much in it.  If you read the Qt Wiki about building Qt5 from source, it says you first clone that repository, and then run a script that clones other repositories.  That’s probably the best way to get Qt5 source. (But for example, the web site says the repository will have all branches, but I found that it only has the 5.3 branch (the stable branch), and is missing the ‘dev’ branch described by the Qt Wiki about Qt’s branching model ?)

At the gitorious qt website, if you open the qt/qtranslations repository, it appears to contain Qt4 translations files.  I could be wrong.  As evidence, the files there seem to be missing the ‘QPlatformTheme’ context, which is a Qt5 innovation.

Reiterating, you must be careful where you get your translations files (.ts).

What worked for me

I downloaded qtbase_xx.ts files from this link (an ftp site) under the ‘5.4’ section (which are ‘future’.)  The files seems to be complete in that they have all the user-facing texts from the Qt5 source code.  However, they are all (?) marked unfinished.  As mentioned earlier, they are awaiting the 5.4 release and ‘blessing.’

But I ran an awk script to mark “finished” all translations that have non-empty translated text.  Its not politically correct, since no human has reviewed the status of the translations.  Some of the translated texts may themselves have been generated by a program (by the conversion from Qt4 to Qt5) and might not be correct.  But I found that at least for my app, which uses a minimal but critical subset of the strings, it is better than the alternatives.  The critical set that I need are the strings in the context ‘QPlatformTheme’, which makes standard buttons in dialogs use the platform’s theme.  (This also very much depends on what language you choose, for example German may already be blessed and complete, but Portuguese not.)

The awk script:

# gawk script to mark translations finished in a .ts file

# Usage: gawk -f finish.nawk foo.ts > tmp.ts

# unfinished translations with non-empty text become finished
# Note .+ matches at least one character
/"unfinished">.+<\/translation/ { 
    sub("unfin", "fin")
    }

# print every line (some modified by earlier pattern)
{ print }

 

Cut and paste can garble a template file!

I found the hard way that you must be careful: you can’t browse to a ‘raw blob’ template file at the gitorious qt website and cut and paste it to your computer.  Somehow it garbles the Unicode encoding.  You are better off downloading the files (say using the link above.)

The relationship between KDE and Qt

KDE (an alternative desktop to Gnome) is alive and well in Europe.  It uses Qt instead of GTK as its windowing framework.

Unfortunately, KDE does their own translations of Qt user-facing strings.  They use their own i18n system, including another format (.po) file.   I don’t think they base their translations on Qt translations, and have little interest in updating the Qt translations.  There doesn’t appear to be much flow of translations from KDE back into Qt.  (Its another case of duplication of effort, and difficulty of coordination?)

Japanese still is not well supported on Ubuntu

I especially had trouble with Japanese language on Ubuntu. For example, in Linguist, many characters in translated texts don’t display properly.  I think it is mostly to do with fonts.  It might depend on many factors, such as whether you have exported LANGUAGE=ja in your shell environment, whether you have used System Settings>Language to set the language, whether you have rebooted, and what fonts you have installed.  I struggled to find a combination that was stable.

Bottom line: use something other than Linux/Ubuntu (a Mac?) to use Linguist to translate Japanese (until the mess is straightened out.)