Applying image algorithms to music

About

This is an idea that interests me.  I seek feedback, references, and collaborators.  I have considered in implementing this in Python.  My background is in image algorithms, and not so much music.

Summary

Some image algorithms can be generalized and applied to music, if music is represented like image pixmaps.

Music maps

A Map is a 2-D addressable array.

A MusicMap is a Map where the addressable unit is a Musicel, like a Pixel.

A Musicel comprises Musicelels, like Pixelels.

Typical Musicelels:

  • pitch
  • volume
  • waveform (attack/decay)

Visually, if you take a sheet of music and divide it into quads, where the size of quads is such that there is only one note  in it, you have a MusicMap.  Some quads contain silence notes.  I gloss over that all notes are not the same duration, you can spread  a long duration note into many Musicels.

(A PixMap is a 2-D array of Pixels comprising Pixelels i.e. RGB values.)

Comparing the self-similarity of PixMaps and MusicMaps

In a Pixmap, each Pixel is related (correlated?) to neighboring Pixels with the same strength.  In a MusicMap, there is less relation between a Musicel and the Musicels above and below it (in the same measure on the lines of music above and below.)

In many images of the real world, there might not be much relation between the top and bottom of an image.  In music, generally there is some relation between all large parts of the piece, i.e. the scale, and the set of chord changes.  Restating, some images (having man-made objects as opposed to natural objects) are not very fractal throughout, whereas music is usually fractal throughout.

Generalizing image processing algorithms to music

An image processing algorithm that works on Pixmaps can be generalized to work on MusicMaps.  You simply make the algorithm polymorphic on the base class of Pixels and Musicels ( call the base class Cel.)

The Cel class might have these methods:

  • compare(self, other)
  • average(self, other)

More generally there would be many methods for reducing two Cels to one, e.g. min(self, other)

An example

Consider a blur algorithm, which reduces the resolution of an image (or music.)   Suppose a symphony of 1,024 lines of 32 notes each.  A blur algorithm applied to the symphony might reduce it to 32 lines of one note each (by a factor of 32.)

Music is played/heard linearly, unlike an image which is parallel processed by the eye/brain.  Still, if you played the reduced symphony, you might get a blurred sense of it.  Suppose the first movement of the symphony used mostly staccato, short, high notes.  That might come through in the blur.

Continuing along this line of thought, there are blur algorithms that are better at retaining ‘features’, knows as image summarization algorithms.  These are used for example to compress images to thumbnails.  Can you create a thumbnail for music, that is not just a sample?

Which image processing algorithms are interesting when applied to music?

Image processing algorithms are categorized as ‘structural image editing’ algorithms seem most applicable to music.  For example:

  • summarization (reduce resolution while retaining features)
  • context sensitive paste (inserting a sample that matches nearby context)
  • reconstruction (filling in holes to match nearby context)
  • collage (making one from two)
  • alteration detection
  • texture transfer (applying texture from one to another)

There is a foundational algorithm called bidirectional similarity that is useful in the structural image editing category of image processing algorithms.

Silence and transparency

There is another interesting question here, about generalizing silence in music and transparency in images.  I think they are analogous, and that many concepts from image processing, such as pre-multiplied alpha, could be carried over.  For an example of the question: is a silence represented by a duration Musicelel having value zero, or represented by all its Musicelels having value zero?

Using pyqtdeploy on MacOS to cross compile a PyQt app for iOS: part 3

Rehash

This is part three of three parts:

  1. install Qt and Xcode, then test build and deploy an example Qt app for iOS
  2. cross compile static libraries for iOS (using pyqtdeploy.)
  3. use pyqtdeploy to cross compile and package your app.

Overview

As discussed in Choosing a Toolchain, there are many possible toolchains.  In previous parts, we only compiled static libraries for the ARM architecture of real iOS devices.  In this example, we will use the pyqtdeploy/Xcode/real iOS device toolchain.

The steps are:

  • use pyqtdeploy GUI
  • sign up and pay for an iOS developer
  • connect and configure your real iOS device for development
  • use Xcode to deploy
  • run app on real device

Pyqtdeploying your app for target iOS

Here you use the pyqtdeploy GUI app to prepare projects, makefiles, and source code.  This is very similar as for any target.

Connecting and configuring your real iOS device

Hook up your device with a USB cable.

What follows is highly automated.  Xcode should start and display the Organizer, featuring your device

Briefly, wait for it to process a while, then choose the ‘Configure for Development’ button, and so forth.

(Dated, confusing, and slightly conflicting instructions are given here.)

Compiling and packaging your app for target iOS using Xcode

Assuming you ran qmake step in pyqtdeploy, it created an xcodeproj.

Open that project in Xcode.

Configure Xcode to deploy to your real device: click in the left side of the title bar where it may say ‘Foo project>iosSimulator’ ??.  A list of devices (real and simulator) should appear.  Choose a real device.

Choose Product>Run.  A few dialogs may appear:

  • allowing codesigning to occur
  • to tell you to unlock your sleeping device using its four character passcode.

This deploys your app, updating it if it was deployed previously.

This also runs your app, displaying stdout and stderr within Xcode.  Your app will become active (displaying its windows) on the device.  Touch the device to generate input events and debug.

Run your app later

On the real device, pan left and right in the display of apps to find your app.  (It probably has a default, nondescript icon.)  Tap to run it.

FAQ or Notes

No QPrintSupport module of Qt.  If you import PyQt5.QtPrintSupport, you get an error at link time.  Unlike on Android where I did not seem to get a link error (but only a crash?)  Apparently Qt does not yet support printing on mobile devices.

Debug on iOS first.  In my opinion, because Xcode generates better error messages at link time, and because Xcode displays your program’s stdout and stderr in a window (unlike using QtCreator on Android? I could not easily get it to work.)

If your app was designed for the desktop, it may work since the QtApplication flag about translating touch events to mouse event defaults to ‘translate touch to mouse events.’  If you change the default, it seems like some widgets e.g. menus and buttons will not work (they depend on mouse events?)

Notes on pyqtdeploy

You should read the official documentation, but it is brief.  Here, I elaborate:

  • discuss general concepts that seem obvious to experienced developers
  • discuss specific commands

About pyqtdeploy

(See my other platform specific posts, I plan to consolidate sections here.)

Installing and upgrading pyqtdeploy

The pyqtdeploy download page is not really a place to download.  It says that you can:

  • install the latest stable release using pip3
  • install the latest development version using mercurial (hg)

Managing pyqtdeploy using pip3

Installing pip3

pip3 is needed to install stable pyqtdeploy

  • OSX: pip3 is installed with Python3.
  • Ubuntu: if pip3 is not already installed:
sudo apt-get install python3-pip
Installing latest stable pyqtdeploy
pip3 install pyqtdeploy

(If that fails on permissions: sudo pip3…. )

Upgrading stable pyqtdeploy

Pyqtdeploy is released frequently.  To upgrade to the latest stable release.

>pip3 install pyqtdeploy --upgrade

Soft linking to pyqtdeploy

It seems that the above does not put pyqtdeploy and pyqtdeploycli executables in the path.  You might need to make soft links:

>cd /usr/local/bin
>sudo ln -s /Library/Frameworks/Python.framework/Versions/3.4/bin/pyqtdeploycli pyqtdeploycli
>sudo ln -s /Library/Frameworks/Python.framework/Versions/3.4/bin/pyqtdeploy pyqtdeploy

Managing your development version of pyqtdeploy using Mercurial

Note that mercurial uses Python2.7 but pyqtdeploy uses Python3.

Installing pyqtdeploy using hg

>hg clone http://www.riverbankcomputing.com/hg/pyqtdeploy
>cd pyqtdeploy          # to the directory you just cloned
>make VERSION           # ? not sure why next two steps are not dependencies in the Makefile
>make pyqtdeploy/version.py
>sudo python3 setup.py install

Again note that we install in python3 site packages.

Updating your development version of pyqtdeploy
>cd ~/Downloads/pyqtdeploy*
>hg pull
>hg update
> ??? make VERSION
>sudo python3 setup.py install
Installing Mercurial

You need Mercurial to download development version of pyqtdeploy.  Mercurial is a version control system.  Also known as the ‘hg’ command.

The Mercurial downloads page.

Note that Mercurial uses Python 2.7 and doesn’t support Python3 but that’s usually OK since Python-2.7 is usually installed.

Pyqtdeploy is now two tools

Starting with version 0.6, there are separate tools:

  • pyqtdeploycli: a command line tool for building static libraries
  • pyqtdeploy: a GUI tool for building your app

(I suppose you could also use pyqtdeploycli to build your app, passing it a configuration file.  But using the GUI pyqtdeploy is much more intuitive and easy.)

Disambiguating deploy

The pyqtdeploy tools don’t actually deploy.  They just prepare for deploy.  They may create or touch:

  • source files (.c, .cpp and .py)
  • qmake projects (.pro)
  • make files (Makefile)

Subsequently you must continue to build using qmake, make, and other tools such as Qt Creator and Xcode.  Then you have a binary, executable app.

Qt Creator does not deploy to the App Store

You must read QT Platform Notes – iOS very carefully.  It says that Qt Creator will do everything for building a Qt iOS app except:  deploy to the App Store.

You don’t need to use Qt Creator, you can use only command line tools (and then possibly the Xcode GUI.)

But this is related to the difference between the iosSimulator and a real device: they use different ABI architectures.  You should think early about whether you will use Qt Creator, and whether you will use the iosSimulator or a real iOS device.  You must fork the process of building early, when you are building static libraries (for either iosSimulator/ i386 architecture, or a real device/ ARM architecture.)

(Will Qt Creator deploy to Google Play store?)

Choosing a toolchain

This is related to choosing a machine for deployment.

Ideally, you learn and use one toolchain that works for every platform (avoid the learning curve for some tools.)

  • use pyqtdeploy/qmake/Xcode/real iOS device for iOS
  • use pyqtdeploy/qmake/Xcode/real Android device for Android

But for Android I have also used, with mixed results:

  • pyqtdeploy/Qt Creator/real Android device
  • pyqtdeploy/qmake/command line scripts/real Android device

I’m not sure whether I need to learn and use the emulator/simulator tools to do sanity testing across device models (e.g. iPhone vs. iPad) and OS versions (e.g. iOSv7 vs iOSv8.)

Cross platform software development

These are not entirely general notes, but often specific to using:

  • Qt
  • PyQt
  • pyqtdeploy

GUI considerations for crossing mobile and desktop platforms

This doesn’t make sense unless your app’s GUI is carefully designed to be portable across mobile and desktop platforms.  I call  this a ‘universal user interface.’

A desktop has, and a mobile platform doesn’t have (?):

  • multiple overlapping windows
  • menubar
  • a file system and file browser

A mobile platform has, and a desktop may not have:

  • multitouch and gestures

I’m still experimenting here (but this a principal reason for my app.)  I don’t think that Qt provides a magic bullet.  That is, everything you can do in Qt may  not be portable and you still might need to have platform specific (mobile versus desktop) code in your app.

(And Qt encompasses many tools, such as Qt Quick, which I am ignoring here, but which may be more portable.)

Qt considerations for crossing mobile and desktop platforms

Printing: Qt does not currently seem to compile the QtPrintSupport module for the mobile platforms.

See QTBUG ????

Early versions of Android and iOS also did not support printing.  Now they do:

  • iOS: AirPrint
  • Android: ???

I suppose it just takes time for Qt to adapt to the mobile print frameworks.  How do they differ from desktop print frameworks?

How many dev machines do you need for cross platform development?

Suppose your software development process includes these steps:

  • development (creating new features and fixing bugs)
  • deployment (packaging)
  • testing

You might want a machine dedicated to each step (so you don’t need to worry about cross contamination.)

Probably you want a Mac for your deployment machine.  A real Mac can deploy to iOS, Android, and MacOS.  You can also create virtual machines (Ubuntu and Windows) on that Mac so that it can also deploy to those platforms.  In other words, a Mac can deploy to all platforms supported by Qt.

The reverse is not true.  No other platform will let you deploy to iOS since you are not legally permitted to create a MacOS virtual machine on anything other than a real Mac machine, and since Xcode runs only on a Mac, and since only Xcode lets you develop for iOS (with exceptions such as HTML5/Javascript?)

Note that Qt does not currently support WindowsRT (Metro).  That’s of little use anyway, since that platform is trailing in the market?

The latest pyqtdeploy user guide has a discussion about directory structures that enable sharing of data across target deployment processes.

Favoring Qt over Python modules for crossing platforms

In your code,  you should favor using Qt classes over using Python modules.  For example, use Qt’s support for file system operations over Python’s support.  More specifically, try not to use the os module (which drags in many other Python modules) of Python.

The reason is, as of this writing, the Python organization does not support mobile platforms.  You may have difficulty cross compiling Python modules to mobile platforms (which you must do when you use pyqtdeploy.)  Also, you will package Qt libraries with your app, and also packaging Python modules that do the same thing is superflous.

Emulators and Simulators for Mobile Devices

To test a mobile app without a real device you can use:

  • Android emulator (AVD)
  • iOS simulator

Note the difference.

An AVD is an emulator, presenting the ARM ABI (the interface of the target hardware).  It executes the same binary that will run on a real device.  Pragmatically, you only need to compile your app once, for the ARM ABI.

The iOS simulator is a simulator, presenting the iOS API (the interface of the target OS.)  It executes a binary that is compiled for the i386 ABI, which is not the same ABI as real devices (ARM.)  Pragmatically, this means you need to compile your app for both ABI’s  (if you want to test on the simulator.)

Deploying PyQt Python Qt apps cross platform using pyqtdeploy

These are general notes and links for this subject.  In other words, this is a table of contents.  Many links are often specific to a target platform.

Some of the links to my posts are to incomplete drafts.   Also, I edit these blogs periodically, since pyqtdeploy is actively developed and I am learning too.  It is best to consult source documentation.

Cross-platform and Localization/Internationalization/i18n

You want the largest possible audience or market.  You are using Qt because it:

  •   is cross platform.
  •  is cross language: supports localization ( internationalization)

You increase your audience by porting to new platforms AND by localizing to new languages.

Contents

Original (source) documentation

pyqtdeploy v0.6 User Guide

Qt for iOS

Qt for Android

General

Cross platform software development

More notes about pyqtdeploy

Deciding between Xcode and QtCreator for packaging PyQt apps for OSX

Target Linux and MacOS

Using pyqtdeploy

MacOS bundling (packaging) a pyqtdeployed app using Xcode

 Target iOS

iOS part 1 (prepare tools)

iOS part 2 (cross compile static libraries)

iOS part 3 (cross compile app and deploy)

 Target Android

Android cross compiling Qt example C apps

Android part 1 (prepare tools and cross compile static libraries)

Android part 2  (cross compile app)

Android part 3 (deploy and debug) DRAFT

Localization

General notes

Localizing PyQt app for target OSX using Xcode

More localizing PyQt app for target OSX

Using Qt Linguist phrasebooks

Adventures in translating an app to Chinese

Universal user interface

Commands that support gestures

Using pyqtdeploy0.5 on Linux to cross compile a PyQt app for Android: Part 3

Rehash

You have:

  1.  set up your dev machine and built static libraries
  2. pyqtdeployed your app so it is ready to deploy

This part discusses:

  • deploying your app to a test target machine
  • debugging your app

More about pyqtdeploy work flow

The pyqtdeploy GUI lets you choose your work flow.  That is, in the ‘Build’ step, you can choose optional steps: qmake, make, ‘run app’.  For example, you can choose none of those, and Build will only create a .pro file.

Then you can switch to:

  • the command line
  • or Qt Creator (a GUI app)

and resume the work flow (qmake, make, deploying, and running.)

(I’m haven’t tried  ‘run’ in pyqtdeploy for target Android, I always switch to another tool.)

pyqtdeploy developer notes

The pyqtdeploy directory contains a file DeveloperNotes.txt.  These notes are by the developer of pyqtdeploy  but are useful hints about Android development.  The author of the notes has said the notes are NOT definitive and should be used with care.  I have found  typos there.

Deploying using the command line

Follow the instructions in the developer notes.   I.e. :

# Assert you ran qmake and make in pyqtdeploy, but not run

# cd to dir that pyqtdeploy created
cd ~/hellobuild/build
export ANDROID_NDK_ROOT=$HOME/android-ndk-r10
export ANDROID_SDK_ROOT=$HOME/android-sdk-linux
# configure install
make install INSTALL_ROOT=deploy
# make deployable
/home/bootch/Qt/5.3/android_armv7/bin/androiddeployqt \
 --input android-lib*.so-deployment-settings.json \
 --output deploy
# deploy
adb install -r deploy/bin/QtApp-debug.apk
echo Now go to AVD and run app

This only installs your app to a target device.  You subsequently must use the target device’s GUI to run your app.

If you get:

Failure [INSTALL_FAILED_ALREADY_EXISTS]

this means you omitted the -r flag, which means reinstall.  An alternative is to manually uninstall the app first (on the device, choose Settings>Apps, then select your app, then choose the Uninstall button.)

Deploying using Qt Creator

When you choose the ‘Run’ icon in Qt Creator, it builds, deploys, and runs your app.  So it is more automated, but harder to understand what Qt Creator is actually doing on your behalf.

Debugging pyqtdeployed apps on android

First, refresh the tools

The tools (especially pqytdeploy, SIP, and PyQt but also Qt and Python) are updated frequently.  You should use the latest versions, which may have fixed some Android crashes.

Remember that pyqtdeploy is the tool that helps build static Python, etc. libraries.   So if you update pyqtdeploy, you should probably rebuild static Python starting with invoking pyqtdeploy (so that any new patches known by pyqtdeploy are applied.)

Stderr and Android logcat

Unfortunately, Python prints exceptions to stdout/stderr.  These by default are dedirected to /dev/null on Android (which means they disappear.)

On android, the system log is called ‘logcat’.  To see it remotely:

>adb logcat > logcat.tmp
>Ctl-C
>vi logcat.tmp

Its a circular file.  Fileness means it persists (even across your debugging sessions.)  Circular means it wraps around, overwriting older data. Pragmatically, one debug session never seems to wrap around on itself.  The end of the file is always the latest log entries?  You can clear it by rebooting the device (powering it off.)

Redirecting stderr to logcat

You should be able to configure your target device to redirect stdout and stderr to logcat.  To do this requires:

  • a rooted real device
  • an AVD (virtual device)

An AVD is always rooted (you have superuser privileges to bypass all security and do what you want).

Rooting a real device is an adventure unto itself, so I chose not to do it yet.  But the advantage is that a real device eliminates the simulator layer, and is quicker to boot and so forth than an AVD.  You can still do certain debugging on an un-rooted real device, you just won’t be able to see Python exceptions directed to stderr.

!!! If a device is not rooted, the adb commands to redirect stdio fail quietly.  So you still won’t see stderr in the logcat.

To do it you can:

  • issue adb commands
  • create a persistent configuration file

adb commands to redirect stdout/stderr

>adb shell stop
>adb shell setprop log.redirect-stdio true
>adb shell start

Remember this doesn’t persist if you reboot the device (whether and AVD or a real device.)

If the device is rooted real device, you must precede that with >adb root.  (AVD’s are always rooted, so >adb root is not necessary.)

Debugging methods

Traditional choices:

  • use a debugger such as gdb (which is built into Qt Creator)
  • harness code with print statements
  • simple code analysis using crash dumps and Python exception tracebacks

It depends on where the bugs are:

  • in the Python interpreter
  • in your pyqtdeploy build process
  • in your app’s Python code

It is not uncommon to find bugs in the Python interpreter, since the Android platform is not supported by Python.org.  (the pyqtdeploy process patches the Python interpreter, and you might find that another patch is required.)

The pyqtdeploy build process is intricate.  It is not uncommmon to find that you have not frozen a needed Python module (and then you might get an ImportError exception.)

Once you get as far as running your own app’s Python code, you probably have installed an exception hook, and can display the exception and traceback on the GUI of the device. (But probably, you are only porting to Android, and have worked out most of your app’s bugs on another platform, except for any Android specific code.)

Harnessing pyqtdeploy apps on android

I experienced the need to harness (since my app seemed to crash before Qt Creator’s debugger could even get started properly and without any Python exceptions.)

This means writing C code.  You can use:

  • qWarning() C function.  It is properly redirected to Android logcat.
  • __android_log foo C function.

If you need to harness the Python interpreter, you might need to study how to write Python extensions, for example to convert a PyObject * to a representation that is a C string suitable for qWarning().

I harnessed pyqtdeploy_start.cpp and Python/import.c.

Here is the code I used:

TODO

Building a debug version of Python

reading an Android crash dump

packaging using Qt Creator

Localizing (i18n) a PyQt App on OSX using Xcode

INCOMPLETE DRAFT.  I’ve done this process, just haven’t written it up yet.

How to tell Xcode to localize strings in native dialogs of a Qt app.

This is also discussed in many posts on Stack Overflow and in the Qt documentation.

This is about using the Xcode GUI.  If you are using the command line to invoke Xcode tools, or using Qt Creator to bundle your Mac app, the details may not apply.

Checklist of product components to localize

  1. user-facing strings in your Qt code (in the GUI of your app that you wrote.)
  2. user-facing strings in the Qt library (in GUI that the Qt library implements)
  3. strings in platform native dialogs (in the GUI of your app that the platform provides.)
  4. your help files (typically html)
  5. your marketing materials (uploaded to stores)

This blog is most specific about 3.

PyQt and Qt’s user facing strings

The user-facing strings 1 and 2 are embedded resources of your app.  While you could package them as Mac resources, it is more general to package them as Qt and PyQt  resources.  This doesn’t discuss how to do that, except:

  • you use the tools pylupdate, Qt Linguist, lrelease, pyrcc
  • the translation resources end up as a foo_rc.py file in your project
  • PyQt provides the mechanism for finding those resources as needed

The difference between native dialog strings and strings in your Qt code

Qt provides mechanisms to localize strings in the code that it knows about.  But your app, via Qt, can invoke native dialogs, such as the Print dialog.  Qt doesn’t know about the strings in that dialog.

(Qt does know about conventions for naming standard buttons and menu items on certain platforms.   These are in the MAC_APP??? context in Qt .ts files, etc.)

How to tell Xcode to localize native dialogs

  1. Add a localizable string to your project
  2. Add a language to your project
  3. Arrange that .lproj files are in your project and bundled
Add a localizable string

In the Navigator pane, click on the + icon at the lower right (or press RMB on your project and choose “Add New

TO DO