What’s Chris Doing?

If you are working with Python in an embedded environment, you will enventually want to build a Python extension with a cross tool chain. These instructions work both with Open Embedded and the DENX ELDK. If you need to go cross compile Python 2.5, 2.6 or 3.1. ELDK and Open Embedded include a version of Python pre-done for you. You will need a version of Python for the build system that matches your target enviroment.

On the host machine install setuptools and distutilscross. If you are using Python 2.6.3 you probably need to also install distribute which is a replacement for setuptools that also happens to fix a conflict between setuptools 0.6c9 and Python 2.6.3 (There is an update of setuptools coming soon too. See the update below). If you want to use virtualenv and distribute together, you might need to use virtualenv-distribute, a fork of virtualenv that uses distribute instead of setuptools.

Distutilscross is a package that uses setuptools to monkey patch an extra parameter, –cross-compile, into the distutils build command. The net result is that you can set your cross compile environment variables, plus PYTHONXCPREFIX and do python setup.py build -x to get your extensions to compile.

Lets take ELDK as an example.

$ export PYTHONXCPREFIX=/opt/eldk/ppc_4xxFP/usr
$ export CROSS_COMPILE=ppc_4xxFP-
$ export CC="${CROSS_COMPILE}gcc -pthread"
$ export LDSHARED="${CC} -shared"
$ export LDFLAGS="-L/opt/eldk/ppc_4xxFP/usr/lib -L/opt/eldk/ppc_4xxFP/lib"
$ python setup.py build -x bdist_egg --plat-name=linux-ppc

We always need to define PYTHONXCPREFIX and LDFLAGS. PYTHONXCPREFIX tells the -x (or –cross-compile) where your target built Python is. Mostly it is looking for the options used at build time from $PREFIX/lib/python2.x/config/Makefile. This will also set the compile up to use the target Python’s headers. We need to set LDFLAGS as above because Distutils always generates the link directory options as pointing to the Host Python’s install location.

We also need to set CC and LDSHARED because the compiler guessed out of the target Python Makefile is wrong for ELDK. If you are compiling C++ code you would also need to set the CXX environment variable.

Once you have the egg built, you can copy it over to a package directory (read an auto-indexed directory on a web server) and install it from your target using:

$ easy_install -f http://mylocalpackageplace/nest package_name

Concrete example time, what do you need in order to build the deps for and install TurboGears 2. You need the following packages:
zope.interface, Genshi, and simplejson. ELDK’s installation of Python 2.5 does not seem to come with sqlite so you will either need to do a new cross compile of python or get a separate install of the sqlite library. Once you have your binary eggs built using the above command copy them to your local http server, I’ll use 192.168.1.99. You need to follow the Manual Installation method for TurboGears but when you get to Install TurboGears 2 add “-f http://192.168.1.99/nest/” to the command line so that you get:

(tg2env)$ easy_install -i http://www.turbogears.org/2.0/downloads/current/index -f http://192.168.1.99/nest/ tg.devtools

This is sufficient for most normal extension building situations. Unfortunately there are some extensions that use autotools and python-config instead of distutils (dbus-python and omniORBpy come to mind). Autotools should let you override the critical settings but I would like to see the situation improve.

Let me know if you have ideas on how to make Python development in an embedded environment better.

Update: PJE, the author of setuptools, has given notice of the impending release of setuptools 6c10 which fixes many bugs, including the Python 2.6.3 compile bug. See his comment below.

We recently launched the CIOC Community Site running on Plone. While we have not actually had any SPAM on the site yet, we continually get automated SPAM registrations that we then need to go and clear out. Last night Kate got fed up, so I decided to see if I could put a CAPTCHA on the form.

Looks like there is a proposal to include CAPTCHA support. The de facto standard CAPTCHA tool for Plone seems to be collective.captcha, but I wanted to be able to use the ReCAPTCHA service.

Luckily there is collective.recaptcha which is in beta and, while not listing any releases on the Plone site, is available from the Cheese Shop. But, alas, there is no documentation. It is supposed to be a drop in replacement for collective.captcha so with a little sleuthing and I got it to work. Here’s how.

Installation

I am using the unified Unix installer so everything is done via buildout. Following the instructions on the collective.captcha page we need to add collective.recaptcha to your eggs list and collective.recaptcha to your zcml list. Then run ./bin/buildout and restart your Plone instance.

Configuration

We need to enter our private and public ReCAPTCHA keys in the collective.recaptcha setup form which is located at http://urltoyourplonesite.com/recaptcha-settings

Registration Form Modifications

Using this site as a helpful reference I was able to figure out the additions I needed to make to the registration form and validator.

You need to go in through your Plone site setup into the Zope Management Interface. Navigate to portal_skins/plone_login. Click join_form and then the customize button. You should now be able to customize the form contents. Down near the bottom of the form there is a line:

<div class="formControls"></div>

Above that line you need to add:

<div class="field"
     tal:define="error errors/captcha|nothing;"
     tal:attributes="class python:test(error, 'field error', 'field')">
  <label for="captcha" i18n:translate="label_captcha">Captcha</label>
 
  <span class="fieldRequired" title="Required"
        i18n:attributes="title"
        i18n:domain="plone"
        i18n:translate="label_required">(Required)</span>
 
  <div class="formHelp" i18n:translate="help_captcha">
    Provide the text in the image. Just to avoid spambots
  </div>
  <p tal:replace="structure here/@@captcha/image_tag" />
 
</div>

Note that the difference from the code provided by Mikel Larreategi is that ReCAPTACH provides the input element itself, so you need to omit the div that includes the <input type="text"> tag.

Once you have that saved, you need to go back to portal_skins/plone_login and click join_form_validate and once again click customize. At the bottom of the validation code, just before

return state

add:

captcha = context.REQUEST.get('recaptcha_response_field')
view = context.restrictedTraverse('@@captcha')
 
if not view.verify(captcha):
    state.setError('captcha', _(u'Are you a bot? Try again...'))
    state.set(status='failure')

Note that the difference in the validation code from Mikel Larreategi’s is that the ReCAPTCHA inserted input tag is called recaptcha_response_field and not captcha.

Hopefully that is helpful to someone other than me :)

Part of MichaÅ‚ Kwiatkowski‘s Summer of Code project was the to create the Cheesecake service. Micheal explains how it all works in his announcement of the cheesecake service.

There was a lot of complaint about the concept of Cheesecake when it first appeared on the scene, but I think the concept is really valuable. Check and see if your Pyhton package is easy_install’able.

To all the people who continued on with Cheesecake in the face of criticism: Thanks!

This announcement found via this post.

Recently I needed to compile Python for an embedded target. I used version 2.5 though it looks like that choice may have made it harder for me. I used 2.5 because I didn’t want to have to figure out how to cross compile the cElementTree extension. Unfortunately I still ended up having to figure out how to get PyXML to build for my target. Fortunately I did get everything to work with a bit of fiddling. For posterity, here are some notes about what I did and what problems I had.

I started with Klaus Reimer’s ARM cross-compiling howto and made made some updates required by changes between Python 2.2 and 2.5.

The changes I made are captured in an updated patch to apply against the 2.5 source tree. The changes made the configure.in are to disable rules that cause configure failures when cross compiling because they look for files on the target system or require a test program to be compiled and run. The other changes I added over the patch from Klaus Reimer are for a specific issue I had where md5 and sha hash algorithms were not built because setup.py which builds the modules is not cross compile aware and detected the OpenSSL libraries on my build machine rather than the target.

You can apply the patch with the following command:
lambacck:~/tmp/Python2.5$ patch -p1 < ../Python2.5_xcompile.patch

I next generated a "host" python as in Reimer's instructions:
lambacck:~/tmp/Python2.5$ ./configure
lambacck:~/tmp/Python2.5$ make python Parser/pgen
lambacck:~/tmp/Python2.5$ mv python hostpython
lambacck:~/tmp/Python2.5$ mv Parser/pgen Parser/hostpgen
lambacck:~/tmp/Python2.5$ make distclean

I exported necessary variables like CC, CXX, AR, RANLIB, LD, CFLAGS, CXXFLAGS, LDFLAGS for my target. The key is for these to point to the libraries you are going to use.

To build and “install” python for the my target I used the following commands:
lambacck:~/tmp/Python2.5$ ./configure --host=ppc-linux --build=i686-pc-linux-gnu --prefix=/python
lambacck:~/tmp/Python2.5$ make EXTRA_CFLAGS="$CFLAGS" HOSTPYTHON=./hostpython HOSTPGEN=./Parser/hostpgen BLDSHARED='ppc-linux-gcc -shared' CROSS_COMPILE=yes
lambacck:~/tmp/Python2.5$ make install EXTRA_CFLAGS="$CFLAGS" HOSTPYTHON=./hostpython BLDSHARED='ppc-linux-gcc -shared' CROSS_COMPILE=yes prefix=/home/lambacck/tmp/dest/python

Note that I needed to use EXTRA_CFLAGS to add my target specific CFLAGS because for some reason the Python configure process does not honor the ones I provided while doing configure. The LDFLAGS variable, however, did work.

Also notice that I set the prefix in the configure step to be /python and the set another prefix in the make install step. The prefix in the configure step is where we are putting python on the file system on the target. The prefix in the install step is where we are putting all of the bits to be able to package them up to be ready to send to the target.

After all that I have a (mostly) functional python environment on my target, but I still needed to get PyXML built. I downloaded the latest distribution, modified setup.py so that expat was forced into little-endian mode and ran the following commands:
lambacck:~/tmp/PyXML-0.8.4$ LDSHARED='ppc-linux-gcc -shared' CC="${CC}" OPT='-DNDEBUG -g -O3 -Wall -Wstrict-prototypes' ~/tmp/Python-2.5/hostpython -E setup.py build
lambacck:~/tmp/PyXML-0.8.4$ LDSHARED='ppc-linux-gcc -shared' CC="${CC}" OPT='-DNDEBUG -g -O3 -Wall -Wstrict-prototypes' ~/tmp/cross_python/Python-2.5/hostpython -E setup.py install --prefix=$HOME/tmp/dest/python/ --install-scripts=/home/lambacck/tmp/dest/python/bin

It looks like Distutils and the Python build process in general could use some cross compile support. I think this is currently far harder than it has to be.

Update Oct 10, 2005: I would like to point out that on the system that I the executable used to compile hostpython was called gcc and I then updated paths so that a gcc targeted at ppc called gcc was first in my path. Therefore in both cases the compiler was gcc and not gcc for the hostpython and ppc-linux-gcc for the target python.

Update Dec 6, 2006: It looks like double dashes have been turned into single dashes by WordPress. Specifically the host, build, prefix and install-scripts arguments to configure and setup.py should have double dashes.

Update Jun 30, 2010: Translation to Portuguese. Commands converted to use arm processor. Thanks Anuma.

I previously talked about build environments in Windows and promised to do some more testing of the reliability of certain tools. I turns out that the only set of tools which actually works as expected are the ones provided by Cygwin. I could not get the AT&T UWIN tools to install and they are just as heavy weight as Cygwin, which at least provides BASH. I thought the GNUWin32 tools had promise, but unfortunately they have problems with quoting, and don’t provide make. I have also determined that MSys has similar quoting problems and a very outdated make.

There are a handful of other options, but Cygwin just seems to be the best. The real downside to Cygwin is that its setup tool that has a UI more suited to torturing its users than letting them install packages. Luckily someone realized that Debian got the model mostly right and created an apt like tool for Cygwin.

The site I linked to above suggests installing rxvt. The command they give to replace the Cygwin Bash shell link needs some help try adding -e /bin/bash --login -i to the end of it.

The comments over here also provide some potential terminal emulator and shell alternatives for Windows.

Recently I have been playing a bit with makefiles on win32 and Unix. I want to be able to accommodate a couple of build environments without using automake, configure and friends. I am also following Peter Miller’s suggestions from Recursive Make Considered Harmful.

The first step to doing this is removing reliance on external tools. There are a couple of things you can rely on, but not very many; the least of which is shell syntax. Very helpful is the GNU Make Standard Library (GMSL). While I have only used a couple functions so far, it hints that there is much that can be done right in the makefile that many people rely on external tools for. The GMSL for instance provides a tr(translate) command and associative arrays.

Some of the things I am currently finding most useful are the gmake built in functions ‘define’, ‘call’ and ‘eval’. These three constructs allow for some very interesting techniques (avoid using spaces between arguments or you will get a cryptic missing separator error). For those of you who are looking for ways to deal with some standard win32 compatibility things here are a couple of snippets:

ifeq ($(notdir $(SHELL)),cmd.exe)
#make didn't find a bourn shell.
#We probably don't have rm, cp or a proper mkdir either
CP=copy $(subst /,\,$(1))
RM=for %f in ( $(1) ) do if exist %f del %f
MKDIR=mkdir $(subst /,\,$(1))
else
#make found some variety of bourn shell,
#assume that we have rm, cp and a sane mkdir
CP=cp $(1)
RM=rm -f $(1)
MKDIR=mkdir -p $(1)
endif

In order to use the piece above you need to get out of the habit of doing:
clean:
  $(RM) file1 file2 file3

And use the following instead:
clean:
  $(call RM, file1 file2 file3)

This probably will require some retraining to be consistent about it, but ultimately will make your makefiles more easily cross platform.

My unfortunately discovery is that not all gmake versions are the same. I know of 3 separate sources of gmake for win32: MinGW, MSys and Cygwin. The make from MSys is the least useful; it is an old version (3.79.x) and does not seem to handle define properly. The make from Cygwin works beautifully, except that when directories change, it uses its /cygdrive/c/path notation, which is more than a little annoying if you are trying to catch directory changes in another tool like VIm or Emacs. The make from MinGW may be slightly more useful in that define works, but I was having trouble with the test suite. Thinking about it now, it is possible that I may have messed things up with pathing. I will have to check that and report back.

While I am talking about makefiles and compatibility on win32 I might as well also talk about my discoveries about sed. The goal of my work was to require only gmake, a compiler and sed. As it turns out not all versions of sed are created equal, much like the problems with make described above. Apparently the sed that ships with MSys is doing some strange things with backslash escaping. The following should work identically using either Bourne shell or cmd.exe:

target:
sed -e 's/ *\\ *$$//' < file.in > file.out

This works with the sed from http://gnuwin32.sf.net/ but not with the GNU sed that ships with MSys. Strangely the following works with the MSys sed:

target:
  sed -e 's/ *\\\\ *$$//' < file.in > file.out


Maybe I should just cut my losses and add a * since given the data, I know I will never have more than one backslash:

target:
  sed -e 's/ *\\\\* *$$//' < file.in > file.out

That should make work with either sed, but it is not very helpful for the general case.

Update:
I just looked again at http://unxutils.sourceforge.net/, who’s make I previously discounted because the version listed is old (3.78.1), and noticed that the news section says they have updated to 3.80. I will try the GNU make and sed from this package and see what I get.

Update 2:
It appears that just MSys is broken. Both the make from MinGW and the make from UnxUtils both pass the gmsl-test once I fixed my path problem. I still noticed some thing funny about those two make programs, Cygwin’s make continues to work flawlessly.

The last sed command, above, does not work, and the sed provided by MSys broke several sed scripts that I was using. I don’t know if taking the scripts from a file would work better since there would be no escaping issues. At this point I recommend staying away from MSys.