Cross Compiling Python Extensions

October 16, 2009 at 11:07 PM | categories: Computers, Programming | View Comments

If you are working with Python in an embedded environment, you will enventually want to build a Python extension with a cross tool chain. These instructions work both with Open Embedded and the DENX ELDK. If you need to go cross compile Python 2.5, 2.6 or 3.1. ELDK and Open Embedded include a version of Python pre-done for you. You will need a version of Python for the build system that matches your target enviroment.

On the host machine install setuptools and distutilscross. If you are using Python 2.6.3 you probably need to also install distribute which is a replacement for setuptools that also happens to fix a conflict between setuptools 0.6c9 and Python 2.6.3 (There is an update of setuptools coming soon too. See the update below). If you want to use virtualenv and distribute together, you might need to use virtualenv-distribute, a fork of virtualenv that uses distribute instead of setuptools.

Distutilscross is a package that uses setuptools to monkey patch an extra parameter, --cross-compile, into the distutils build command. The net result is that you can set your cross compile environment variables, plus PYTHONXCPREFIX and do python setup.py build -x to get your extensions to compile.

Lets take ELDK as an example.

$ export PYTHONXCPREFIX=/opt/eldk/ppc_4xxFP/usr
$ export CROSS_COMPILE=ppc_4xxFP-
$ export CC="${CROSS_COMPILE}gcc -pthread"
$ export LDSHARED="${CC} -shared"
$ export LDFLAGS="-L/opt/eldk/ppc_4xxFP/usr/lib -L/opt/eldk/ppc_4xxFP/lib"
$ python setup.py build -x bdist_egg --plat-name=linux-ppc

We always need to define PYTHONXCPREFIX and LDFLAGS. PYTHONXCPREFIX tells the -x (or --cross-compile) where your target built Python is. Mostly it is looking for the options used at build time from $PREFIX/lib/python2.x/config/Makefile. This will also set the compile up to use the target Python's headers. We need to set LDFLAGS as above because Distutils always generates the link directory options as pointing to the Host Python's install location.

We also need to set CC and LDSHARED because the compiler guessed out of the target Python Makefile is wrong for ELDK. If you are compiling C++ code you would also need to set the CXX environment variable.

Once you have the egg built, you can copy it over to a package directory (read an auto-indexed directory on a web server) and install it from your target using:

$ easy_install -f http://mylocalpackageplace/nest package_name

Concrete example time, what do you need in order to build the deps for and install TurboGears 2. You need the following packages: zope.interface, Genshi, and simplejson. ELDK's installation of Python 2.5 does not seem to come with sqlite so you will either need to do a new cross compile of python or get a separate install of the sqlite library. Once you have your binary eggs built using the above command copy them to your local http server, I'll use 192.168.1.99. You need to follow the Manual Installation method for TurboGears but when you get to Install TurboGears 2 add "-f http://192.168.1.99/nest/" to the command line so that you get:

(tg2env)$ easy_install -i http://www.turbogears.org/2.0/downloads/current/index -f http://192.168.1.99/nest/ tg.devtools

This is sufficient for most normal extension building situations. Unfortunately there are some extensions that use autotools and python-config instead of distutils (dbus-python and omniORBpy come to mind). Autotools should let you override the critical settings but I would like to see the situation improve.

Let me know if you have ideas on how to make Python development in an embedded environment better.

Update: PJE, the author of setuptools, has given notice of the impending release of setuptools 6c10 which fixes many bugs, including the Python 2.6.3 compile bug. See his comment below.

Read and Post Comments

Protecting Plone Registration Forms with ReCAPTCHA

February 24, 2009 at 10:56 AM | categories: Computers, Programming | View Comments

We recently launched the CIOC Community Site running on Plone. While we have not actually had any SPAM on the site yet, we continually get automated SPAM registrations that we then need to go and clear out. Last night Kate got fed up, so I decided to see if I could put a CAPTCHA on the form.

Looks like there is a proposal to include CAPTCHA support. The de facto standard CAPTCHA tool for Plone seems to be collective.captcha, but I wanted to be able to use the ReCAPTCHA service.

Luckily there is collective.recaptcha which is in beta and, while not listing any releases on the Plone site, is available from the Cheese Shop. But, alas, there is no documentation. It is supposed to be a drop in replacement for collective.captcha so with a little sleuthing and I got it to work. Here's how.

Installation

I am using the unified Unix installer so everything is done via buildout. Following the instructions on the collective.captcha page we need to add collective.recaptcha to your eggs list and collective.recaptcha to your zcml list. Then run ./bin/buildout and restart your Plone instance.

Configuration

We need to enter our private and public ReCAPTCHA keys in the collective.recaptcha setup form which is located at http://urltoyourplonesite.com/recaptcha-settings

Registration Form Modifications

Using this site as a helpful reference I was able to figure out the additions I needed to make to the registration form and validator.

You need to go in through your Plone site setup into the Zope Management Interface. Navigate to portal_skins/plone_login. Click join_form and then the customize button. You should now be able to customize the form contents. Down near the bottom of the form there is a line:

<div class="formControls"></div>

Above that line you need to add:

<div class="field"
     tal:define="error errors/captcha|nothing;"
     tal:attributes="class python:test(error, 'field error', 'field')">
  <label for="captcha" i18n:translate="label_captcha">Captcha</label>

  <span class="fieldRequired" title="Required"
        i18n:attributes="title"
        i18n:domain="plone"
        i18n:translate="label_required">(Required)</span>

  <div class="formHelp" i18n:translate="help_captcha">
    Provide the text in the image. Just to avoid spambots
  </div>
  <p tal:replace="structure here/@@captcha/image_tag" />

</div>

Note that the difference from the code provided by Mikel Larreategi is that ReCAPTACH provides the input element itself, so you need to omit the div that includes the <input type="text"> tag.

Once you have that saved, you need to go back to portal_skins/plone_login and click join_form_validate and once again click customize. At the bottom of the validation code, just before return state add:

captcha = context.REQUEST.get('recaptcha_response_field')
view = context.restrictedTraverse('@@captcha')

if not view.verify(captcha):
    state.setError('captcha', _(u'Are you a bot? Try again...'))
    state.set(status='failure')

Note that the difference in the validation code from Mikel Larreategi's is that the ReCAPTCHA inserted input tag is called recaptcha_response_field and not captcha.

Hopefully that is helpful to someone other than me :)

Read and Post Comments

Installing Windows XP on a New Computer Without a Floppy Drive

December 14, 2007 at 10:32 PM | categories: Computers | View Comments

New computers with an Intel Chipset like this one support RAID and AHCI. Unfortunately you can't install Windows XP without the use of the F6 + driver disk method of loading drivers for installation, but only using a floppy. What happens when you don't have a floppy drive? You have to jump through hoops.

In my search for how to handle the situation, I quickly found this post which sent me on the right track, in search of nLite which allows you to add drivers directly to the Windows install CD.

Unfortunately, if you don't happen to have another computer with a floppy drive both Intel and Asus make things even more difficult. The Asus install CD has a make disk utility to create the driver disk, but requires a floppy drive in order to get at the drivers. Intel does not make things much better. They also provide a disk creation utility. It happens that they use WinImage, which at least means you can extract the drivers, but it is unfortunate that they don't just provide a zip disk.

Once I figured out that I HAD to use the drivers from the disk image, things went smoothly.

For those of you shocked that I actually installed Windows XP, the computer is in fact for me and XP is necessary for the work I will be doing with Kate starting in January when my paternity leave ends.

Read and Post Comments

HOWTO Cross Compile Python 2.5

October 06, 2006 at 04:04 PM | categories: Computers, Programming | View Comments

Recently I needed to compile Python for an embedded target. I used version 2.5 though it looks like that choice may have made it harder for me. I used 2.5 because I didn't want to have to figure out how to cross compile the cElementTree extension. Unfortunately I still ended up having to figure out how to get PyXML to build for my target. Fortunately I did get everything to work with a bit of fiddling. For posterity, here are some notes about what I did and what problems I had.

I started with Klaus Reimer's ARM cross-compiling howto and made made some updates required by changes between Python 2.2 and 2.5.

The changes I made are captured in an updated patch to apply against the 2.5 source tree. The changes made the configure.in are to disable rules that cause configure failures when cross compiling because they look for files on the target system or require a test program to be compiled and run. The other changes I added over the patch from Klaus Reimer are for a specific issue I had where md5 and sha hash algorithms were not built because setup.py which builds the modules is not cross compile aware and detected the OpenSSL libraries on my build machine rather than the target.

You can apply the patch with the following command:

~/tmp/Python2.5$ patch -p1 < ../Python2.5_xcompile.patch

I next generated a "host" python as in Reimer's instructions:

~/tmp/Python2.5$ ./configure
~/tmp/Python2.5$ make python Parser/pgen
~/tmp/Python2.5$ mv python hostpython
~/tmp/Python2.5$ mv Parser/pgen Parser/hostpgen
~/tmp/Python2.5$ make distclean

I exported necessary variables like CC, CXX, AR, RANLIB, LD, CFLAGS, CXXFLAGS, LDFLAGS for my target. The key is for these to point to the libraries you are going to use.

To build and "install" python for the my target I used the following commands:

~/tmp/Python2.5$ ./configure --host=ppc-linux --build=i686-pc-linux-gnu --prefix=/python
~/tmp/Python2.5$ make EXTRA_CFLAGS="$CFLAGS" HOSTPYTHON=./hostpython HOSTPGEN=./Parser/hostpgen BLDSHARED='ppc-linux-gcc -shared' CROSS_COMPILE=yes
~/tmp/Python2.5$ make install EXTRA_CFLAGS="$CFLAGS" HOSTPYTHON=./hostpython BLDSHARED='ppc-linux-gcc -shared' CROSS_COMPILE=yes prefix=/home/lambacck/tmp/dest/python

Note that I needed to use EXTRA_CFLAGS to add my target specific CFLAGS because for some reason the Python configure process does not honor the ones I provided while doing configure. The LDFLAGS variable, however, did work.

Also notice that I set the prefix in the configure step to be /python and the set another prefix in the make install step. The prefix in the configure step is where we are putting python on the file system on the target. The prefix in the install step is where we are putting all of the bits to be able to package them up to be ready to send to the target.

After all that I have a (mostly) functional python environment on my target, but I still needed to get PyXML built. I downloaded the latest distribution, modified setup.py so that expat was forced into little-endian mode and ran the following commands:

~/tmp/PyXML-0.8.4$ LDSHARED='ppc-linux-gcc -shared' CC="${CC}" OPT='-DNDEBUG -g -O3 -Wall -Wstrict-prototypes' ~/tmp/Python-2.5/hostpython -E setup.py build
~/tmp/PyXML-0.8.4$ LDSHARED='ppc-linux-gcc -shared' CC="${CC}" OPT='-DNDEBUG -g -O3 -Wall -Wstrict-prototypes' ~/tmp/cross_python/Python-2.5/hostpython -E setup.py install --prefix=$HOME/tmp/dest/python/ --install-scripts=/home/lambacck/tmp/dest/python/bin

It looks like Distutils and the Python build process in general could use some cross compile support. I think this is currently far harder than it has to be.

Update Oct 10, 2005: I would like to point out that on the system that I the executable used to compile hostpython was called gcc and I then updated paths so that a gcc targeted at ppc called gcc was first in my path. Therefore in both cases the compiler was gcc and not gcc for the hostpython and ppc-linux-gcc for the target python.

Update Dec 6, 2006: It looks like double dashes have been turned into single dashes by Wordpress. Specifically the host, build, prefix and install-scripts arguments to configure and setup.py should have double dashes.

Update Jun 30, 2010: Translation to Portuguese. Commands converted to use arm processor. Thanks Anuma.

Read and Post Comments

Why I Switched to Ubuntu

October 05, 2006 at 11:59 PM | categories: Computers, Programming | View Comments

In a comment to my last post, Installing Ubuntu on a drive with an existing RAID and LVM, Jay Parlar asks, "Why the switch to Ubuntu?" Well Jay, I'm glad you asked.

To answer that, I have to go back to the beginning, back to my first experiences with Linux. If you just want the executive summary without all the detail and witty prose, just skip to the end.

In my first year of University I had heard about this cool new OS called Linux (It was already 5 years old at that point, but what did I know). I of course didn't have a clue what it was or what I was getting myself in for but I decided the best way to understand it was to install it and take it for a spin. At the time I didn't quite get why people were giving it away for free, but after looking around at Red Hat, Slackware Debian and a couple others, I decided that the right one must be Debian.

Over the course of a week I proceeded to download a ridiculous amount of Debian 1.3 (bo) over dial up. Then I went through a not very friendly install process which eventually dumped me off a login prompt. After logging in I of course had no idea what to do because I had never touched any kind of Unix before. I played around with that with a bit of help from a Linux tutorial for DOS users, but generally that attempt was not very successful.

My adventures with Linux picked up when I took the McMaster University IEEE Student Branch Unix crash course in my second year of University. There, in one easy session, I got bumped from my computer and forced to be the typist for the demo computer attached to the projector. The second day the course was offered (two days later) I assisted in teaching the course by being one of the people who helped the students who were lost. The next semester I co-wrote the manual and taught the course.

For a long time I disto hopped. I was taking in all the cool new graphical features and installers and things that were coming out at the time and discovered that for the most part it was hard to get any new software on the system unless it got done during the initial installation. The exception to that was Debian. It seems that Debian figured out a lot of things between when I first installed them and when I eventually allocated them permanent space on my hard drive. In particular, they invented APT so that you could easily install and remove of software. Seemingly every program you could possibly want to use could be installed with the new APT system. They also seemed to have users that knew what they were doing.

Even though I mostly used Debian, I continued to use Windows and distro hope until after third year. When I went on internship, I gave up Windows went entirely Debian. I was satisfied with Debian for a fairly long time because I was using the unstable version. I decided that I had to do that because otherwise Debian was just too far behind on all the cool toys. Eventually I found myself being too often annoyed that unstable really meant unstable or that it took a really long time for the really cool new stuff to get packaged up and included.

When I started my Masters, I decided to switch to Gentoo. The source based distribution attracted me because it promised to be more agile about accepting packages and gave the opportunity to optimize what got compiled in or didn't. I also believed that machine specific optimizations were going to make a speed difference, something I never found a lot of evidence for.

I was already starting to be a bit dissatisfied with Gentoo before I moved to Ottawa, but the new living arrangements were the nail in the coffin. Suddenly my box was headless, and I had limited time to maintain my box because only one of Kate or myself could be using a computer at a time. I was also more interested in doing things with my computer than maintaining it.

I had had heard good things about Ubuntu, which is based on Debian. At work Pedro, a long time Debian user, said that he was using Ubuntu for desktop stuff and was really happy with it. Since I didn't have anything installed at work, I decided to take the Ubuntu plunge and was happy to find that it took little work to get it configured to my liking. It had the familiar Debian flavour with a new polished presentation.

It took me a couple months to get around to it but since I already wanted to redo my box at home I decided to go with Ubuntu. Thus far I have not regretted the choice and am even excited about some of the new features in the upcoming Edgy Eft which include Gnome 2.16, Firefox 2 and not having to muck with /etc/apt/sources.list to get Vim 7.

So far I have not been disappointed with my choice to move to Ubuntu. I think it provides all the features I liked about Debian and Gentoo, but with guaranteed regular updates and a usable desktop out of the box. And yes, I am using Gnome. It doesn't get in my way and at work, where I actually have a Linux desktop, I have enough processor and memory that I don't care that its a bit of a hog. I mostly only use Gnome Term anyways. At home I mostly ssh into my box. I use x11vnc to export a GUI when I need it, but I have been doing most of my graphical work on Windows (Firefox does work there ya know).

Ok, so lots of prose, mabey not so witty. Here is the promised executive summary:

  • I had a lot of experience with unmanageable distros
  • I found Debian and fell in love with its manageability
  • I wanted to be more cutting edge, especially in desktopy things so I switched to Gentoo
  • Gentoo was too much work and I wanted out
  • Ubuntu gives me the desktopy things I want while still giving me the Debian Manageability
Read and Post Comments