HOWTO Cross Compile Python 2.5

October 06, 2006 at 04:04 PM | categories: Computers, Programming | View Comments

Recently I needed to compile Python for an embedded target. I used version 2.5 though it looks like that choice may have made it harder for me. I used 2.5 because I didn't want to have to figure out how to cross compile the cElementTree extension. Unfortunately I still ended up having to figure out how to get PyXML to build for my target. Fortunately I did get everything to work with a bit of fiddling. For posterity, here are some notes about what I did and what problems I had.

I started with Klaus Reimer's ARM cross-compiling howto and made made some updates required by changes between Python 2.2 and 2.5.

The changes I made are captured in an updated patch to apply against the 2.5 source tree. The changes made the configure.in are to disable rules that cause configure failures when cross compiling because they look for files on the target system or require a test program to be compiled and run. The other changes I added over the patch from Klaus Reimer are for a specific issue I had where md5 and sha hash algorithms were not built because setup.py which builds the modules is not cross compile aware and detected the OpenSSL libraries on my build machine rather than the target.

You can apply the patch with the following command:

~/tmp/Python2.5$ patch -p1 < ../Python2.5_xcompile.patch

I next generated a "host" python as in Reimer's instructions:

~/tmp/Python2.5$ ./configure
~/tmp/Python2.5$ make python Parser/pgen
~/tmp/Python2.5$ mv python hostpython
~/tmp/Python2.5$ mv Parser/pgen Parser/hostpgen
~/tmp/Python2.5$ make distclean

I exported necessary variables like CC, CXX, AR, RANLIB, LD, CFLAGS, CXXFLAGS, LDFLAGS for my target. The key is for these to point to the libraries you are going to use.

To build and "install" python for the my target I used the following commands:

~/tmp/Python2.5$ ./configure --host=ppc-linux --build=i686-pc-linux-gnu --prefix=/python
~/tmp/Python2.5$ make EXTRA_CFLAGS="$CFLAGS" HOSTPYTHON=./hostpython HOSTPGEN=./Parser/hostpgen BLDSHARED='ppc-linux-gcc -shared' CROSS_COMPILE=yes
~/tmp/Python2.5$ make install EXTRA_CFLAGS="$CFLAGS" HOSTPYTHON=./hostpython BLDSHARED='ppc-linux-gcc -shared' CROSS_COMPILE=yes prefix=/home/lambacck/tmp/dest/python

Note that I needed to use EXTRA_CFLAGS to add my target specific CFLAGS because for some reason the Python configure process does not honor the ones I provided while doing configure. The LDFLAGS variable, however, did work.

Also notice that I set the prefix in the configure step to be /python and the set another prefix in the make install step. The prefix in the configure step is where we are putting python on the file system on the target. The prefix in the install step is where we are putting all of the bits to be able to package them up to be ready to send to the target.

After all that I have a (mostly) functional python environment on my target, but I still needed to get PyXML built. I downloaded the latest distribution, modified setup.py so that expat was forced into little-endian mode and ran the following commands:

~/tmp/PyXML-0.8.4$ LDSHARED='ppc-linux-gcc -shared' CC="${CC}" OPT='-DNDEBUG -g -O3 -Wall -Wstrict-prototypes' ~/tmp/Python-2.5/hostpython -E setup.py build
~/tmp/PyXML-0.8.4$ LDSHARED='ppc-linux-gcc -shared' CC="${CC}" OPT='-DNDEBUG -g -O3 -Wall -Wstrict-prototypes' ~/tmp/cross_python/Python-2.5/hostpython -E setup.py install --prefix=$HOME/tmp/dest/python/ --install-scripts=/home/lambacck/tmp/dest/python/bin

It looks like Distutils and the Python build process in general could use some cross compile support. I think this is currently far harder than it has to be.

Update Oct 10, 2005: I would like to point out that on the system that I the executable used to compile hostpython was called gcc and I then updated paths so that a gcc targeted at ppc called gcc was first in my path. Therefore in both cases the compiler was gcc and not gcc for the hostpython and ppc-linux-gcc for the target python.

Update Dec 6, 2006: It looks like double dashes have been turned into single dashes by Wordpress. Specifically the host, build, prefix and install-scripts arguments to configure and setup.py should have double dashes.

Update Jun 30, 2010: Translation to Portuguese. Commands converted to use arm processor. Thanks Anuma.

Read and Post Comments

Why I Switched to Ubuntu

October 05, 2006 at 11:59 PM | categories: Computers, Programming | View Comments

In a comment to my last post, Installing Ubuntu on a drive with an existing RAID and LVM, Jay Parlar asks, "Why the switch to Ubuntu?" Well Jay, I'm glad you asked.

To answer that, I have to go back to the beginning, back to my first experiences with Linux. If you just want the executive summary without all the detail and witty prose, just skip to the end.

In my first year of University I had heard about this cool new OS called Linux (It was already 5 years old at that point, but what did I know). I of course didn't have a clue what it was or what I was getting myself in for but I decided the best way to understand it was to install it and take it for a spin. At the time I didn't quite get why people were giving it away for free, but after looking around at Red Hat, Slackware Debian and a couple others, I decided that the right one must be Debian.

Over the course of a week I proceeded to download a ridiculous amount of Debian 1.3 (bo) over dial up. Then I went through a not very friendly install process which eventually dumped me off a login prompt. After logging in I of course had no idea what to do because I had never touched any kind of Unix before. I played around with that with a bit of help from a Linux tutorial for DOS users, but generally that attempt was not very successful.

My adventures with Linux picked up when I took the McMaster University IEEE Student Branch Unix crash course in my second year of University. There, in one easy session, I got bumped from my computer and forced to be the typist for the demo computer attached to the projector. The second day the course was offered (two days later) I assisted in teaching the course by being one of the people who helped the students who were lost. The next semester I co-wrote the manual and taught the course.

For a long time I disto hopped. I was taking in all the cool new graphical features and installers and things that were coming out at the time and discovered that for the most part it was hard to get any new software on the system unless it got done during the initial installation. The exception to that was Debian. It seems that Debian figured out a lot of things between when I first installed them and when I eventually allocated them permanent space on my hard drive. In particular, they invented APT so that you could easily install and remove of software. Seemingly every program you could possibly want to use could be installed with the new APT system. They also seemed to have users that knew what they were doing.

Even though I mostly used Debian, I continued to use Windows and distro hope until after third year. When I went on internship, I gave up Windows went entirely Debian. I was satisfied with Debian for a fairly long time because I was using the unstable version. I decided that I had to do that because otherwise Debian was just too far behind on all the cool toys. Eventually I found myself being too often annoyed that unstable really meant unstable or that it took a really long time for the really cool new stuff to get packaged up and included.

When I started my Masters, I decided to switch to Gentoo. The source based distribution attracted me because it promised to be more agile about accepting packages and gave the opportunity to optimize what got compiled in or didn't. I also believed that machine specific optimizations were going to make a speed difference, something I never found a lot of evidence for.

I was already starting to be a bit dissatisfied with Gentoo before I moved to Ottawa, but the new living arrangements were the nail in the coffin. Suddenly my box was headless, and I had limited time to maintain my box because only one of Kate or myself could be using a computer at a time. I was also more interested in doing things with my computer than maintaining it.

I had had heard good things about Ubuntu, which is based on Debian. At work Pedro, a long time Debian user, said that he was using Ubuntu for desktop stuff and was really happy with it. Since I didn't have anything installed at work, I decided to take the Ubuntu plunge and was happy to find that it took little work to get it configured to my liking. It had the familiar Debian flavour with a new polished presentation.

It took me a couple months to get around to it but since I already wanted to redo my box at home I decided to go with Ubuntu. Thus far I have not regretted the choice and am even excited about some of the new features in the upcoming Edgy Eft which include Gnome 2.16, Firefox 2 and not having to muck with /etc/apt/sources.list to get Vim 7.

So far I have not been disappointed with my choice to move to Ubuntu. I think it provides all the features I liked about Debian and Gentoo, but with guaranteed regular updates and a usable desktop out of the box. And yes, I am using Gnome. It doesn't get in my way and at work, where I actually have a Linux desktop, I have enough processor and memory that I don't care that its a bit of a hog. I mostly only use Gnome Term anyways. At home I mostly ssh into my box. I use x11vnc to export a GUI when I need it, but I have been doing most of my graphical work on Windows (Firefox does work there ya know).

Ok, so lots of prose, mabey not so witty. Here is the promised executive summary:

  • I had a lot of experience with unmanageable distros
  • I found Debian and fell in love with its manageability
  • I wanted to be more cutting edge, especially in desktopy things so I switched to Gentoo
  • Gentoo was too much work and I wanted out
  • Ubuntu gives me the desktopy things I want while still giving me the Debian Manageability
Read and Post Comments

Installing Ubuntu on a drive with an existing RAID and LVM

September 20, 2006 at 09:51 AM | categories: Computers, Programming | View Comments

I recently switched this server from running Gentoo Linux to Ubuntu Linux. In the process I only wanted to replace the OS and not loose my home directory and other data. 160G hard drives which are partitioned identically so that I can used RAID in a mirrored setup (RAID1). There are 4 partitions on the drive: a 10G primary partition which I had intended for windows but never used, 100M primary partition used as /boot, a 2G logical partition for the root filesystem (/), a 500M swap partition (Not raided gives me 1G of swap), and the rest ~151G used as LVM for the rest of the system.

Running # cat /proc/mdstat gives the following output:

Personalities : [raid1]
md2 : active raid1 hde6[0] hdg6[1]
      147717568 blocks [2/2] [UU]

md1 : active raid1 hde5[0] hdg5[1]
      1999936 blocks [2/2] [UU]

md0 : active raid1 hde2[0] hdg2[1]
      96320 blocks [2/2] [UU]

unused devices: <none>

Running # lvscan gives the following ouput:

ACTIVE            '/dev/vg0/tmp' [2.00 GB] inherit
ACTIVE            '/dev/vg0/home' [20.00 GB] inherit
ACTIVE            '/dev/vg0/usr' [15.00 GB] inherit
ACTIVE            '/dev/vg0/local' [5.00 GB] inherit
ACTIVE            '/dev/vg0/opt' [5.00 GB] inherit
ACTIVE            '/dev/vg0/var' [5.00 GB] inherit
ACTIVE            '/dev/vg0/data' [88.87 GB] inherit

I wanted to format md0, md1, /dev/vg0/tmp, /dev/vg0/usr, /dev/vg0/local, /dev/vg0/opt, /dev/vg0/var, and keep the data on /dev/vg0/home, /dev/vg0/data. I did a backup of /etc for reference and /var/www because I would need to restore some stuff. I also backed up my mysql databases since they are also stored in /var. I did a backup of most of the important data on /dev/vg0/data and /dev/vg0/home. Some data I did not backup (media) because there was just too much of it.

Once all of my planning and backing up was done I booted with the Ubuntu Dapper Drake alternate i386 disk which is required for RAID and/or LVM configurations. My first supprise was that there was no GUI install. We were back to the Debian console install manager. Everything went smoothly until I got to the partitioning. It did not detect my existing RAID and LVM configuration. I was able to get my existing configuration up and running via the command line on another Virtual Terminal(VT), but by that point I had the installer wedged in a state where it just would not continue.

I had to reboot and start all over again. Here are the steps I took to get it working:

  1. boot the installer
  2. switch VTs to get a console
  3. run modprobe raid1
  4. run modprobe dm_mod
  5. run mdrun
  6. run vgchange -a y
  7. cat /proc/mdstat and lvscan should give output like above
  8. change back to the VT with the installer running in it
  9. continue with installer steps
  10. when you get to the partitioning step, it should recognise your raid and lvm devices

I ended up getting everything installed with no loss of data.

Read and Post Comments

Read and Post Comments

Stanley Park & Aquarium

August 24, 2006 at 01:28 PM | categories: Photography, General | View Comments

Yesterday we saw Stanley park and the Aquarium. We had previously seen many of the species from the tropical section, like the jelly fish to the right, while diving. They also have a section of species from the BC coast. It really showed that no matter where in the ocean you go, there is some really interesting form of life. It's not the fact that lakes are cold that makes lake life uninteresting, it's just the lakes themselves.

The best part were the big water mamals like the dolphins, the sea otters, baluga whales and sea lion. The sea lion was huge! You can get a bit of an idea of the size from the picture, but it still pales in comparison to standing beside it popping out the water like that.

There was also a Amazon rain forest section. Apparently there are about 200 species of fish that feed on fruit and seeds when the Amazon rain forest floods. In North America there are none. As part of the Amazon section there was a little aviary that had ibis and butterflies(left).

After the aquarium we took the free Stanley Park bus which circles the 5.5 mile perimeter of the park. A stop at Prospect Point finally gave me an opportunity to take a half way decent picture of the Lionsgate Bridge. A very tasty dinner at Joe Fortes finished the day for us. Exhausted we returned to the hotel room for our final night before our 11pm flight tonight.

We are having a bit of a lazy morning since we have to give up the hotel room soon, and will not have the option of returning for a rest. This afternoon the plan is to head to Gastown, Chinatown and Canada Place. I don't know if we are going to make it to Granville Island. At the end of what is sure to be a full day, we will have to make our way to the airport for our 5 hour flight. I have to go pretty much directly from the flight to work, at which I am sure to be well rested and alert :)

Read and Post Comments