The actual apt-get dist-upgrade went fine, with only minor problems which were easy enough to fix. The problem came when I rebooted. The boot started fine, but then got
stuck at the purple boot page, which showed “Ubuntu 10.4″ and 5 dots which cycled
between white and red. It never got past that point.

The usual thing to do at this point is to reboot in single user mode and start debugging startup scripts. Unfortunately I found that single user mode with Ubuntu Lucid was not useful as it doesn’t start a shell until after a huge pile of other things are started. In my case a ’single’ boot got stuck at the same point. Getting rid of the quiet and splash options, and adding nomodeset also didn’t help.

I found that if I booted an older kernel (2.6.31-19) then the system came up OK. That pointed to a likely driver issue. I could have just settled for that older kernel, but part of the reason for going to Lucid was to get a newer ALSA with better support for HDMI audio, so I didn’t really want to stick to an older kernel. I also wanted to know why the problem was happening.

I was also able to get a shell using the latest kernel by using the init=/bin/bash trick, but that doesn’t help to actually debug the problem. To debug startup problems you need to be able to watch the startup process in action, to see what is waiting. This is much harder these days with the new upstart init system now used in Ubuntu, as startup is much more parallel than it used to be. Adding some echo lines to init scripts used to be a useful technique, but it is much harder to get anything sensible out of that when using upstart.

To try to debug the problem I initially had a look for any startup debugging options. I found some promising options in /etc/default/rcS, and tried setting VERBOSE=yes and SULOGIN=yes. I found that the VERBOSE=yes option was somewhat useful, as it gave me some information on what jobs were started/waiting, but it didn’t really allow me to pin down the problem. The parallelism in upstart again made interpreting the output hard. When it says that a job is waiting it doesn’t say what it is waiting on, so you have no idea what the underlying problem really is.

Despite the promising name, and the nice description in the rcS(5) manpage, the SULOGIN=yes option didn’t seem to do anything at all. A grep for SULOGIN in the startup scripts didn’t find any hits, so I suspect it isn’t actually implemented.

As usual, the real key to solving the problem was a hack. I added the following to /etc/default/rcS:

(
/bin/sleep 10
/sbin/ifconfig eth0 192.168.2.10 up
/usr/sbin/sshd
) > /dev/null 2>&1 &

The idea behind this hack was to allow me to login with ssh from my laptop during the startup process and watch what was going on. This worked really well and meant that I was finally able to debug the startup process with the most recent Lucid kernel.

I rebooted again, logged into the system with ssh from my laptop, and started poking around with ps and initctl to see what was going on. I had assumed that “initctl list” would give me the information I needed. It does show what jobs are waiting, but as with the VERBOSE=yes messages it doesn’t tell you what it is waiting on.

Poking around some more I saw 3 things that were suspicious:

1) cryptdisks-enable was shown as “waiting”. I don’t have any encrypted disks on this system, so why should it be waiting?

2) dmesg showed a segfault in plymouth, which is the process that asks for user input during startup (it also does splash screens). This could be linked to why cryptdisks was waiting, as its possible that cryptdisks wanted a passphrase (for what disk though? I don’t have any encrypted disks)

3) dmesg also showed a lot of warnings from the dvb-usb-cxusb driver

As I was running low on time I decided to try the triple whammy of removing the cryptsetup package, removing the dvb-usb and dvb-usb-cxusb drivers (by moving them out of /lib/modules and running depmod) and removing the plymouth-theme-ubuntu-text package to try to simplify plymouth. This did the trick and my system now boots fine.

I still have the puzzle as to what is really causing the problem (and thus which of the changes matter), but I can leave that for another day. I thought it would be worthwhile sharing the ssh debug hack in case other people are also trying to debug upstart startup problems.

The system will consist of 132 Sunpower 225W panels, and 6 Xantrex GT5 inverters. It is being installed on a colorbond roof, using a landscape layout of the panels:

I got quotes from a lot of installers, and found a big variance in the level of knowledge that the various installers showed when they quoted for the system. I ended up getting quotes from 2 good Canberra based installers (Armada and Enviro-Friendly) and one national installer (Clear Solar). The Clear Solar quote was the one I went with because of the really good mounting system they were able to show me. The rails are mounted vertically up the roof face, and the mounting brackets are shaped to the contours of the colorbond roofing, giving a strong mounting system that allows for airflow on the back of the panels which should keep them a bit cooler (the efficiency of PV panels drops quite quickly if they get too hot). It should also stop leaves and other rubbish from building up behind the rails, which would tend to happen if the rails were mounted horizontally.

I’m getting a data logger and Paul Wilson from Clear has sent me the manual for the protocol to talk to the inverters, so I should be able to write some python code pretty quickly to analyze the performance of the system and watch for any degradation.

It will be fun to be producing more electricity than I use!

After returning from vacation I decided to update my laptop from Ubuntu Karmic to Lucid, and then found that kaddressbook wouldn’t start. Digging into it I found a maze of errors related to mysql and innodb. I hadn’t even realized that kaddressbook was using mysql for my addressbook till now. It’s a bit of an overkill for my little home addressbook

After spending some time tracing mysql and trying to work out why I was getting “InnoDB: No valid checkpoint found”, I realised there was an easier way. Inside that mysql database was just a set of binary VCARD records. So the quick fix was:

mlgrep.py BEGIN:VCARD END:VCARD ~/.local/share/akonadi/db_data/ibdata1

putting the result in a file. I then imported the file into the evolution contacts app and all was well. Of course, this assumes records aren’t split within the db, but it seems to work well enough for this quick fix.

See http://samba.org/ftp/unpacked/junkcode/mlgrep.py for mlgrep.

Cheers, Tridge

After I came back from a long vacation last week (long service leave is nice!) I got a message from internode telling me my SIP service had been suspended due to suspicious activity. I had a look at my asterisk logs, and found that while I had been away my asterisk box had been busy making calls to various unusual places, especially North Korea and Somalia. I do fair number a lot of overseas calls, especially to the US, Germany and Sweden, but this certainly didn’t look legit!

I looked a bit more into my logs and found that I’d been compromised twice – once on June 1st and another time on June 15th. The first attacker only made a couple of test calls to Romania, Sierra Leone and Zimbabwe, plus one to North Korea. The second attacker took a lot more advantage of their ill-gotten SIP secret however, and made 151 calls, most of them to North Korea plus some to Somalia.

What the logs show is that the attacks both followed the same pattern. Both attackers first scanned extensions, trying all 1 digit extensions, then all 2 digit extensions then all 3 digit extensions etc. From that they found the list of five 4-digit extensions that were active on my asterisk box. Then they went back and brute-forced the SIP secrets on those extensions. Unfortunately I had chosen digit-only secrets when I first installed asterisk (thinking it may help with setup on keypad only devices) and I’d never changed them. Not a good idea! They needed a total of about 39k SIP auth requests to find the secrets.

Of course, this service was never intended to be exposed to the internet. The way it had become exposed was that I had been trying to track down a call dropout problem with engin, and thinking that they may be using a variant of the asterisk “try an OPTIONS request every few minutes to see if the other end is still there” approach, I temporarily had allowed UDP port 5060 through my firewall. Of course I then forgot to disable it again once I determined that wasn’t the problem. Darn!

Luckily whatever the people had to say to the person they were calling in North Korea wasn’t long – the calls were all about 5 minutes long (suspiciously close to 300 seconds actually – maybe they had dropouts too?). It still added up to a fair bit though – over $250 worth of calls. Ouch!

I contacted internode and they re-enabled my SIP account once I changed my internode SIP password (it wasn’t actually my internode SIP password that was compromised, but this was still a very reasonable precaution).  At the suggestion of Travis, a very helpful internode helpdesk person, I put in a feedback request describing what happened, and they very kindly agreed to credit me with the charges for the fraudulent calls. That was a very nice surprise! Internode do seem to be well ahead of the crowd in customer service. Of course, this was a one-off gesture of goodwill, so it would not be a good idea to rely on this sort of generosity.

The calls made via engin were a bit strange. They showed up in my asterisk logs, but didn’t show up in the engin web call log or billing interface. I rang and asked engin about that, and a helpful person called Erick assured me that the bill was correct, and I wasn’t going to receive a big bill later. Perhaps someone at engin had already stripped their logs of calls to North Korea after someone else was hit by this?

Meanwhile, I’ve now got long random passwords in asterisk, and I’ve fixed my firewall. I’ve also setup a little UDP logger on port 5060 to grab any future attempts anyone makes to brute force my SIP service (just for amusement value, I’m curious to see how often I get scanned).

I wonder who was at the receiving end of all those calls to North Korea?

Cheers, Tridge

If you run a non-Australian site but you want to also show your support, then you might like to grab the hacked up popup code I used on samba.org.

For Australian sites, please use the official code. Hopefully something appropriate for non-au sites will be added there soon.

Cheers, Tridge

I think the key to LCA is the amazing volunteers who run it. The combination of moving the conference between cities each year along with enthusiastic volunteers from the FOSS community makes for an incredible conference year after year.

Thank you to everyone who made LCA 2010 in Wellington such a success. If you missed it, then make sure you don’t miss next years LCA in Brisbane. The highlight of every year for me is LCA, and I am so grateful that we have such a great community of FOSS hackers in Australia and New Zealand to make it possible.

Cheers, Tridge

You can see the video here:

http://samba.org/tridge/DRS-demo/s4-DRS-demo.avi

(if you have a fast link and can play Ogg videos, there is a high quality version too)

The video shows a few things:

• provisioning a Samba4 domain controller
• joining a Windows 2008 R2 machine as an additional domain controller for the Samba domain
• joining an additional Samba domain controller to the same domain
• adding a user and seeing changes to the user propogate between the domain controllers

There is a lot more to do in Samba4, but we are making very rapid progress now. Thanks to everyone who has contributed to this effort!

Cheers, Tridge

PS: If you’ve having trouble playing the videos on windows, you may want to install the x264 codec. See http://sourceforge.net/projects/x264vfw/files/ for a downloadable codec pack for windows media player.

The result is a new use for pyRoast, and some great “techno truffles” !

See http://coffeesnobs.com.au/YaBB.pl?num=1263085051 for all the details and a short movie.

Cheers, Tridge

One that he missed in his talk is some cute stuff you can do with the p command (the short form for ‘print’). For example, say you have some process that is sending its output to /dev/null, but you want to see that output. That can happen with long running daemons that were started without debugging enabled for example. What you need to do is tell that program to change file descriptor 1 to point to a file, instead of /dev/null.

Let’s look at an example:

 lsof -n |grep null|grep 1u
 gconfd-2  17243     tridge    1u      CHR                1,3              5961 /dev/null

We see from lsof that gconfd-2 has pid of 17243 and is redirecting stdout (fd 1) to /dev/null. Now let’s attach with gdb, close fd 1, and re-open it on a file in /tmp

 tridge@blu:~$ gdb --pid 17243
 GNU gdb 6.8-debian
 Copyright (C) 2008 Free Software Foundation, Inc.
 License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
 This is free software: you are free to change and redistribute it.
 There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
 and "show warranty" for details.
 This GDB was configured as "x86_64-linux-gnu".
 Attaching to process 17243

ok, we’re attached, now the dirty work:

 (gdb) p close(1)
 $1 = 0
 (gdb) p fopen("/tmp/gconfd.log", "w")
 $2 = 27041792
 (gdb) p fileno($2)
 $6 = 1
 (gdb) quit

Now if we check with lsof, we see the fd is now open on a log file

 gconfd-2  17243     tridge    1u      REG                8,1        0 12277473 /tmp/gconfd.log

You may ask why I used fopen() and not open(). First off, “w” is easier than remembering the right flag bits to open. Secondly, open may in fact be open64 or some other symbol, whereas I’ve found that fopen() always seems to work.

Notice also that I checked that fopen() gave us the right file descriptor back by calling fileno() on the output of fopen(). It will always give you the lowest available fd, which is usually right. If it isn’t the right one then use p with dup2() to move it to the right one.

Another use

This type of hack can also be used to solve the problem of unmounting a filesystem that has a process running with a file open on the filesystem read-write. The kernel doesn’t allow that, and the usual approach is to kill off all those processes. But what if you don’t want one of those processes to die? You can use the above trick to move the fd to point at a different filesystem. By using lsof to find all the ones you need to move, you can eventually mount the filesystem read-only, or even unmount it. This allows you to swap filesystems with running processes using them. You can minimise the effect on the processes by using gdb -x to script it (so it isn’t paused waiting for nervous fingers to type the right commands).

Most things can be taught of course, but free software development is rarely taught. Instead hundreds of thousands of people have learned it by slowly absorbing the culture and the development practices from other people who already develop free software. I wasn’t absolutely certain that it could be successfully taught using more traditional teaching techniques in a university environment.

Last week gave me the answer very clearly – FOSS development can be taught, and indeed it can be taught very quickly. Most of the 21 students who took the course started off with almost no knowledge of what free software is. Only 2 of the students had any previous involvement with free software development, and many of the others had never used a free operating system before. The first day and a half had me a bit worried, as it was clear that many of the students were struggling, but by the end of the 2nd day the students were starting to understand. By the start of the third day I was starting to see the glow of understanding in many of their faces as they did the lab work and contributed in the discussions during the lectures.

By the end of the week the students were really enjoying it, and many of them had successfully contributed to a free software project of their choice. It was amazing to see them interacting with other developers from all over the world, and it was fantastic that they found the experience so rewarding.

Scaling it up

So, FOSS development can be taught. Now how do we scale it up? Bob and I have released all our lecture and lab notes, plus videos of all of the lectures on the course website under a creative commons license. We will also soon be releasing the scripts we used to setup the lab machines. So now I am looking forward to some other universities from other parts of the world adopting the course material and offering their own courses in FOSS development.

If we can make this a standard part of CS curriculums around the world, then the next generation of free software developers will be a bit better prepared.

Thanks!

Many thanks to everyone who made it possible to run this course. Special thanks to Bob Edwards and Henry Gardner for suggesting the course, and to Deanne Drummond for helping to get it all organised.

I’d also like to thank CASE, who were the official providers of the course. I love doing volunteer work for case, and this was the most enjoyable project I’ve done for them so far.

Finally, thanks to the students who took part. Taking a course like this when it hasn’t been offered before is always a bit of a stab in the dark. I hope you all got as much out of it as I did.