So I started to work on my own suite of tools for it, and recently I decided to release what I’ve done so far. It’s by no means complete, but it’s very useful for me and I thought it would be useful for others. And, with a bit of luck, someone will help me improving it.

So what is it? Arepa (it stands for “Apt REPository Assistant”, but obviously I called it like that after the yummy Venezuelan sandwiches) is a suite of tools that allow you to manage an APT repository. It contains two command-line tools and a web interface, and its main features are:

• Manages the whole process after a package arrives to the upload queue: from approving it to re-building from source to signing the final repository.
• It allows you to “approve” source packages uploaded to some “incoming” directory, via a web interface.
• It only accepts source packages, and those are re-compiled automatically in the configured autobuilders. It can even “cross-compile” for other distributions (treated like binNMUs).
• Far from reinventing (many) wheels, it integrates tools like reprepro, GPG, Rsync, debootstrap and sbuild so you don’t have to learn all about them.

The approval via some web interface was actually sort of the driving force for the project. One of my pet peeves was that there wasn’t an easy way to have an upload queue and easily approve/reject packages with the tools I knew. From what I had seen, the tools were either for “single person” repositories (no approval needed because the package author is the owner of the repository) or full-blown distribution-size tools like dak and such. My use-case, however, is the following:

• You have an installation of Arepa for an entire organisation (say, a whole company or a big department).
• People inside that organisation upload packages to the upload queue (possibly using dput; the point is, the end up in some directory in the machine hosting Arepa).
• Someone (or a small group of people) are the “masters” of the repository, and they’ll have access to the web interface. From time to time they check the web UI, and they’ll approve (or not) the incoming source packages.
• If they’re approved, the source will be added to the repository and it’ll be scheduled for compilation in the appropriate combination(s) of architectures and distributions.
• A cronjob compiles pending packages every hour; when the compilation is successful, they’re added to the repository.
• At this point, the repository hosted by the Arepa installation has the new packages, but you probably want to serve the repository from a different machine. If that’s the case, Arepa can sync the repository to your production machine with a simple command (“arepa sync”).

I imagine that a lot of people have the same need, so I uploaded all the code to CPAN (you can see it with the rest of the contributions by Opera Software). Sadly there’s a silly bug in the released code (I wanted to release ASAP to be able to focus on other things, and I ended up rushing the release), but it has both a workaround and a patch. So, please give it a try if you’re interested and tell me if you would like to contribute. I haven’t released the code in GitHub or similar yet, but I’ll probably do if there’s interest.

• Software automated testing 123, an entry-level talk about software automated testing. Why you should be doing it (if you’re not already), some advice for test writing, some basic concepts and some basic examples (in Perl, but I trust it shouldn’t be too hard to follow even if you don’t know the language).
• Taming the Snake: Python unit tests, another entry-level talk, but this time about Python unit testing specifically. How to write xUnit style tests with unittest, some advice and conventions and some notes on how to use the excellent nosetests tool.
• Introduction to Debian packaging, divided in four sessions: Introduction, Packaging a simple app, Backporting software and Packaging tools.

Just a quick note about them: the slides shouldn’t be too hard to understand without me talking, but of course you’ll lose some stuff that is not written down, some twists, clarifications of what I mean exactly by different things and whatnot. In particular, the “They. don’t. make. sense. Don’t. write. them” stuff refers to tests that don’t have a reliable/controlled environment to run into. I feel really strong about them, so I wanted to dedicate a few more seconds to smashing the idea that they’re ok, hence the extra slides :-)

Enjoy them, and please send me any comments you have about them!

The talks themselves went pretty good I think, although I’d have preferred more people attending. I guess it was normal that there were less people than I’m used to, since the Linköping office is much smaller. But anyway. The Debian course went quite well and some people got started packaging stuff almost right away. The other talks were an introduction to automated testing (advocacy and arguments for it, advice, basic examples and small rant about a different kind of QA), which went ok, and an entry-level talk about unit testing in Python (thanks Ask and Batiste for the information and reviewing the slides!), which went very well. I’ll try to get the slides for all the talks available somewhere.

About the city itself, it’s a charming little part of Sweden where:

• Restaurants have insanely different prices for food whether it’s for lunch or dinner. Typical prices for lunch are 80 SEK (around 8 EUR) and typical prices for dinner are around 250 SEK just the main course!
• Restaurants usually serve some Swedish dish for lunch… and I mean every restaurant, meaning all the Greek, Vietnamese, etc. Considering “real” Swedish restaurants are very expensive, you usually go to those foreign cuisine ones when you actually want to eat Swedish food.
• Restaurants typically have some salad (that you have to take yourself) while you wait for the food… and some coffee, tea and cookies (that obviously you have to take yourself) for the end.
• Related to this, restaurants are usually very self-service. I thought service in Norway sucked, but boy was I wrong, at least there is some service. And: there were typically long but pretty-fast-moving queues, and there was this one place where you didn’t even get the food on the table after ordering at the bar; instead, you were given some gadget with some wireless receiver, and when your food was ready it’d beep so you knew you had to go to some special place and fetch your food. Is it really cheaper maintaining some gadgets than hiring a waiter? I guess so.
• The restrictions on the amount of alcohol that can be bought outside the special Government booze stores are even harder than in Norway. You can only buy booze with up to 3.5% alcohol outside “Systembolaget”. Now that is sad. And I was complaining about Norway’s 5%.
• Partly because of that (I assume/hope) the Swedish “cider” you get in Sweden is even sweeter and worse and the Swedish cider you get in Norway.
• We went to this nice student pub… which was literally for students. They actually checked your student id, but each student could bring one non-student along. Once you were “identified” as a non-student-coming-with-a-student, you’d get a stamp on your hand so you wouldn’t have to bring along the student when you ordered again. Also, the place was so very slow it was almost funny. One of the good sides was that they had what I thought it was the only decent Swedish cider… but after checking just now, it seems it’s actually American. Bummer. And the name of it was funny too: “Hardcore Cider”.
• Right before leaving the office on Friday there was a small gathering in the canteen (the “Friday Beer”), where they had a Dreamcast with one of the most awesome games I’ve seen in a long while: The Typing of the Dead, a version of The House of the Dead 2 in which you kill the zombies by typing words that appear on the screen, instead of aiming and shooting with a gun:

The long(er) story:

• I bought a laptop with that card, and I wanted to make it work.
• Apparently the open source driver (b43) doesn’t recognise my card, although it seems it should?
• I tried to download the proprietary driver provided by the vendor (wl), but it didn’t compile at first. After applying some patch for the kernel 2.6.29 (I’m using kernel 2.6.30) it did compile, but it didn’t quite work. Meaning, it locked up my machine seconds after loading.
• After a couple of days of wondering and trying to make it work… I realised the driver is already compiled in Debian (in particular, broadcom-sta-modules-2.6.30-1-686). Just installing and loading it worked like a charm. Oh well.

The two modules I’ve released so far are:

• Parse::Debian::PackageDesc, a module for parsing both .dsc and .changes files from Debian. This is actually a support module for something bigger that I hope I’ll release soon-ish.
• Migraine, a still somewhat primitive database change manager inspired on the Ruby on Rails migration system.

As I feel that Migraine could be useful to a lot of people, but it’s easy to misunderstand what it really does (unless you already know Rails migrations of course), I’ll elaborate a bit. Imagine that you are developing some application that uses a database. You design the schema, write some SQL file with it, and everybody creates their own databases from that file. Now, as your application evolves, your schema will evolve too. What do you do now to update all databases (every developer installation, testing installations, and don’t forget the production database)? One painful way to do it could be documenting which SQL statements you have to execute in order to have the latest version of the schema, and expect people to apply copying-and-pasting from the documentation. However, it’s messy, confusing, and it needs someone to know both which databases to update and when.

Migraine offers a simpler, more reliable way to keep all your databases up to date. Basically, you write all your changes (“migrations”) in some files in a directory, following a simple version number naming convention (e.g. 001-add_users_table.sql, 002-change_passwd_field_type.sql), and migraine will allow you to keep your databases up to date. In the simplest, most common case, you call migraine with a configuration file specifying which database to upgrade, and it will figure out which migrations are pending to apply, if any, and apply them. The system currently only supports raw SQL, but it should be easy to extend with other types.

In principle, you shouldn’t need to write any Perl code to use migraine (it has a Perl module that you can use to integrate with your Perl programs if you like, but also a command-line tool), so you can use it even in non-Perl projects. Of course, some modern ORMs have their own database migration system, but very often you have to maintain legacy code that doesn’t use any fancy ORM, or you don’t like the migration system provided by the ORM, or you prefer keeping a single system for schema and data migrations… I think in those cases Migraine can help a lot reducing chaos and keeping things under control. Try it out and tell me what you think :-)

In a couple of days I’ll blog again about other contributions to free software I’ve made lately, but this time in the form of Opera widgets…

The point now is that, while working on it, I needed some module to parse command-line options and “subcommands” (like git commit, svn update, etc.). As it’s written in Perl, I had a look at CPAN to see if I could see anything. The most promising module was App::Rad, but it lacked a couple of things that were very important for me: my idea was “declaring” all the possible commands and options and have the module do all the work for me (generating the help pages and the default --help implementation, generate the program help subcommand and so on). App::Rad didn’t have that, and it didn’t seem to me like that was the direction they wanted to go to with the module. But I figured I’d drop the author an e-mail anyway and see if he liked the idea so I could start adding support for all that…

And boy was that a good idea. He replied a couple of days later, and said that they had liked the idea so much that they had implemented it already (that’s why he took a couple of days to reply), and he sent me an example of the new syntax they had introduced and asked if that was what I was thinking. And not only that, but they added me to the list of contributors just for giving the idea! That completely made my day, free software rocks!

Apart from other apps I knew already and didn’t particularly like, I found dfo (Desktop Flickr Organizer), a GNOME application. It was nice, and it was easy enough to upload pictures to Flickr with it, but it felt weird. What I would like to have is some application to manage my gallery, with some option to upload certain pictures to Flickr. However, this applications is more like a local Flickr mirror with synchronisation options. I don’t want all my pictures in Flickr, even marked as private. I just don’t care, and I don’t want to wait for all synchronisation between the app and Flickr. Moreover, I feel kind of tied to Flickr using that, and I’d rather work in a more “agnostic” environment. So it was cool using it to upload the pictures I had to upload, but I wasn’t really going to keep using it.

At the same time, one friend suggested using Picasa to upload some pictures, so I gave it a try. I had tried it briefly in the past, and I remember that some things were nice, but for some reason it was never my gallery manager of choice. So, trying it again, and even using the synchronisation options for the Picasa web albums, somehow I got the same feeling again: it’s nice, but there’s something undefined that makes me not use it. I have to admit that the interface is really fancy and easy to use, and it works decently well, but I don’t completely like the way the synchronisation works, not to mention that I don’t want to be stuck with only Picasa web albums. Also, I’m not happy with it being proprietary, not available in the Debian repositories, and with that special, anti-integrated interface. Some things work much better than in digiKam (I’m especially thinking fullscreen/slideshow, which sucks pretty badly in it), but I still prefer digiKam overall.

As I wasn’t too happy with the alternatives, I decided to have a look at the problem with digiKam. It turns out that digiKam just uses the so-called Kipi-plugins for picture exporting and other things, and that there was a new version of it that fixed a couple of issues… one of them being a problem with Flickr upload. The package is not available on Debian unstable because we’re currently in freeze (unfortunately, that means that Lenny will ship without a functional Flickr-uploading Kipi plugin). However, I saw that the new package was actually uploaded to experimental, so I decided to give it a try. Not only it works like a charm, but the new version 1.6 reworks the Flickr export plugin completely, and now it’s much nicer. So I’m happy now, back to digiKam with a working Flickr export o/. To install it yourself, make sure that you have this line in your /etc/apt/sources.list:

deb http://ftp.de.debian.org/debian/ experimental main non-free contrib

Then, update your available package list and install kipi-plugins from experimental, like this:

sudo aptitude update && sudo aptitude -t experimental install kipi-plugins

That should do it.

But. I. don’t. test. my. programs.

By hand, that is. And sometimes (usually) the coverage is not enough, and I end up making embarrassing mistakes. It usually happens outside of work, although at work I also have my share. The last one was with the Debian package dhelp, where trying to fix an issue before Lenny is released, I ended up making it even worse. The story goes like this:

There was some problem with the indexing of documents on installation/upgrade (namely, it would take ages for most people upgrading to Lenny, and they would think the upgrade process had hung). So, I go and change the indexing code so it ignores documents on installation/upgrade. Also, as suggested by someone, I created some small example utility to reindex documentation for certain packages. I test installation, upgrades, upgrade of the dhelp package itself, the utility, searching for keywords before and after all that… and everything worked.

Only that I made a typo. A typo that would make all indexing to be ignored (except for the example utility, because it was a bit lower level). And I didn’t realise, because it “only” broke some cronjob, a completely different part of the package. And it happens that the cronjob reindexed everything weekly, to make sure that you had reasonably up-to-date search indices. And it also happens that, given that the documentation reindexing was being ignored on package installation/upgrade, the weekly total reindex process was the only thing that could provide the user with indexed documentation. But I screwed up. Oh well.

Someone filed a bug yesterday, and I fixed more or less right away. But this time I spent a couple of hours thinking of test paths and ways to make it fail, and actually doing all that testing. Thanks to that, I found some potential bug in the example utility, that I fixed just in case. So hopefully everything is fine now, if I can convince the Release Masters to allow the new, less broken update to dhelp to be accepted for Lenny.

I think I need personal QA. Anyone up to the task?

The point is, I was really confused for quite a bit because, after reading the gpg manpage, I was trying things like:

gpg —no-default-keyring —keyring keys.gpg —import … # Wrong!

But that wouldn’t add anything to the keys.gpg, which I swear I had in the current directory. After a lot of wondering, I realised that gpg interprets paths for keyrings as relative to… ~/.gnupg, not the current directory. I guess it’s because of security reasons, but I find it really confusing.

The lesson learned, always use --keyring ./keys.gpg or, better, never use keys.gpg as filename for external keyrings, but something more explicit and “non-standard” like my-archive-keyring.gpg or whatever.

The first thing I tried was looking in APT’s cache for “video editing”. The most promising was kino. I had tried that some time ago a couple of times, and I never made it to work, but I figured I would try again. Unfortunately, same result: I just can’t figure out how to import my videos. Maybe I’m just hitting the wrong button or whatever, but it’s really frustrating.

Second thing was having a look in the internet. I found the (dead and being rewritten?) Cinelerra, as always, and I didn’t feel like installing the old one from source, only to lose my time and not get it to work, so I just ignored it. Maybe they had it in debian-multimedia and wouldn’t have been a tough install after all. Anyway.

Next thing, I found some program called openmovieeditor. This one apparently worked, but I couldn’t figure out how to crop the image (or almost any other thing for that matter).

Next, some neat program written in Python, called pitivi. When I tried to run it though, it just said Error: Icon 'misc' not present in theme on the console and died. I later figured out that I had to install gnome-icon-theme for it to work (yeah, Debian maintainer’s fault). It’s funny, because on the webpage it says that it has some “advanced view” that you can access via the “View” menu… but I couldn’t find it. My menu only had one entry: “Fullscreen”. Great.

Oh, wait, there’s a gimp-gap. I could just import my animation in Gimp, crop the frames, and convert again to video. Easier said than done. I needed some programs that I didn’t have, and I wasn’t sure if they were so easy/quick/clean to install (sure, I could have exported to GIF animation and probably convert to video, I just didn’t want to lose so much color quality in the GIF step). Forget for now. At least I had the images, so if I could just turn them into a movie…

So, I started wondering if, given that I had decided to just crop, and especially now that I had a lot of images that were the frames, maybe I could just use some command line tool or something. So I found this tiny little program, images2mpg. Long story short, after installing some dependencies from source (that gave compilation errors, but luckily I could compile only the binaries I really needed) that program was completely retarded and didn’t even do what I wanted (it wanted at least one second between images, but I didn’t want a slideshow, just a normal movie from the frames). It looks some simple and it’s so buggy. Gah.

So I started wondering if I could just crop with mplayer… Hmmm… after a couple of problems (like documented switches that were not there and other crap), I ended up with this command line:

mencoder -vf crop=320:200:0:40 MVI_2160.AVI
         -ovc lavc -nosound -o metro-crop.avi

That was reasonably quick and easy but it was so frustrating after all that lost time.

In any case, I ended up with the video I wanted, so I went to YouTube to upload it. When uploading, I realised that there was some option I had never seen: annotations.

YouTube annotations are really cool. They are like the notes on Flickr, but on a video :-D Actually I kind of wanted to make a note like that on this video, to show the automatic doors on the Metro station, so I was really happy to see that I could actually do it. And the interface is really easy to use and very clear. I really like it! You can see the result here:

EDIT: WTF? The annotations don’t appear on the embedded videos? You’ll have to go to the video page to see them, then…