Arepa - Apt REPository Assistant

For some time now I had been frustrated by the tools to manage APT repositories. The only ones I knew of either covered too little (only adding/removing packages from a repository and such, like reprepro) or were way too complex (like the official tools used by Debian itself). Maybe/probably I’m a moron and I just didn’t know of some tool that would solve all my problems, but now it’s kind of late ;-) And before you say it, no, Launchpad is not what I was looking for as far as I understand it.

So I started to work on my own suite of tools for it, and recently I decided to release what I’ve done so far. It’s by no means complete, but it’s very useful for me and I thought it would be useful for others. And, with a bit of luck, someone will help me improving it.

So what is it? Arepa (it stands for “Apt REPository Assistant”, but obviously I called it like that after the yummy Venezuelan sandwiches) is a suite of tools that allow you to manage an APT repository. It contains two command-line tools and a web interface, and its main features are:

• Manages the whole process after a package arrives to the upload queue: from approving it to re-building from source to signing the final repository.

• It allows you to “approve” source packages uploaded to some “incoming” directory, via a web interface.

• It only accepts source packages, and those are re-compiled automatically in the configured autobuilders. It can even “cross-compile” for other distributions (treated like binNMUs).

• Far from reinventing (many) wheels, it integrates tools like reprepro, GPG, Rsync, debootstrap and sbuild so you don’t have to learn all about them.

The approval via some web interface was actually sort of the driving force for the project. One of my pet peeves was that there wasn’t an easy way to have an upload queue and easily approve/reject packages with the tools I knew. From what I had seen, the tools were either for “single person” repositories (no approval needed because the package author is the owner of the repository) or full-blown distribution-size tools like dak and such. My use-case, however, is the following:

• You have an installation of Arepa for an entire organisation (say, a whole company or a big department).

• People inside that organisation upload packages to the upload queue (possibly using dput; the point is, the end up in some directory in the machine hosting Arepa).

• Someone (or a small group of people) are the “masters” of the repository, and they’ll have access to the web interface. From time to time they check the web UI, and they’ll approve (or not) the incoming source packages.

• If they’re approved, the source will be added to the repository and it’ll be scheduled for compilation in the appropriate combination(s) of architectures and distributions.

• A cronjob compiles pending packages every hour; when the compilation is successful, they’re added to the repository.

• At this point, the repository hosted by the Arepa installation has the new packages, but you probably want to serve the repository from a different machine. If that’s the case, Arepa can sync the repository to your production machine with a simple command (“arepa sync”).

I imagine that a lot of people have the same need, so I uploaded all the code to CPAN (you can see it with the rest of the contributions by Opera Software). Sadly there’s a silly bug in the released code (I wanted to release ASAP to be able to focus on other things, and I ended up rushing the release), but it has both a workaround and a patch. So, please give it a try if you’re interested and tell me if you would like to contribute. I haven’t released the code in GitHub or similar yet, but I’ll probably do if there’s interest.

• arepa
• auto builders
• cpan
• packages
• perl
• projects

BCM4312 on Linux: easier than expected

Just a quick post to say that I was being stupid and it took me a couple of days of fighting, lockups and reading to realise that the driver for the wireless card in my new laptop is actually already packaged and it works like a charm.

The long(er) story:

• I bought a laptop with that card, and I wanted to make it work.

• Apparently the open source driver (b43) doesn’t recognise my card, although it seems it should?

• I tried to download the proprietary driver provided by the vendor (wl), but it didn’t compile at first. After applying some patch for the kernel 2.6.29 (I’m using kernel 2.6.30) it did compile, but it didn’t quite work. Meaning, it locked up my machine seconds after loading.

• After a couple of days of wondering and trying to make it work… I realised the driver is already compiled in Debian (in particular, broadcom-sta-modules-2.6.30-1-686). Just installing and loading it worked like a charm. Oh well.

• bcm
• bcm4312
• broadcom
• card
• Debian
• packages
• problems
• wireless
• wl

Frustrated by Python module management

I don’t really do any Python development myself, but at work I do support some automated testing infrastructure (in this particular case, I’m talking about a CruiseControl.rb installation), and some of the projects that use that infrastructure use Python. The setup is so that the tests are actually executed in the CC.rb server, so I have to have Python installed there, and it has to have some basic dependencies to be able to run the tests.

A couple of times something strange happened: suddenly, those tests would start failing with no apparent reason, and looking at the logs, it looked like some dependencies were not installed (error messages such as ImportError: No module named sqlalchemy). Of course that didn’t make any sense, because SQLAlchemy is needed for the tests and they were working like a charm for weeks. I was totally and completely confused by the error message, and I tried to install SQLAlchemy again. That solved the problem, luckily, so I decided to forget about it because it wasn’t my thing anyway.

But the problems appeared again. And again. And I got another error message that was really confusing, because it looked like Python was using some old version of some module (a version that wasn’t there anymore, because the code had been updated from SVN). So I just got tired of not knowing what was going on, and decided to investigate enough to find out the root of the problem. And I found something surprising.

What I found is that the famous python setup.py develop (that everyone told me to use) actually adds the “current directory” to the list of paths where Python searches for modules, so you can develop your module and use it from anywhere. I had heard some comment on that, but I didn’t quite get what it meant, and I don’t think the person that said it realised either.

The fun thing with setup.py develop is that when you have several branches of the same project in the same machine, and you use that to make the modules available… well, I guess that knowing which versions of which modules Python will use becomes an interesting question to say the least. I’m not saying that the way it works is necessarily wrong, but I do think it is dangerous, and people shouldn’t think of it as the “normal” way of developing modules in Python. It should be used with care.

After having realised that and thought about it a bit, I still don’t understand why those modules simply “dissappeared”, but it seems that there was some corruption of /usr/lib/python2.5/site-packages/easy_install.pth or similar (that file seems to be modified when you install packages with easy_install, and it had references to the directories I ran setup.py develop from, so that’s my main suspect for now). At least I know now that I could backup a working easy_install.pth file, and restore when we have problems again, but I’m far from happy with that “solution” ;-)

Also, I’m wondering what the hell should I do in the future to prevent more problems, because using setup.py develop sounds like a terrible idea to me. I tried to set PYTHONPATH instead, but apparently I failed. Any suggestions?

EDIT: I’m finally using PYTHONPATH. I have no idea what I tried last time, but using it was easy, quick and clean. I still have no idea why the hell Python sometimes forgets where some modules are, though.

• easy_install
• modules
• packages
• python
• setuptools

Dhelp strikes back

In the last days I have gone back to working on dhelp, a Debian package for documentation indexing and search. Months ago I had started rewriting dhelp_parse, the only program in the suite written in C, in Ruby.

The rewrite was almost done, but the program wasn’t tested much (some modules had unit tests, but the program itself didn’t), so I found a couple of big bugs easily :-D Now it looks better, so hopefully I’ll be done soon, and I’ll upload the new package to Debian so people can start testing it.

• dhelp
• packages