Posts Tagged “cpan”
Mar 22, 2010
For some time now I had been frustrated by the tools to manage APT repositories. The only ones I knew of either covered too little (only adding/removing packages from a repository and such, like reprepro) or were way too complex (like the official tools used by Debian itself). Maybe/probably I’m a moron and I just didn’t know of some tool that would solve all my problems, but now it’s kind of late ;-) And before you say it, no, Launchpad is not what I was looking for as far as I understand it.
So I started to work on my own suite of tools for it, and recently I decided to release what I’ve done so far. It’s by no means complete, but it’s very useful for me and I thought it would be useful for others. And, with a bit of luck, someone will help me improving it.
So what is it? Arepa (it stands for “Apt REPository Assistant”, but obviously I called it like that after the yummy Venezuelan sandwiches) is a suite of tools that allow you to manage an APT repository. It contains two command-line tools and a web interface, and its main features are:
Manages the whole process after a package arrives to the upload queue: from approving it to re-building from source to signing the final repository.
It allows you to “approve” source packages uploaded to some “incoming” directory, via a web interface.
It only accepts source packages, and those are re-compiled automatically in the configured autobuilders. It can even “cross-compile” for other distributions (treated like binNMUs).
The approval via some web interface was actually sort of the driving force for the project. One of my pet peeves was that there wasn’t an easy way to have an upload queue and easily approve/reject packages with the tools I knew. From what I had seen, the tools were either for “single person” repositories (no approval needed because the package author is the owner of the repository) or full-blown distribution-size tools like dak and such. My use-case, however, is the following:
You have an installation of Arepa for an entire organisation (say, a whole company or a big department).
People inside that organisation upload packages to the upload queue (possibly using dput; the point is, the end up in some directory in the machine hosting Arepa).
Someone (or a small group of people) are the “masters” of the repository, and they’ll have access to the web interface. From time to time they check the web UI, and they’ll approve (or not) the incoming source packages.
If they’re approved, the source will be added to the repository and it’ll be scheduled for compilation in the appropriate combination(s) of architectures and distributions.
A cronjob compiles pending packages every hour; when the compilation is successful, they’re added to the repository.
At this point, the repository hosted by the Arepa installation has the new packages, but you probably want to serve the repository from a different machine. If that’s the case, Arepa can sync the repository to your production machine with a simple command (“arepa sync”).
I imagine that a lot of people have the same need, so I uploaded all the code to CPAN (you can see it with the rest of the contributions by Opera Software). Sadly there’s a silly bug in the released code (I wanted to release ASAP to be able to focus on other things, and I ended up rushing the release), but it has both a workaround and a patch. So, please give it a try if you’re interested and tell me if you would like to contribute. I haven’t released the code in GitHub or similar yet, but I’ll probably do if there’s interest.
Jun 28, 2009
I have been using Perl for many years, but I had never uploaded anything to CPAN. That’s unfortunate, because I’ve probably written several programs or modules that could have been useful for other people. The point is, now I have. Not only that, but it was code I wrote at work, so if I’m not mistaken these are my first contributions to free software from Opera. Yay me!
The two modules I’ve released so far are:
Parse::Debian::PackageDesc, a module for parsing both
.changesfiles from Debian. This is actually a support module for something bigger that I hope I’ll release soon-ish.
As I feel that Migraine could be useful to a lot of people, but it’s easy to misunderstand what it really does (unless you already know Rails migrations of course), I’ll elaborate a bit. Imagine that you are developing some application that uses a database. You design the schema, write some SQL file with it, and everybody creates their own databases from that file. Now, as your application evolves, your schema will evolve too. What do you do now to update all databases (every developer installation, testing installations, and don’t forget the production database)? One painful way to do it could be documenting which SQL statements you have to execute in order to have the latest version of the schema, and expect people to apply copying-and-pasting from the documentation. However, it’s messy, confusing, and it needs someone to know both which databases to update and when.
Migraine offers a simpler, more reliable way to keep all your databases up to date. Basically, you write all your changes (“migrations”) in some files in a directory, following a simple version number naming convention (e.g.
002-change_passwd_field_type.sql), and migraine will allow you to keep your databases up to date. In the simplest, most common case, you call migraine with a configuration file specifying which database to upgrade, and it will figure out which migrations are pending to apply, if any, and apply them. The system currently only supports raw SQL, but it should be easy to extend with other types.
In principle, you shouldn’t need to write any Perl code to use migraine (it has a Perl module that you can use to integrate with your Perl programs if you like, but also a command-line tool), so you can use it even in non-Perl projects. Of course, some modern ORMs have their own database migration system, but very often you have to maintain legacy code that doesn’t use any fancy ORM, or you don’t like the migration system provided by the ORM, or you prefer keeping a single system for schema and data migrations… I think in those cases Migraine can help a lot reducing chaos and keeping things under control. Try it out and tell me what you think
In a couple of days I’ll blog again about other contributions to free software I’ve made lately, but this time in the form of Opera widgets…
Jun 1, 2009
I haven’t written in some time, I know. I haven’t done much worth blogging about. Just a new release of the Kiva World Loanmeter widget, and also a couple of things at work that I’ll be releasing soon (including a small tool for managing database changes and some Perl module to parse Debian
However, recently I watched a really funny and interesting talk at TED, Are we in control of our own decisions?, by Dan Ariely. In the talk he mentions his book, Predictably Irrational, which funnily enough a friend had already mentioned to me.
Well, I just finished the book and I have to say it was very interesting and eye-opening. It’s interesting how it shows our minds are biased for certain kinds of decisions or behaviour, even though they are often not the best for us. Some of the experiments are truly brilliant and they show totally unexpected (at least before starting reading the book ;-P) outcomes. One of the experiments that got me thinking was this:
> > Research on stereotypes shows […] that stereotyped people themselves react differently when they are aware of the label that they are forced to wear […] One stereotype of Asian-Americans, for instance, is that they are especially gifted in mathematics and science. A common stereotype of females is that they are weak in mathematics […] In a remarkable experiment, […] asked Asian-American women to take an objective math exam. But first they divided the women into two groups. The women in one group were asked questions related to their gender […] The women in the second group were asked questions related to their race […] The performance of the two groups differed in a way that matched the stereotypes of both women and Asian-Americans. Those who had been reminded that they were women performed worse than those who had been reminded that they were Asian-American. > >
I can’t stop thinking about the implications this has to working conditions and productivity in different countries, and also to project management.
May 10, 2009
I’ve been working on something lately that I hope I will publish sometime next month: it’s a set of tools to manage an APT package repository. The idea is that, given an upload queue (you can set it up as an anonymous FTP, or some directory accessible via SSH/SCP, or whatever floats your boat in your setup and team), you’ll have a web interface to approve those packages, a set of integrated autobuilders building the approved packages in whatever combination of architectures and distributions you want, and all that integrated with reprepro to keep your repository updated. I’ll write more about it when I have released something.
The point now is that, while working on it, I needed some module to parse command-line options and “subcommands” (like
svn update, etc.). As it’s written in Perl, I had a look at CPAN to see if I could see anything. The most promising module was App::Rad, but it lacked a couple of things that were very important for me: my idea was “declaring” all the possible commands and options and have the module do all the work for me (generating the help pages and the default
--helpimplementation, generate the
program help subcommandand so on).
App::Raddidn’t have that, and it didn’t seem to me like that was the direction they wanted to go to with the module. But I figured I’d drop the author an e-mail anyway and see if he liked the idea so I could start adding support for all that…
And boy was that a good idea. He replied a couple of days later, and said that they had liked the idea so much that they had implemented it already (that’s why he took a couple of days to reply), and he sent me an example of the new syntax they had introduced and asked if that was what I was thinking. And not only that, but they added me to the list of contributors just for giving the idea! That completely made my day, free software rocks!
Nov 23, 2008
I have always thought that systems should be something integrated. Each “system” has its own conventions, cultural values, etc. and I think you have to respect that. I believe in the Debian way (adapting programs to an integrated system, not just creating a large collection of packages that are identical to the upstream versions), I like to adapt my style of programming to the language (indentation conventions, identifiers, tools for building and testing, etc.), I prefer cross-platform applications that look and feel like each platform they run on, etc.
In the same way, I feel that the mere idea of having a programming-language-dependent packaging system is a broken idea. I know it has advantages, and I know that being specific to the language, some things work better or are more flexible, but I just don’t believe in that idea. Why should I use a different packaging system for certain things just because they’re written in Ruby? Why do I, as a user of those programs/modules, even have to know that there’s some Ruby-specific packaging system, that doesn’t integrate at all with my system’s packaging system, and mixing both leads to a mess?
Not only that, but Rubygems in particular is quite hostile to repackaging into a platform-specific packaging system. A lot of people only provide the gems for their software, which are harder to work with than “normal” tarballs. They also use their own conventions for directories, that break the FHS (for example) and basically only make sense in the context of Rubygems. In that sense, CPAN is much better (although I think using it for application deployment is a very bad idea, but that’s a different matter), because at least it installs everything in sane directories, it doesn’t change Perl in any way, and it’s not a special format, just a repository of easy-to-install, easy-to-work-with, easy-to-hack, easy-to-repackage “distributions”.
Why, oh, why?