If you need any introduction you should really read the first part. I’ll just present the other three ideas I wanted to cover.

Focusing on common cases

This consists of testing only/mostly common cases. These tests rarely fail and give a false sense of security. Thus, tests are better when they also include less common cases, as they’re much more likely to break inadvertently. Common cases not only break far less often, but will probably be caught reasonably fast once someone tries to use the buggy code, so testing them has comparatively less value than testing less common cases.

The best example I found was in the wrap_string tests. The relevant example was adding the string “A test of string wrapping…”, which wraps not to two lines, but three (the wrapping is done only on spaces, so “wrapping…” is taken as a single unit; in this sense, my test case could have been clearer and use a very long word, instead of a word followed by ellipsis). Most of the cases we’ll deal with will simply wrap a given word in two lines, but wrapping in three must work, too, and it’s much more likely to break if we decide to refactor or rewrite the code in that function, with the intention to keep the functionality intact.

See other examples of this in aa20bce (no tests with more than one consecutive newline, no tests with lines of only non-printable characters), b248b3f (no tests with just dots, no valid cases with more than one consecutive slash, no invalid cases with content other than slashes), 5e771ab (no directories or hidden files), f8ecac5 (invalid hex characters don’t fail, but produce strange behaviour instead; this test actually discovered a bug), 7856643 (broken escaped content) and 87e9f89 (trailing garbage).

Not trying to make the tests fail

This is related to the previous one, but the emphasis is on trying to choose tests that we think will fail (either now or in the future). My impression is that people often fail to do this because they are trying to prove that the code works, which misses the point of testing. The point is trying to prove the code doesn’t work. And hope that you fail at it, if you will.

The only example I could find was in the strcasecmpend tests. Note how there’s a test that checks that the last three characters of string “abcDEf” (ie. “DEf”) is less than “deg” when compared case-insensitively. That’s almost pointless, because if we made that same comparison case-sensitively (in other words, if the “case” part of the function breaks) the test still passes! Thus it’s much better to compare the strings ”abcdef” and “Deg”.

Addendum: trying to cover all cases in the tests

There’s another problem I wanted to mention. I have seen several times before, although not in the Tor tests. The problem is making complicated tests that try to cover many/all cases. This seems to stem from the idea that having more test cases is good by itself, when actually more tests are only useful when they increase the chances to catch bugs. For example, if you write tests for a “sum” function and you’re already testing [5, 6, 3, 7], it’s probably pointless to add a test for [1, 4, 6, 5]. A test that would increase the chances of catching bugs would probably look more like [-4, 0, 4, 5.6] or [].

So what’s wrong with having more tests than necessary? The problem is they make the test suite slower, harder to understand at a glance and harder to review. If they don’t contribute anything to the chance of catching bugs anyway, why pay that price? But the biggest problem is when we try to cover so many test cases than the code produces the test data. In this cases, we have all the above problems, plus that the test suite becomes almost as complex as production code. Such tests become much easier to introduce bugs in, harder to follow the flow of, etc. The tests are our safety net, so we should be fairly sure that they work as expected.

And that’s the end of the tips. I hope they were useful :-)

These ideas are presented in no particular order. Each of them has a brief explanation, a concrete example from the Tor tests, and, if applicable, pointers to other commits that illustrate the same idea. Before you read on, let me explicitly acknowledge that (1) I know that many people know these principles, but writing about them is a nice reminder; and (2) I’m fully aware that sometimes I need that reminder, too.

Edit:  see the second part of this blog.

Tests as spec

Tests are more useful if they can show how the code is supposed to behave, including safeguarding against future misunderstandings. Thus, it doesn’t matter if you know the current implementation will pass those tests or that those test cases won’t add more or different “edge” cases. If those test cases show better how the code behaves (and/or could catch errors if you rewrite the code from scratch with a different design), they’re good to have around.

I think the clearest example were the tests for the eat_whitespace* functions. Two of those functions end in _no_nl, and they only eat initial whitespace (except newlines). The other two functions eat initial whitespace, including newlines… but also eat comments. The tests from line 2280 on are clearly targeted at the second group, as they don’t really represent an interesting use case for the first. However, without those tests, a future maintainer could have thought that the _no_nl functions were supposed to eat whitespace too, and break the code. That produces confusing errors and bugs, which in turn make people fear touching the code.

See other examples in commits b7b3b99 (escaped ‘%’, negative numbers, %i format string), 618836b (should an empty string be found at the beginning, or not found at all? does “\n” count as beginning of a line? can “\n” be found by itself? what about a string that expands more than one line? what about a line including the “\n”, with and without the haystack having the “\n” at the end?), 63b018ee (how are errors handled? what happens when a %s gets part of a number?), 2210f18 (is a newline only \r\n or \n, or any combination or \r and \n?) and 46bbf6c (check that all non-printable characters are escaped in octal, even if they were originally in hex; check that characters in octal/hex, when they’re printable, appear directly and not in octal).

Testing boundaries

Boundaries of different kinds are a typical source of bugs, and thus are among the best points of testing we have. It’s also good to test both sides of the boundaries, both as an example and because bugs can appear on both sides (and not necessarily at once!).

The best example are the tor_strtok_r_impl tests (a function that is supposed to be compatible with strtok_r, that is, it chops a given string into “tokens”, separated by one of the given separator characters). In fact, these extra tests discovered an actual bug in the implementation (ie. an incompatibility with strtok_r). Those extra tests asked a couple of interesting questions, including “when a string ends in the token separator, is there an empty token in the end?” in the “howdy!” example. This test can also be considered valuable as in “tests as spec”, if you consider that the answer to be above question is not obvious and both answers could be considered correct.

See other examples in commits 5740e0f (checking if tor_snprintf correctly counts the number of bytes, as opposed the characters, when calculating if something can fit in a string; also note my embarrassing mistake of testing snprintf, and not tor_snprintf, later in the same commit), 46bbf6c (check that character 21 doesn’t make a difference, but 20 does) and 725d6ef (testing 129 is very good, but even better with 128—or, in this case, 7 and 8).

Testing implementation details

Testing implementation details tends to be a bad idea. You can usually argue you’re testing implementation details if you’re not getting the test information from the APIs provided by whatever you’re testing. For example, if you test some API that inserts data in a database by checking the database directly, or if you test the result of a method call was correct by checking the object’s internals or calling protected/private methods. There are two reasons why this is a bad idea: first, the more implementation details you tests depend on, the less implementation details you can change without breaking your tests; second, your tests are typically less readable because they’re cluttered with details, instead of meaningful code.

The only example I encountered of this in Tor were the compression tests. In this case it wasn’t a big deal, really, but I have seen this before in much worse situations and I feel this illustrates the point well enough. The problem with that deleted line is that it’s not clear what’s it’s purpose (it needs a comment), plus it uses a magic number, meaning if someone ever changes that number by mistake, it’s not obvious if the problem is the code or the test. Besides, we are already checking that the magic number is correct, by calling the detect_compression_method. Thus, the deleted memcmp doesn’t add any value, and makes our tests harder to read. Verdict: delete!

I hope you liked the examples so far. My next post will contain the second half of the tips.

But enough with the reasons. It’s starting to look like it was a pretty useful mini-project. Not just because I learned a bit more Javascript, the excellent express web development framework and other things, but also because the result itself, even though it didn’t take long to develop (and it was pretty fun, even!), feels useful. It feels like a nice place to put all my notes, TODOs, random ideas for projects, etc. A similar feeling of freedom as when I started using my first Moleskine. Not that I would ditch paper for computer-anything, but it’s useful and freeing in its own way, for specific purposes.

About the technology, I used the Markdown format for the pages thanks to the markdown-js library (it’s really nice that the module has an intermediate tree format that you can parse to add your own stuff before converting to HTML, like e.g. wikipage links!), express for the whole application structure and js-test-driver + jsautotest + a bit of syntax sugar from Sinon.js for the tests (but looking forward to trying out Buster.js when it’s released!). The deployment to Joyent’s Node.js SmartMachine was reasonably easy. Actually, it was pretty easy once I figured the following:

• You must not forget to listen in the correct port, with server.listen(process.env.PORT || 8001)
• There are a couple of pretty useful Node.js-related command-line utilities to check logs, restart applications and so on
• The configuration of the application can be done via npm config, see npm integration on Joyent’s Wiki

If you’re curious to see the code, play with it or use it yourself, take a peek to the Wiki-Toki repository on GitHub. Happy hacking!

So I started to work on my own suite of tools for it, and recently I decided to release what I’ve done so far. It’s by no means complete, but it’s very useful for me and I thought it would be useful for others. And, with a bit of luck, someone will help me improving it.

So what is it? Arepa (it stands for “Apt REPository Assistant”, but obviously I called it like that after the yummy Venezuelan sandwiches) is a suite of tools that allow you to manage an APT repository. It contains two command-line tools and a web interface, and its main features are:

• Manages the whole process after a package arrives to the upload queue: from approving it to re-building from source to signing the final repository.
• It allows you to “approve” source packages uploaded to some “incoming” directory, via a web interface.
• It only accepts source packages, and those are re-compiled automatically in the configured autobuilders. It can even “cross-compile” for other distributions (treated like binNMUs).
• Far from reinventing (many) wheels, it integrates tools like reprepro, GPG, Rsync, debootstrap and sbuild so you don’t have to learn all about them.

The approval via some web interface was actually sort of the driving force for the project. One of my pet peeves was that there wasn’t an easy way to have an upload queue and easily approve/reject packages with the tools I knew. From what I had seen, the tools were either for “single person” repositories (no approval needed because the package author is the owner of the repository) or full-blown distribution-size tools like dak and such. My use-case, however, is the following:

• You have an installation of Arepa for an entire organisation (say, a whole company or a big department).
• People inside that organisation upload packages to the upload queue (possibly using dput; the point is, the end up in some directory in the machine hosting Arepa).
• Someone (or a small group of people) are the “masters” of the repository, and they’ll have access to the web interface. From time to time they check the web UI, and they’ll approve (or not) the incoming source packages.
• If they’re approved, the source will be added to the repository and it’ll be scheduled for compilation in the appropriate combination(s) of architectures and distributions.
• A cronjob compiles pending packages every hour; when the compilation is successful, they’re added to the repository.
• At this point, the repository hosted by the Arepa installation has the new packages, but you probably want to serve the repository from a different machine. If that’s the case, Arepa can sync the repository to your production machine with a simple command (“arepa sync”).

I imagine that a lot of people have the same need, so I uploaded all the code to CPAN (you can see it with the rest of the contributions by Opera Software). Sadly there’s a silly bug in the released code (I wanted to release ASAP to be able to focus on other things, and I ended up rushing the release), but it has both a workaround and a patch. So, please give it a try if you’re interested and tell me if you would like to contribute. I haven’t released the code in GitHub or similar yet, but I’ll probably do if there’s interest.

The initial idea of the loanmeter widget was to show where in the world Kiva was offering loans. However, as I used the widget myself, I realised that the location in the world was less important for me, and I was more interested in knowing what the person was going to use the money for. So, I added some options to filter by “sector” and I figured that having some graphs comparing how much money was requested and already funded, for each sector, would be a very quick and visual way to get the information I wanted. I started playing with flot, and I have to say that except for a couple of relatively minor problems, it was quite easy to use. I don’t have screenshots showing the graphs, but feel free to try the widget itself and have a look (hint: you have two buttons at the bottom right corner to switch between “map view” and “graph view”).

The other widget I have been working on is a monitor widget for projects in CruiseControl.rb (a really simple and neat continuous integration server we use at Opera). More than one year ago, my colleague Nico had written a very quick & dirty widget for monitoring the result of the test runs of the My Opera functional testsuite. There were a couple of things I wanted to change, and I also wanted to monitor other projects, so I figured that I’d rewrite the widget to have a more maintainable codebase and then make it generic, so you could configure which CC.rb installation and which project to monitor. I’m moderately happy with the result of the refactoring, and happy enough with the final result. I know it has several issues, and I expect that once anyone outside our team starts using it, there will be things to improve and fix :-) If you use CruiseControl.rb, give it a try!

• It does have a (lame) name now: Bubug (supposedly stands for “Barely Unconventional Bug Untracking Gizmo”. Whatever).
• It has improved a lot here and there, and it now has authentication and multi-user support, not to mention a lot of UI bling bling and goodies.
• The development has moved to BitBucket, an excellent free service built by ex-Opera’s Jesper Noehr, where you can follow it more easily, comment on, check the Wiki, fork, or whatever you want. You even have a screenshot there ;-)

As you can guess from the last point, for this project I’ve been using Mercurial instead of Git. Although I certainly don’t have sophisticated needs, so YMMV (heavily), I find Git more pleasant to use. Which is kind of surprising, because I always thought that Git’s UI was a pain in the ass. Oh, well. That doesn’t mean that Mercurial is hard to use, though. I think it’s more that I’m used to Git now, and there are a couple of things that I find more convenient: the coloured diff (possible in Hg, but you have to install some extension for it, and only thinking about installing some Python extension that is not even packaged for Debian makes me want to switch to Git) and the staging area are the most important ones I can think of.

So, if you thought I had abandoned the TODO application thing, you were wrong ;-) If you’re interested, have a look at the Bubug BitBucket project page, download it, play with it, and tell me what you think.

The point now is that, while working on it, I needed some module to parse command-line options and “subcommands” (like git commit, svn update, etc.). As it’s written in Perl, I had a look at CPAN to see if I could see anything. The most promising module was App::Rad, but it lacked a couple of things that were very important for me: my idea was “declaring” all the possible commands and options and have the module do all the work for me (generating the help pages and the default --help implementation, generate the program help subcommand and so on). App::Rad didn’t have that, and it didn’t seem to me like that was the direction they wanted to go to with the module. But I figured I’d drop the author an e-mail anyway and see if he liked the idea so I could start adding support for all that…

And boy was that a good idea. He replied a couple of days later, and said that they had liked the idea so much that they had implemented it already (that’s why he took a couple of days to reply), and he sent me an example of the new syntax they had introduced and asked if that was what I was thinking. And not only that, but they added me to the list of contributors just for giving the idea! That completely made my day, free software rocks!

1. Possibility to “postpone” a task, so it doesn’t appear in the main view for a defined time.
2. Possibility to associate a task to a “person to nag”.

When you have a lot of small tasks to do, and they are not the kind of things you put in a BTS (say, stuff that you have to do that is not really connected to some project’s code) I think these two features are really useful, and I was surprised that no applications I saw seemed to have those. I mean, don’t people have the same problems as me?

That’s why, as I had mentioned, I started writing my own TODO application: I’d have what I wanted, and I’d learn a thing or two about Merb, DataMapper and jQuery. The application has several design limitations that I used to simplify things, like not having any notion of users (single user app without authentication) or supporting only a “title” for the tasks, without any longer description. It isn’t something I really plan to publish for other people to use (I mean, the code is in my Darcs repo, I’m just not going to make a project page for it or anything like that), so I don’t really care how much it fits other people’s needs :-)

As it is a pet project and I didn’t really mind how long it would take to finish it, I started by making some mockup of the application in HTML (+ a bit of Javascript with jQuery), and once I was happy, I started with the actual design and code. I think some parts of the code are nice, and it has some Ajax sweetness, but I admit I haven’t used it yet for myself: only as a kind of underpowered BTS for the application. Maybe I’ll upload some screenshot some day. In the meantime, feel free to download and try it out ;-)