If you need any introduction you should really read the first part. I’ll just present the other three ideas I wanted to cover.

Focusing on common cases

This consists of testing only/mostly common cases. These tests rarely fail and give a false sense of security. Thus, tests are better when they also include less common cases, as they’re much more likely to break inadvertently. Common cases not only break far less often, but will probably be caught reasonably fast once someone tries to use the buggy code, so testing them has comparatively less value than testing less common cases.

The best example I found was in the wrap_string tests. The relevant example was adding the string “A test of string wrapping…”, which wraps not to two lines, but three (the wrapping is done only on spaces, so “wrapping…” is taken as a single unit; in this sense, my test case could have been clearer and use a very long word, instead of a word followed by ellipsis). Most of the cases we’ll deal with will simply wrap a given word in two lines, but wrapping in three must work, too, and it’s much more likely to break if we decide to refactor or rewrite the code in that function, with the intention to keep the functionality intact.

See other examples of this in aa20bce (no tests with more than one consecutive newline, no tests with lines of only non-printable characters), b248b3f (no tests with just dots, no valid cases with more than one consecutive slash, no invalid cases with content other than slashes), 5e771ab (no directories or hidden files), f8ecac5 (invalid hex characters don’t fail, but produce strange behaviour instead; this test actually discovered a bug), 7856643 (broken escaped content) and 87e9f89 (trailing garbage).

Not trying to make the tests fail

This is related to the previous one, but the emphasis is on trying to choose tests that we think will fail (either now or in the future). My impression is that people often fail to do this because they are trying to prove that the code works, which misses the point of testing. The point is trying to prove the code doesn’t work. And hope that you fail at it, if you will.

The only example I could find was in the strcasecmpend tests. Note how there’s a test that checks that the last three characters of string “abcDEf” (ie. “DEf”) is less than “deg” when compared case-insensitively. That’s almost pointless, because if we made that same comparison case-sensitively (in other words, if the “case” part of the function breaks) the test still passes! Thus it’s much better to compare the strings ”abcdef” and “Deg”.

Addendum: trying to cover all cases in the tests

There’s another problem I wanted to mention. I have seen several times before, although not in the Tor tests. The problem is making complicated tests that try to cover many/all cases. This seems to stem from the idea that having more test cases is good by itself, when actually more tests are only useful when they increase the chances to catch bugs. For example, if you write tests for a “sum” function and you’re already testing [5, 6, 3, 7], it’s probably pointless to add a test for [1, 4, 6, 5]. A test that would increase the chances of catching bugs would probably look more like [-4, 0, 4, 5.6] or [].

So what’s wrong with having more tests than necessary? The problem is they make the test suite slower, harder to understand at a glance and harder to review. If they don’t contribute anything to the chance of catching bugs anyway, why pay that price? But the biggest problem is when we try to cover so many test cases than the code produces the test data. In this cases, we have all the above problems, plus that the test suite becomes almost as complex as production code. Such tests become much easier to introduce bugs in, harder to follow the flow of, etc. The tests are our safety net, so we should be fairly sure that they work as expected.

And that’s the end of the tips. I hope they were useful :-)

These ideas are presented in no particular order. Each of them has a brief explanation, a concrete example from the Tor tests, and, if applicable, pointers to other commits that illustrate the same idea. Before you read on, let me explicitly acknowledge that (1) I know that many people know these principles, but writing about them is a nice reminder; and (2) I’m fully aware that sometimes I need that reminder, too.

Edit:  see the second part of this blog.

Tests as spec

Tests are more useful if they can show how the code is supposed to behave, including safeguarding against future misunderstandings. Thus, it doesn’t matter if you know the current implementation will pass those tests or that those test cases won’t add more or different “edge” cases. If those test cases show better how the code behaves (and/or could catch errors if you rewrite the code from scratch with a different design), they’re good to have around.

I think the clearest example were the tests for the eat_whitespace* functions. Two of those functions end in _no_nl, and they only eat initial whitespace (except newlines). The other two functions eat initial whitespace, including newlines… but also eat comments. The tests from line 2280 on are clearly targeted at the second group, as they don’t really represent an interesting use case for the first. However, without those tests, a future maintainer could have thought that the _no_nl functions were supposed to eat whitespace too, and break the code. That produces confusing errors and bugs, which in turn make people fear touching the code.

See other examples in commits b7b3b99 (escaped ‘%’, negative numbers, %i format string), 618836b (should an empty string be found at the beginning, or not found at all? does “\n” count as beginning of a line? can “\n” be found by itself? what about a string that expands more than one line? what about a line including the “\n”, with and without the haystack having the “\n” at the end?), 63b018ee (how are errors handled? what happens when a %s gets part of a number?), 2210f18 (is a newline only \r\n or \n, or any combination or \r and \n?) and 46bbf6c (check that all non-printable characters are escaped in octal, even if they were originally in hex; check that characters in octal/hex, when they’re printable, appear directly and not in octal).

Testing boundaries

Boundaries of different kinds are a typical source of bugs, and thus are among the best points of testing we have. It’s also good to test both sides of the boundaries, both as an example and because bugs can appear on both sides (and not necessarily at once!).

The best example are the tor_strtok_r_impl tests (a function that is supposed to be compatible with strtok_r, that is, it chops a given string into “tokens”, separated by one of the given separator characters). In fact, these extra tests discovered an actual bug in the implementation (ie. an incompatibility with strtok_r). Those extra tests asked a couple of interesting questions, including “when a string ends in the token separator, is there an empty token in the end?” in the “howdy!” example. This test can also be considered valuable as in “tests as spec”, if you consider that the answer to be above question is not obvious and both answers could be considered correct.

See other examples in commits 5740e0f (checking if tor_snprintf correctly counts the number of bytes, as opposed the characters, when calculating if something can fit in a string; also note my embarrassing mistake of testing snprintf, and not tor_snprintf, later in the same commit), 46bbf6c (check that character 21 doesn’t make a difference, but 20 does) and 725d6ef (testing 129 is very good, but even better with 128—or, in this case, 7 and 8).

Testing implementation details

Testing implementation details tends to be a bad idea. You can usually argue you’re testing implementation details if you’re not getting the test information from the APIs provided by whatever you’re testing. For example, if you test some API that inserts data in a database by checking the database directly, or if you test the result of a method call was correct by checking the object’s internals or calling protected/private methods. There are two reasons why this is a bad idea: first, the more implementation details you tests depend on, the less implementation details you can change without breaking your tests; second, your tests are typically less readable because they’re cluttered with details, instead of meaningful code.

The only example I encountered of this in Tor were the compression tests. In this case it wasn’t a big deal, really, but I have seen this before in much worse situations and I feel this illustrates the point well enough. The problem with that deleted line is that it’s not clear what’s it’s purpose (it needs a comment), plus it uses a magic number, meaning if someone ever changes that number by mistake, it’s not obvious if the problem is the code or the test. Besides, we are already checking that the magic number is correct, by calling the detect_compression_method. Thus, the deleted memcmp doesn’t add any value, and makes our tests harder to read. Verdict: delete!

I hope you liked the examples so far. My next post will contain the second half of the tips.

1. The NodeIterator/NodeFilter Javascript API. I ended up not using it for the extension, but it’s good to know.
2. The DOM event “DOMNodeInserted”, very useful when you want to do some work based on new elements “appearing” on the page (as it’s more and more common).
3. Javascript regular expression lookahead/lookbehind. The latter, which I needed, is not supported by Javascript, so I had to use a lookbehind mimicking trick/workaround to get what I wanted.
4. It gave me a bit more experience in Test-Driven Development. Which reminds me: if you are interested in Javascript and TDD and you happen to understand some Norwegian, have a look at the excellent zombietdd screencast series!

As always, the code is on GitHub, so you can read it, fork it, make your own extension based on it or whatever you want.

EDIT: I forgot to mention that the meme smiley extension got quite popular because it was featured in the Choose Opera blog (twice!) and then “recommended” in the Opera extensions homepage :-D

As you can imagine, the conference was great. It was my first big conference and my first time outside of Europe, so it was doubly exciting for me. And even though there were many interesting talks, meeting all that bunch of testing nerds was much better. It shows that Google really worked hard to make people socialise.

But let’s start from the beginning. Probably around a year ago now I had written a talk about testability that I had submitted to EuroSTAR 2010, but had been rejected. That had been my third rejection I believe, so I started losing hopes that I’d ever speak at an international conference. However, relatively shortly after being rejected Google announced this year’s event, and the theme was “Test to testability”, so I said “what the hell!”.

They said from the start that it would be invitation-only, meaning you had to apply even for simply attending. That was actually pretty cool, because the idea was that Google would choose the attendees, and once selected and notified, those attendees would vote for each other’s talks to decide what the program would be. It would also mean that attendees would be chosen because they had something interesting to add to the conference, not simply money to pay the registration fee.

And one day, right before leaving for a short vacation, I received the news that I had been chosen to attend. At that point, of course, I had no idea if I would actually talk, but just attending was awesome and I was really happy and a bit surprised (I was going to a conference! in India!). A couple of days later I received a lot of proposed talks to rate. That was pretty exciting, and seeing a lot of very interesting topics was kind of cool, because it was so promising, but also a bit discouraging, because I thought the chances of getting chosen were pretty low. Still I didn’t lose all hope, and when the deadline came, I was notified that I had been chosen to talk. At that point I was pretty surprised, but when I kept reading and saw that there were only 8 talks selected (+ 3 keynotes), then I was pretty shocked.

The rest of the story you don’t have to imagine, because in the typical Google fashion, all the conference material is available on the website (both videos and slides). As I imagine that the conference page link will break sooner or later, I’ll just give you the official GTAC 2010 YouTube playlist. My favourite talks were (in order of appearance):

• Twist, A Next Generation Functional Testing Tool – really nice tool and very good demo, although not being open source and being for Eclipse was kind of a let down
• The Future of Front-End Testing – kind of everything a professional QA Engineer should know about front-end testing, but it’s not always the case; I thought it was kind of basic, but it was a useful reminder and listening to Simon Stewart is just fun
• Flexible Design? Testable Design? You Don’t Have To Choose! – great talk with unit testing tips/patterns; one of the nice things is that those patterns are not only for statically-typed languages
• Crowd Source Testing, Mozilla Community Style – very nice talk about making the community help you testing complex products, with many examples and details

I guess I should also mention “Measuring and Monitoring Experience in Interactive Streaming Applications” and “Turning Quality on its Head“. The first, because I thought it was a cool story about how hard it is to find bugs that are important for users, but are vague and hard to reproduce. The second, mostly because of the tool that James shows off. You can see screenshots and an explanation of it from minute 52.

About my own talk, “Lessons Learned from Testability Failures“, I was really worried that I was going to freak out and block on stage. After all, I was used to talking in front of 5, 10, 20 or maybe 30 people. Speaking in front of around 100 and knowing that I was being recorded for YouTube (and that a lot of people interested in the subject would watch those videos) was quite scary in itself. And then there was the other factor: I usually speak to people who (theoretically) know less than me about that concrete subject, but it wasn’t like that at all in this case. However, people there were so cool and friendly that I felt less nervous than I usually feel. Watching the video, I do look nervous the first minutes, but after the introduction and such it felt really good. Kudos to the organisation and the attendees for being so open, cool and friendly. Meeting all that crowd was clearly the best of going to be conference.

All in all, it was a great experience and I made a lot of contacts and friends, and I’m looking forward to attending another similar conference (maybe next year’s GTAC?). We’ll see.

How to create a story

Stories have four elements: audience, ingredients, structure and medium.

Audience

There are two important relationships in stories: story-audience and you-audience. About the first, you want to include details that fill the gap, and also stories are a good way to make the audience see a different perspective by feeling it. Finally, endings are important. They should be memorable and settled (“take them home”).

Ingredients

See checklist on p. 209.

• Perspective. there isn’t a neutral POV in stories. Types of perspectives are realist (3rd person, “absent” author), confessional (focused on author experience) and impressionist (mixes descriptions of events with a strong structure). The last intends to spark ideas/actions and while they can have an ending, they might end with implicit question. An easy way to add perspective is letting the main character do the talking.
• Characters. One of the reasons why UX stories are useful is because they add specificity and texture to the usually one-dimensional view of users. Also useful to highlight needs outside the mainstream. Tips to build characters: (1) choose (only) details that add meaning; (2) show, don’t tell (show in action instead of describing traits); (3) set up “hooks” that you can use later in the story; (4) leave room for imagination.
• Context. Five types: physical (time, date, location, location scale), emotional (how characters feel), sensory (5 senses), historical (“when phones had dials”), memory (storyteller’s memory, flashbacks).
• Imagery. Things that make us picture the story (example in p. 205). Don’t use too much!
• Language. Tips: (a) speak in the language of the characters, (b) make the story active, (c) focus on telling the story, not describing, (d) don’t judge characters, context or events.

Structure/plot

Structure is the framework/skeleton of the story. Plot is the arrangement of the events. Strong structures help the audience, the author and the story (p. 215). See types of stories on p. 216. “Checklist” for good structure and plot on p. 235.

Medium

Four big media: oral (mind the gap to written, p. 243), written (make the point explicit, keep it short, make use of cultural cues as in p. 253), visual (comics and storyboards work, see p. 258-260), multimedia/video.

See tips on how to integrate stories in reports on p. 265 and p. 266. See strong sides of different media on p. 272.

Mini-review and conclusions

I quite liked the book, although I admit that the last part (the one summarised in this post) was a bit disappointing. I guess it’s hard to give tips about something as complex as creating a story, in a book. The book has a very clear structure and it’s easy to follow and read, which helps in figuring out what to read, what to skim and what to leave for later.

Another thing that really struck me while reading the book (the second book I read following the tips from “How to Read a Book“) is how little I used to understand of the books I read. I now go through the book three times: one to get an idea of the structure and the most interesting parts, one to read the content, and one to review and make a summary. So even while I was reading it for the last time, I made sense of things that I hadn’t realised while reading the book (and that was after knowing the structure, knowing what to expect from each chapter, and having made some preliminary notes!). Not only that, but I also feel that I’m much more critical with what I read and I compare it much more with what I think myself.

If you aren’t doing it already, I strongly recommend that you give those tips a try…

There shouldn’t be “a storyteller” in the team, as many as possible should be familiar with the technique. Prototypes based on stories allow exploration of new ideas, esp. if they’re big changes.

There are several parts of the UX process were stories are useful:

• Collecting input from users. You’re already hearing those stories. Do it consciously.
• Exploring user research and other data. Summary of hard data.
• Experimenting with design ideas. See stories that help launch a design discussion (type) and the role “spark new ideas”.
• Testing designs. They can evaluate if you have stayed true to original needs and if they will work with real users.

Collecting input

Being in the user work environment helps noticing things people don’t mention. When you just arrive, everything is unfamiliar. Take notes then. If you can’t talk to your users, you can get some limited info from: search/server logs, customer service records, people who do training and sales demos, market research and satisfaction surveys.

Getting people in groups can help make people talk (build on each other). Also asking people to recall specific events is really useful.

Tip: be open to tangents, but don’t waste too much time in them if you don’t see value. Also, a user avoiding talking about what you want is information, too.

Tip: Use a structure for the interview (first closed questions, then open), see p. 82. Try to have the interview in the context the product will be used.

Selecting stories

Characteristics of good stories:

• Heard from more than one source
• With action detail
• Make user data easy to understand
• Illustrate an aspect the UX team is interested in
• Surprise or contradict common beliefs

They should help explain something about UX beyond data, bring data to life. They should also connect with other stories and resonate, leading to action.

Experimenting with design ideas

Three possible uses of stories: brainstorming, concept and specification. When no user research is available, you can brainstorm to create user stories. See a good technique/game for it on page 111.

When you do have user research, you can develop those stories. For that, some rules: (1) defer judgement, (2) encourage wild ideas and (3) build on the ideas of others. See adaptation of the game for this case, p. 118. Concept stories should include: (a) focus on activity set in a specific context, (b) description of motivations that trigger action, (c) describe the characters well enough to set them in context.

Specification stories are useful to summarise results. They are included in specs. They keep the real-world context available for reference.

Testing/evaluating designs

Three uses of stories: create scenarios/tasks for usability testing, serve as guide for expert reviews, and quality testing.

If in usability testing you ask the user first what her interests are, you can turn that story into a usability test task.

Stories and personas from them are very useful to set a context for expert reviews. Give each expert a persona and make them try to complete a task from that persona POV.

[I didn't really get the "quality testing" part, whatever that means, so I don't have notes about it]

Sharing stories

When communicating with people, stories get the audience attention, set context and inspire action.

Listening exercises make you understand your audience, and make them understand how diverse/similar they are.

There are three typical types of audiences:

1. Strategic leaders: generate and maintain a common vision (p. 143). Things that work for them: identify point of pain and offer a solution, identify gap in market and show how to fill, show new approach by reconfiguring common/existing components, and identify UX trends and show impact on business.
2. Managers: have a mission and have to make decisions (p. 146). Don’t have time to spare, prefer short meetings to brainstorming sessions. If you bring bad news, show why it’s important to care. Don’t go into much detail.
3. Technical experts: implement a vision (p. 149). Can be difficult to reach them with stories, esp. if not grounded in details. Tips: (a) use representative characters and situations and be ready to back up with hard data, (b) make the action of the story specific and tangible, (c) keep the story on track, (d) use technical terminology accurately.

And that was the end of this part of the summary. In the next and last post I’ll cover the tips about creating stories and will write some sort of mini-review and conclusions.

Introduction (chapters 1-2)

Stories help keeping people at the center (p. 2). There are different types of stories (p. 5):

• Those that describe context/situation: describe the world today. Not only sequence of events, but also reasons and motivations.
• Those that illustrate problems: show a problem that a new product or design change can fix. They should describe it in a way that opens the door for brainstorming.
• Those that help launch a design discussion: starting point for a brainstorming session. Enough detail to make sense but leave room for the imagination.
• Those that explore a design concept: explain/explore idea or concept and its implications for the experience. Helps shape the design by showing it in action.
• Those that prescribe the result of a new design: describe the world as it will be in more detail. Similar to the 1st, but describe a user experience that doesn’t exist yet.

Interesting quote in page 10, with the message “until you hear a story, you can’t understand the experience”.

Stories are interactive, change with the audience (p. 14). They also not only describe actions, but add context and why (motivation). There is a fine line with how many detail to include in motivation, because of shared cultural understanding and other things (p. 17, 19).

Stories have different roles:

• Explain: give context and sensory experience, not just events. This is different from use-cases.
• Engage the imagination: surpass linear logic and evoke new ideas.
• Spark new ideas: as we fill in the gaps, we can hint details but let people come up with their own ideas.
• Create a shared understanding.
• Persuade.

In any case, stories are not “made up”: they’re based on data.

Listening (chapter 3)

Really listening to users (e.g. in interviews and such) gives you access to a lot of info you can’t get anywhere else. Open questions are very important for this. Giving time to answer sometimes gives people time for second thoughts (not just what they think you want to hear), which has more value than the first reply. Also, pay attention to the context, people forget to mention “obvious” (for them) everyday facts.

Practising active listening is very important, see the following links:

• Eight barriers to effective listening
• 10 Tips to Effective & Active Listening Skills

And that’s it for the first part. Stay tuned for the rest of the summary.

Different stuff I’ve been reading or otherwise been exposed to in the last weeks has made me reach a sort of funny comparison: code is (or can be) like science. You come up with some “theory” (your code) that explains something (solves a problem)… and you make sure you can measure it and test it for people to believe your theory and build on top of it.

I mean, something claiming to be science that can’t be easily measured, compared or peer-reviewed would be ridiculous. Scientists wouldn’t believe in it and would certainly not build anything on top of it because the foundation is not reliable.

I claim that software should be the same way, and thus it’s ridiculous to trust software that doesn’t have a good test suite, or even worse, that may not even be particularly testable. Trusting software without a test suite is not that different from taking the word of the developer that it “works on my machine”. Scientists would call untested science pseudo-science, so I am tempted to call code without tests pseudo-code.

Don’t get me wrong: sure you can test by hand, and hand-made tests are useful and necessary, but that only proves that the exact code you tested, without any changes, works as expected. But you know what? Software changes all the time, so that’s not a great help. If you don’t have a way to quickly and reliably measure how your code behaves, every time you make a change you are taking a leap of faith. And the more leaps of faith you take, the less credible your code is.

So anyway, recently Opera released 10.60, which is awesome news because finally Linux has a modern stable release, because of the amount of new eye candy in the UI, the new supported web standards (like Geolocation or WebM video, yay!) and… because of the amazing speed (“much faster than a potato“).

On Saturday, DailyTech published an article comparing the speed of several browsers, Opera 10.60 included. Obviously the conclusion was that Opera is the fastest (I wouldn’t link to that article from this post if it wasn’t the case, would I? :-P), and shortly after reading that, I came across this hilarious video that sort of follows up on that:

I mean, the video even mentions Opera Link, I have to like it :-P (although yeah, the claim is not correct, Chrome does have something similar). My favourite quotes are:

• “You promised innovation, but look at Opera!”
• “Maybe Opera is hiring”

And the second reminded me that yes, we are hiring!

But that meant I had to edit video. Which I had never done before. And I figured it wouldn’t be trivial if I only wanted to use Free Software tools under Linux. I was partly wrong, because after looking around a bit I found OpenShot, which I found pleasant enough to use (at least for my very basic, very limited needs). However, the final footage I used made OpenShot export corrupted videos. I know it was something specific to that source video (a MOV format, H.264 codec, EPICly HD resolution (1920×1080) video) because I had tried to do exactly the same things with earlier, lower-resolution, MPEG-format takes, and it had worked like a charm.

In any case, I was sort of fucked because I couldn’t get the final edited video out, so I had to resize it and change the format somehow. I won’t list here everything I tried (that includes trying to download and use several programs on Windows, as well as using mencoder on Linux), but after a very long and frustrating process, only ffmpeg did the trick for me. My first attempt with ffmpeg did export the video, but with awful quality. After looking around a bit, I found what worked for me:

ffmpeg -i original.mov -s hd720 -b 3200k resized.mpeg

The trick to get a decent result was forcing the bitrate (“-b” option), which will hopefully help someone in need. Meanwhile, I’m going to stop typing so I can go back to crossing my fingers to get picked for GTAC ;-)