Thanks partner! A year of pair programming

This week marks the first anniversary of starting to do daily pair programming with another Samba developer, Andrew Bartlett. It has been an absolutely fantastic experience, and I thought this would be a good time to write up what we’ve been doing.

For many years I’ve been doing ad-hoc pair programming with various people. I have used a variety of techniques, from combining IRC with a shared GNU screen session, VNC sessions, NX sessions and lots of other combinations. What is different about the last year is that it wasn’t just a bit here and a bit there – we started working together on a daily basis.

We started off using long phone calls combined with a VNC session to share one of our screens. I originally used x11vnc to share my desktop, but we’ve more recently moved to using x0vncserver, part of the TigerVNC package. We’ve also moved from SIP phone calls to using a mumble server, which has allowed us to open up our coding sessions to other Samba developers, which has been very helpful.

daily coding sessions

The pattern we’ve got into now is that we both login to mumble around breakfast time. Like Andrew I tend to eat breakfast in front of my computer, while having a quick look at sites like lwn and slashdot. Andrew usually joins mumble around the same time, and we chat about what we are going to work on that day. After we’ve both finished our cereal we dive into coding.

The most common pattern is that Andrew connects to my desktop over VNC, which I have shared over a VPN. I have a script which runs this command in a terminal:

x0vncserver QueryConnect=1 QueryConnectTimeout=30 AlwaysShared=1 PollingCycle=300 ZlibLevel=9 SecurityTypes=None \
     AcceptKeyEvents=off AcceptPointerEvents=off PasswordFile=$HOME/private/VNC/team.pass

that shares my current desktop read-only, and prompts me when someone connects. It also sets up a slow polling cycle, which means the VNC session doesn’t chew too much bandwidth, which means our mumble VOIP session doesn’t degrade. It allows for multiple people to connect, which is really useful when Zahari or Kamen join in from Bulgaria, or Jelmer joins from the Netherlands.

audio setup

Andrew uses a fancy wireless headset which he can switch between being a pulseaudio device for mumble, and being a phone handset for when people ring him. The headset means he can walk around his house while we’re coding, which really suits him, as he often needs to help Kirsty out by hanging nappies on the line or doing a bit of cooking, and we can keep talking about the latest kerberos issue while he’s doing that.

I use a cheap desktop microphone (a Logitech one) along with desktop speakers. That means I don’t need to wear a headset all day, which is much more comfortable. If you adjust the mumble audio settings carefully you can avoid echo and avoid having the noise of typing come through, while still having really clear audio with a desktop microphone. I put the microphone on a rubber mouse mat to isolate it a bit from the desk.

writing code

With the above setup I just use my usual coding environment, which is emacs plus a bunch of GNU screen sessions in a gnome-terminal. Andrew watches me code and test, while we discuss the approach to each problem as it comes up, and he suggests different approaches. When we are working on a piece of code where Andrew is the expert (eg. all the auth code, kerberos etc), we often switch around so I’m connecting to his VNC server instead and I comment on his code while he is writing it. I use a command like this to connect:

 xtightvncviewer -passwd $HOME/private/VNC/abartlet.pass -viewonly -quality 3 -compresslevel 9 server

That gives me a pretty good view, while minimising bandwidth usage. When we need to share files, we either rsync it somewhere, or we push it to a personal branch on a git repository.

the results

The results over the last year have been really amazing. Between the two of us Andrew and I have pushed over 2500 patches to the Samba master repository over a year of pair programming, which is more than twice what we managed  in the previous year. I find it really interesting that despite only one of us typing at a time, we get much more done with pair programming than when we work separately. The results are even more notable when you take into account that in the last year Andrew has been rebuilding his house and looking after a new baby!

I think the reason it works so well is that it tends to minimise procrastination. When I code alone and I’m stuck on a bit of code, I often find myself drifting off to read slashdot or muck about with some new application that I’ve found. That happens a lot less when someone else is watching over your shoulder on VNC. We discuss how we’re going to solve the problem and then we solve it, without the hours of procrastination in between.

The code also ends up as much higher quality, with a lot less bugs. Andrew is great at spotting subtle issues in my code while I’m typing it in, which saves a lot of debugging time.

It’s also interesting that we’ve found that pair programming works a lot better when we aren’t in the same room. We both live in Canberra, which means we could drive over and code right next to each other, but we find we just don’t get nearly as much done that way.

some highlights from the year

Pair programming is also just a lot more fun than programming alone. After having been a lone coder for 20 years, moving to pair programming was a revelation. By being on mumble all day, I get a lot more social interaction with other developers. I think it’s fair to say that most of my social interaction is now over the mumble VOIP link.

One really fun time was during the SNIA CIFS conference in Santa Clara in September and the AD plugfest the following week in Redmond. Andrew couldn’t travel as Celeste had just been born, so I took my desktop microphone with me and used it to allow Andrew to attend the plugfests remotely. That worked extremely well! We setup some speakers attached to my laptop, and again shared my screen with VNC. Andrew was able to participate fully in the plugfests, even if it meant he didn’t get any of the free food. His knowledge of authentication protocols was essential to many of the problems we faced during those two weeks.

There have also been lots of fun moments when Celeste has decided to chime into the conversations as only a 2 month old baby can, with Andrew often holding her on his lap while we are coding. At my end of the link I have a similar effect when my little dog Nessie (a king charles cavalier) comes into my study to ask for a biscuit by whining pathetically. She nearly always gets the biscuit.

Another amusing incident was when I got a letter from my SIP provider saying threatening to cut me off as I was costing them too much with these 8 hour untimed phone calls every day. That was before we discovered mumble, so I just switched to a different SIP provider.

The switch to mumble, which made it easy for us to have several people connected at once, was also a big advance. It allowed Andrew and I to help a number of new Samba developers to find their feet with some very tricky code by working with them directly than we could do over IRC. We’ve spent many hours working with Nadya, Kamen, Anatoliy, Zahari, Kai and many others on some very tricky code, and it has helped them to become better Samba developers. It works well despite them being spread out all over the world.

give it a go!

If you haven’t tried pair programming then you really should give it a try. Find someone else working on the same project as yourself and see if your coding styles are compatible enough for shared screen coding sessions to work. I’m sure it won’t work for everyone, but when it does work its a fantastic way of making yourself both more productive while having a lot of fun.

Thanks partner!

Finally I’d like to say a huge thank you to Andrew for being such a great coding partner for the last year. Your patience when I’m coding something badly has been fantastic, and you’re support when a test just refuses to pass has been brilliant. Thank you!

16 Responses to “Thanks partner! A year of pair programming”

  1. Greg says:

    Hey thanks for the description and update, I’ve heard of pair programming but never really got my head around how it might work in practice. It sounds totally bizarre, but for the reasons you’ve said maybe it is a great thing! I’m a sysadmin so I don’t know how it would translate, but it’s still cool to think about.
    Cheers,
    Greg.

  2. very encouraging story. those are some real impressive metrics.

  3. Abe says:

    After using VNC for a while discovered the beauty of NX. You should give NX (http://www.nomachine.com/) a go if not aware and not tried. Much more responsive than VNC or similar ilk. Intelligently compressed and tunnelled X. Great to know enjoyed peer programming. Cheers Abe.

  4. tridge says:

    yes, I use NX a lot (usually with neatx) for remote admin. I often use a VNC or rdesktop
    client within a NX session, with NX handling the long haul to Europe, then VNC or rdesktop for the LAN hop to a box.

    The problem is that NX is not so good at exporting my existing X desktop, at least using the free software neatx server. So for pair programming I use VNC with setting to reduce the refresh rate.

  5. Jeff Waugh says:

    Hey Tridge,

    GNOME has a built-in VNC server (System > Preferences > Remote Desktop) with support for the DAMAGE extension and so on. Is there a reason you use x0vncserver instead? Does it do additional helpful things to reduce traffic, regardless of how the client is configured, perhaps?

    Thanks,

    - Jeff

  6. John says:

    Great story. I’d done a lot of pair programming for front end web apps; was wondering if anyone was doing it for other engineering efforts. It’s by far the most enjoyable, most productive form of development for me; I’m not doing it as much any more and I really want to get back to it.

  7. Joe Moore says:

    I wonder why you have set it up such that only one person can drive at a time? After 10 years of side-by-side pair programming I have been remote pair programming full time for the last 6 months (though I remote paired often before that) and have always pair with both developers fully enabled to edit.

    I have always found that one of the huge benefits of pair programming is that often the two developer’s “flow cycle”, for lack of a better term, complement each other. When the driving developer gets tired or stuck the other can take over, or if the non-driver has an alternative implementation they can jump right in.

  8. Andrew Bartlett says:

    Jeff,

    Yes, we need to use x0vncserver. When I shared my desktop with tridge, I originally did so with the built in server (because it was easy), but the VoIP would drop out every time I switched windows (particularly if those windows contained photo-like graphics, like the Windows 7 logon screen).

    These would be beautifully rendered, but when our ‘real work’ was in emacs text buffers, and consoles it was just a pain.

    On my client, I am using ‘Remmina’, set to 256 colours for similar reasons (we couldn’t find a way to only allow 256 colours in x0vncserver). I think xtightvncviewer may be vncviewer in Fedora, so I’ll try the script tridge uses above a some point, to see if matching client and server libraries helps things.

  9. tridge says:

    Joe,

    I do sometimes do pair programming with both programmers able to drive. When I do that I tend to use “screen -x” to give us both equal access.

    When using VNC the problem with two drivers is the mouse. Two people controling the same mouse pointer doesn’t work well.

    We could probably make it work better if we had the server RW, but enabled viewonly in the clients when not typing.

    Cheers, Tridge

  10. Joe Moore says:

    tridge — In my experience two people with one mouse and keyboard works just fine, just as it does when you are pairing side by side. I’ve been doing it for years, and many of my coworkers at Pivotal Labs have been, too.

    Can you describe what doesn’t work well for you? If the issue is that one person is switching context, scrolling, etc, when the other is in the middle of something, then you have a people problem, not a technology problem.

    We have found that when remote pairing the pair needs to be more aware of basic pairing etiquette — it’s all about communication, patience, and common courtesy. We find our selves saying the following often:

    - “Can I drive for a second? I want to see something.”
    - “I’m almost done, then you can drive.”
    - “Let me know when you’re done reading.”
    - “Hold on a sec, I need to finish this thought…”
    - “Do you mind if we switch for a while?”

    It might take a few hours to adjust, but soon it’s almost the same as pairing in person.

    BTW, we use VNC (actually Mac “Screen Sharing.app”) and Skype + Video. While pairing with voice only works, adding video makes a huge difference. You can see when your pair is distracted, in the zone, looking elsewhere, laughing, smiling, unhappy, has their hands behind their head, etc.

    I’m very excited to find more people finding ways to pair program, especially remotely. Pairing can sound very strange to people who have never tried it, and remote pairing programming even stranger. But I feel that despite the productivity gains you have realized that you have left even more productivity on the table. I have found to be one of the best aspects of pairing is capturing the ebb and flow, the rhythm, the dance of two programmers. I encourage you to try dual editing again with vnc, perhaps adding video to the mix, if not together then with others you might pair with in the future.

    Another thought: though I don’t know you two personally, I’m deducing from this post that tridge, not Andrew, is the main developer, and that Andrew was fine letting tridge do most of the coding. While this works for you, it likely will not work well when two dominant, wanna-code-all-the-time developers. A pair like that needs a balance, and a one-driver setup will likely turn them off.

    – Joe

  11. Dave Täht says:

    Interesting. The freeswitch developers use a voice conference room – additionally tied into irc – to do group development. I personally found it both incredibly useful and incredibly distracting – with the (small) added benefit that I could also run a sip client on a different box than my laptop.

    I hadn’t heard of mumble before now. I’ll try it. Perhaps you could improve your interactivity by doing some QoS by prioritizing the mumble packets higher than ssh+vnc?

    I used to use emacs’s “new-frame-on-display” feature to collaborate with others, that used to also be very low bandwidth, very fast, and highly interactive.

    You could prioritize the packets better.

  12. tridge says:

    Hi Joe,

    The mouse is a problem as with Andrew and me it is very common for the person not typing to be looking things up. While I’m writing some boilerplate code Andrew might be checking the WSPP docs, or looking up how we solved a similar problem elsewhere. He also might be answering queries from other Samba developers on IRC. I do the same when Andrew is coding.

    This means we’re not always watching each line of code being written by the other developer, but we watch enough that the pair programming works.

    Regarding a video link, I don’t think that would work for us! I don’t want to watch while Andrew has to change a nappie :-)

    Cheers, Tridge

  13. tdci100 says:

    re: SIP, you probably could have used something like ekiga.net on both side.
    Or better, run a asterisk based appliance (like Askozia) on a VM and invite all your friends over :-)

  14. João Matos says:

    Thank you for this post!
    I’ll try your method for group university projects and other coding projects.
    Hopefully, it’ll reduce my tendency to procrastinate.

    s/you’re support/your support/

  15. ridvan says:

    thanks for this! I’ll share this article to my officemates. We’ve just started pair programming and it was really fun and knowledge-driven. :D

  16. My way of doing remote pair programming involves skype and some kind of screen sharing. What I’ve learned is that the person typing should do it locally. That’s why we push the changes, then switch the presenter and then continue from where we left.

    Shameless plug: I also started a free remote pair programming tour – details here: http://www.alexbolboaca.ro/wordpress/the-remote-pair-programming-tour

Leave a Reply