Fun with gdb

Ben Elliston just gave a great lunchtime talk at OzLabs on gdb. Ben has actually read the gdb manual, and in doing so has discovered a lot of neat tricks!

One that he missed in his talk is some cute stuff you can do with the p command (the short form for ‘print’). For example, say you have some process that is sending its output to /dev/null, but you want to see that output. That can happen with long running daemons that were started without debugging enabled for example. What you need to do is tell that program to change file descriptor 1 to point to a file, instead of /dev/null.

Let’s look at an example:


 lsof -n |grep null|grep 1u
 gconfd-2  17243     tridge    1u      CHR                1,3              5961 /dev/null

We see from lsof that gconfd-2 has pid of 17243 and is redirecting stdout (fd 1) to /dev/null. Now let’s attach with gdb, close fd 1, and re-open it on a file in /tmp


 tridge@blu:~$ gdb --pid 17243
 GNU gdb 6.8-debian
 Copyright (C) 2008 Free Software Foundation, Inc.
 License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
 This is free software: you are free to change and redistribute it.
 There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
 and "show warranty" for details.
 This GDB was configured as "x86_64-linux-gnu".
 Attaching to process 17243

ok, we’re attached, now the dirty work:


 (gdb) p close(1)
 $1 = 0
 (gdb) p fopen("/tmp/gconfd.log", "w")
 $2 = 27041792
 (gdb) p fileno($2)
 $6 = 1
 (gdb) quit

Now if we check with lsof, we see the fd is now open on a log file


 gconfd-2  17243     tridge    1u      REG                8,1        0 12277473 /tmp/gconfd.log

You may ask why I used fopen() and not open(). First off, “w” is easier than remembering the right flag bits to open. Secondly, open may in fact be open64 or some other symbol, whereas I’ve found that fopen() always seems to work.

Notice also that I checked that fopen() gave us the right file descriptor back by calling fileno() on the output of fopen(). It will always give you the lowest available fd, which is usually right. If it isn’t the right one then use p with dup2() to move it to the right one.

Another use

This type of hack can also be used to solve the problem of unmounting a filesystem that has a process running with a file open on the filesystem read-write. The kernel doesn’t allow that, and the usual approach is to kill off all those processes. But what if you don’t want one of those processes to die? You can use the above trick to move the fd to point at a different filesystem. By using lsof to find all the ones you need to move, you can eventually mount the filesystem read-only, or even unmount it. This allows you to swap filesystems with running processes using them. You can minimise the effect on the processes by using gdb -x to script it (so it isn’t paused waiting for nervous fingers to type the right commands).

2 Responses to “Fun with gdb”

  1. sanjeev says:

    neat.

  2. Sam Watkins says:

    Sounds like this could be useful.

Leave a Reply