gary is currently certified at Master level.

Name: Gary Benson
Member since: 2001-05-31 00:39:54
Last Login: 2012-10-24 12:18:42

FOAF RDF Share This

Homepage: http://gbenson.net/

Notes:

I work for Red Hat.

Projects

Articles Posted by gary

Recent blog entries by gary

Syndication: RSS 2.0

Remote debugging with GDB

→ originally posted on developerblog.redhat.com

This past few weeks I’ve been working on making remote debugging in GDB easier to use. What’s remote debugging? It’s where you run GDB on one machine and the program being debugged on another. To do this you need something to allow GDB to control the program being debugged, and that something is called the remote stub. GDB ships with a remote stub called gdbserver, but other remote stubs exist. You can write them into your own program too, which is handy if you’re using minimal or unusual hardware that cannot run regular applications… cellphone masts, satellites, that kind of thing. I bet you didn’t know GDB could do that!

If you’ve used remote debugging in GDB you’ll know it requires a certain amount of setup. You need to tell GDB how to access to your program’s binaries with a set sysroot command, you need to obtain a local copy of the main executable and supply that to GDB with a file command, and you need to tell GDB to commence remote debugging with a target remote command.

Until now. Now all you need is the target remote command.

This new code is really new. It’s not in any GDB release yet, let alone in RHEL or Fedora. It’s not even in the nightly GDB snapshot, it’s that fresh. So, with the caveat that none of these examples will work today unless you’re using a Git build, here’s some things you can do with gdbserver using the new code.

Here’s an example of a traditional remote debugging session, with the things you type in bold. In one window:

abc$ ssh xyz.example.com
xyz$ gdbserver :9999 --attach 5312
Attached; pid = 5312
Listening on port 9999

gdbserver attached to process 5312, stopped it, and is waiting for GDB to talk to it on TCP port 9999. Now, in another window:

abc$ gdb -q
(gdb) target remote xyz.example.com:9999
Remote debugging using xyz.example.com:9999
...lots of messages you can ignore...
(gdb) bt
#0 0x00000035b5edf098 in *__GI___poll (fds=0x27467a0, nfds=8,
timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:83
#1 0x00000035b76449f9 in ?? () from target:/lib64/libglib-2.0.so.0
#2 0x00000035b76451a5 in g_main_loop_run ()
from target:/lib64/libglib-2.0.so.0
#3 0x0000003dfd34dd17 in gtk_main ()
from target:/usr/lib64/libgtk-x11-2.0.so.0
#4 0x000000000040913d in main ()

Now you have GDB on one machine (abc) controlling process 5312 on another machine (xyz) via gdbserver. Here I did a backtrace, but you can do pretty much anything you can with regular, non-remote GDB.

I called that a “traditional” remote debugging session because that’s how a lot of people use this, but there’s a more flexible way of doing things if you’re using gdbserver as your stub. GDB and gdbserver can communicate over stdio pipes, so you can chain commands, and the new code to remove all the setup you used to need makes this really nice. Lets do that first example again, with pipes this time:

abc$ gdb -q
(gdb) target remote | ssh -T xyz.example.com gdbserver - --attach 5312
Remote debugging using | ssh -T xyz.example.com gdbserver - --attach 5312
Attached; pid = 5312
Remote debugging using stdio
...lots of messages...
(gdb)

The “-” in gdbserver’s argument list replaces the “:9999” in the previous example. It tells gdbserver we’re using stdio pipes rather than TCP port 9999. As well as configuring everything with single command, this has the advantage that the communication is through ssh; there’s no security in GDB’s remote protocol, so it’s not the kind of thing you want to do over the open internet.

What else can you do with this? Anything you can do through stdio pipes! You can enter Docker containers:

(gdb) target remote | sudo docker exec -i e0c1afa81e1d gdbserver - --attach 58
Remote debugging using | sudo docker exec -i e0c1afa81e1d gdbserver - --attach 58
Attached; pid = 58
Remote debugging using stdio
...

Notice how I slipped sudo in there too. Anything you can do over stdio pipes, remember? If you’re using Kubernetes you can use kubectl exec, or with OpenShift osc exec.

gdbserver can do more than just attach, you can start programs with it too:

(gdb) target remote | sudo docker exec -i e0c1afa81e1d gdbserver - /bin/sh
Remote debugging using | sudo docker exec -i e0c1afa81e1d gdbserver - /bin/sh
Process /bin/sh created; pid = 89
stdin/stdout redirected
Remote debugging using stdio
...

Or you can start it without any specific program, and then tell it what do do from within GDB. This is by far the most flexible way to use gdbserver. You can control more than one process, for example:

(gdb) target extended-remote | ssh -T root@xyz.example.com gdbserver --multi -
Remote debugging using | gdbserver --multi -
Remote debugging using stdio
(gdb) attach 774
...messages...
(gdb) add-inferior
Added inferior 2
(gdb) inferior 2
[Switching to inferior 2 [<null>] (<noexec>)]
(gdb) attach 871
...messages...
(gdb) info inferiors
Num Description Executable
* 2 process 871 target:/usr/sbin/httpd
  1 process 774 target:/usr/libexec/mysqld

Ready to debug that connection issue between your webserver and database?

Syndicated 2015-04-30 13:14:25 from gbenson.net

Judgement Day

GDB will be the weapon we fight with if we accidentally build Skynet.

Syndicated 2015-04-17 20:07:19 from gbenson.net

libiberty demangler fuzzer

I wrote a quick fuzzer for libiberty’s demangler this morning. Here’s how to get and use it:

$ mkdir workdir
$ cd workdir
$ git clone git@github.com:gbenson/binutils-gdb.git src
  ...
$ cd src
$ git co demangler
$ cd ..
$ mkdir build
$ cd build
$ CFLAGS="-g -O0" ../src/configure --with-separate-debug-dir=/usr/lib/debug
  ...
$ make
  ...
$ ../src/fuzzer.sh 
  ...
+ /home/gary/workdir/build/fuzzer
../src/fuzzer.sh: line 12: 22482 Segmentation fault      (core dumped) $fuzzer
# copy the executable and PID from the two lines above
$ gdb -batch /home/gary/workdir/build/fuzzer core.22482 -ex "bt"
...
Core was generated by `/home/gary/workdir/build/fuzzer'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000040d5f2 in op_is_new_cast (op=0x7ffffac19930) at libiberty/cp-demangle.c:3064
3064	  const char *code = op->u.s_operator.op->code;
#0  0x000000000040d5f2 in op_is_new_cast (op=0x7ffffac19930) at libiberty/cp-demangle.c:3064
#1  0x000000000040dc6f in d_expression_1 (di=0x7ffffac19c90) at libiberty/cp-demangle.c:3232
#2  0x000000000040dfbc in d_expression (di=0x7ffffac19c90) at libiberty/cp-demangle.c:3315
#3  0x000000000040cff6 in d_array_type (di=0x7ffffac19c90) at libiberty/cp-demangle.c:2821
#4  0x000000000040c260 in cplus_demangle_type (di=0x7ffffac19c90) at libiberty/cp-demangle.c:2330
#5  0x000000000040cd70 in d_parmlist (di=0x7ffffac19c90) at libiberty/cp-demangle.c:2718
#6  0x000000000040ceb8 in d_bare_function_type (di=0x7ffffac19c90, has_return_type=0) at libiberty/cp-demangle.c:2772
#7  0x000000000040a9b2 in d_encoding (di=0x7ffffac19c90, top_level=1) at libiberty/cp-demangle.c:1287
#8  0x000000000040a6be in cplus_demangle_mangled_name (di=0x7ffffac19c90, top_level=1) at libiberty/cp-demangle.c:1164
#9  0x0000000000413131 in d_demangle_callback (mangled=0x7ffffac19ea0 "_Z1-Av23*;cG~Wo2Vu", options=259, callback=0x40ed11 , opaque=0x7ffffac19d70)
    at libiberty/cp-demangle.c:5862
#10 0x0000000000413267 in d_demangle (mangled=0x7ffffac19ea0 "_Z1-Av23*;cG~Wo2Vu", options=259, palc=0x7ffffac19dd8) at libiberty/cp-demangle.c:5913
#11 0x00000000004132d6 in cplus_demangle_v3 (mangled=0x7ffffac19ea0 "_Z1-Av23*;cG~Wo2Vu", options=259) at libiberty/cp-demangle.c:6070
#12 0x0000000000401a87 in cplus_demangle (mangled=0x7ffffac19ea0 "_Z1-Av23*;cG~Wo2Vu", options=259) at libiberty/cplus-dem.c:858
#13 0x0000000000400ea0 in main (argc=1, argv=0x7ffffac1a0a8) at /home/gary/workdir/src/fuzzer.c:26

The symbol that crashed it is one frame up from the bottom of the backtrace.

Syndicated 2014-05-15 13:20:10 from gbenson.net

Run-time linker interface

If you’re debugging an application that loads thousands of shared libraries then be sure to read the LinkerInterface page on the GDB wiki.

Syndicated 2013-10-25 15:52:14 from gbenson.net

r_debug

I’ve been trying to figure out how to get information about libraries loaded with dlmopen out of glibc‘s runtime linker and into GDB.

The current interface uses a structure called r_debug that’s defined in link.h. If the executable’s dynamic section has a DT_DEBUG element, the runtime linker sets that element’s value to the address where this structure can be found. I tried to discover where this interface originated, but I didn’t get very far. The only mention of it I found anywhere in any standard is in the System V Application Binary Interface, where it says:

If an object file participates in dynamic linking, its program header table will have an element of type PT_DYNAMIC. This “segment” contains the .dynamic section. A special symbol, _DYNAMIC, labels the section…

and later:

DT_DEBUG
This member is used for debugging. Its contents are not specified for the ABI; programs that access this entry are not ABI-conforming.

No help there then. In glibc, r_debug looks like this:

struct r_debug
{
  int r_version;              /* Version number for this protocol.  */

  struct link_map *r_map;     /* Head of the chain of loaded objects.  */

  /* This is the address of a function internal to the run-time linker,
     that will always be called when the linker begins to map in a
     library or unmap it, and again when the mapping change is complete.
     The debugger can set a breakpoint at this address if it wants to
     notice shared object mapping changes.  */
  ElfW(Addr) r_brk;
  enum
    {
      /* This state value describes the mapping change taking place when
         the `r_brk' address is called.  */
      RT_CONSISTENT,          /* Mapping change is complete.  */
      RT_ADD,                 /* Beginning to add a new object.  */
      RT_DELETE               /* Beginning to remove an object mapping.  */
    } r_state;

  ElfW(Addr) r_ldbase;        /* Base address the linker is loaded at.  */
};

With glibc, r_version == 1. At least some versions of Solaris have r_version == 2, and when this is the case there are three extra fields, r_ldsomap, r_rdevent, r_flags. GDB uses r_ldsomap if r_version == 2; the other two seem to be the interface with librtld_db. That’s not documented anywhere to my knowledge, and may not even be fixed: applications are supposed to use the external interface to librtld_db as documented here.

Here is the problem: r_debug, as it stands, has no way to access more than one namespace. The objects in r_map are the default namespace, directly linked, or opened with dlopen, or opened with dlmopen with lmid set to LM_ID_BASE. The r_ldsomap field in Solaris’s r_debug gives access to the linker’s namespace, opened with dlmopen with lmid set to LM_ID_LDSO, but you still can’t see any other namespaces.

glibc uses multiple r_debug structures internally, one per namespace. It would be trivial to add a “next r_debug” link to r_debug if it were possible to extend the structure, but to do this you’d need to set r_version > 2. Applications could arguably expect a runtime linker with r_version > 2 to support the version 2 interface in full, but it wouldn’t be possible to do that in glibc without reverse engineering Solaris’s implementation. glibc is therefore stuck at r_version == 1, and the r_debug structure is effectively immutable for all time.

Syndicated 2013-08-30 11:03:04 from gbenson.net

270 older entries...

 

gary certified others as follows:

  • gary certified mjcox as Master
  • gary certified joe as Master
  • gary certified tromey as Master
  • gary certified Anthony as Master
  • gary certified leonardr as Journeyer
  • gary certified pedro as Apprentice
  • gary certified nutella as Apprentice
  • gary certified robilad as Master
  • gary certified mjw as Master
  • gary certified timp as Journeyer
  • gary certified aph as Master
  • gary certified twisti as Master
  • gary certified avdyk as Journeyer
  • gary certified gnuandrew as Journeyer

Others have certified gary as follows:

  • chromatic certified gary as Apprentice
  • olandgren certified gary as Apprentice
  • MikeGTN certified gary as Apprentice
  • hub certified gary as Journeyer
  • jao certified gary as Apprentice
  • uweo certified gary as Apprentice
  • sye certified gary as Master
  • cannam certified gary as Journeyer
  • jooon certified gary as Apprentice
  • mike750 certified gary as Journeyer
  • hacker certified gary as Apprentice
  • sad certified gary as Journeyer
  • cdent certified gary as Apprentice
  • opiate certified gary as Journeyer
  • johnnyb certified gary as Apprentice
  • slef certified gary as Apprentice
  • jarod certified gary as Journeyer
  • maelstorm certified gary as Journeyer
  • mjcox certified gary as Journeyer
  • Ausmosis certified gary as Journeyer
  • monk certified gary as Journeyer
  • jono certified gary as Journeyer
  • fxn certified gary as Journeyer
  • ignatz certified gary as Journeyer
  • sethcohn certified gary as Journeyer
  • rupert certified gary as Journeyer
  • larry certified gary as Journeyer
  • sdodji certified gary as Journeyer
  • StevenRainwater certified gary as Journeyer
  • mharris certified gary as Journeyer
  • TheCorruptor certified gary as Journeyer
  • Ilan certified gary as Master
  • criswell certified gary as Master
  • DV certified gary as Journeyer
  • sprite certified gary as Master
  • richdawe certified gary as Journeyer
  • mulix certified gary as Journeyer
  • jrf certified gary as Journeyer
  • rkrishnan certified gary as Journeyer
  • nixnut certified gary as Journeyer
  • jcv certified gary as Journeyer
  • Omnifarious certified gary as Journeyer
  • e8johan certified gary as Journeyer
  • mjw certified gary as Journeyer
  • badvogato certified gary as Master
  • tromey certified gary as Master
  • robilad certified gary as Master
  • reenoo certified gary as Master

[ Certification disabled because you're not logged in. ]

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

X
Share this page