[liboping] Running into some limits in the vicinity of 1024 hosts (exactly 1021+, in fact)

Sun Apr 16 12:55:24 CEST 2017

I have attached a testcase so that you can explore the same phenomena.
Please bear in mind that I am both rusty and relatively inexperienced with
C debugging strategies, this reached beyond my gdb depth for example and
appears to crash within the ping_send function anyway. ;)

I have tested this on two computers. On either, testing 1021 rounds works
fine:

# ./testcase 1021
Testing 1021 simultaneous hosts.
Testing send...[
]... send test did not crash.
Done!

On 1022 or above, the first computer I tested (Debian Linux) would
terminate like this:

>>>>>>>>>>>>>>>
# ./testcase 1022
Testing 1022 simultaneous hosts.
Testing send...[
*** buffer overflow detected ***: ./testcase terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x731af)[0x7f0c80eae1af]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f0c80f33aa7]
/lib/x86_64-linux-gnu/libc.so.6(+0xf6cc0)[0x7f0c80f31cc0]
/lib/x86_64-linux-gnu/libc.so.6(+0xf8a17)[0x7f0c80f33a17]
/usr/lib/x86_64-linux-gnu/liboping.so.0(+0x1a4d)[0x7f0c811e7a4d]
./testcase[0x400b66]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f0c80e5cb45]
./testcase[0x4008c9]
======= Memory map: ========
00400000-00401000 r-xp 00000000 ca:01 2506811
 /root/dev/pingcore_v7/testcase
00601000-00602000 rw-p 00001000 ca:01 2506811
 /root/dev/pingcore_v7/testcase
016be000-01742000 rw-p 00000000 00:00 0
 [heap]
7f0c80c25000-7f0c80c3b000 r-xp 00000000 ca:01 450563
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f0c80c3b000-7f0c80e3a000 ---p 00016000 ca:01 450563
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f0c80e3a000-7f0c80e3b000 rw-p 00015000 ca:01 450563
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f0c80e3b000-7f0c80fdc000 r-xp 00000000 ca:01 450699
/lib/x86_64-linux-gnu/libc-2.19.so
7f0c80fdc000-7f0c811dc000 ---p 001a1000 ca:01 450699
/lib/x86_64-linux-gnu/libc-2.19.so
7f0c811dc000-7f0c811e0000 r--p 001a1000 ca:01 450699
/lib/x86_64-linux-gnu/libc-2.19.so
7f0c811e0000-7f0c811e2000 rw-p 001a5000 ca:01 450699
/lib/x86_64-linux-gnu/libc-2.19.so
7f0c811e2000-7f0c811e6000 rw-p 00000000 00:00 0
7f0c811e6000-7f0c811ea000 r-xp 00000000 ca:01 3424331
 /usr/lib/x86_64-linux-gnu/liboping.so.0.2.10
7f0c811ea000-7f0c813e9000 ---p 00004000 ca:01 3424331
 /usr/lib/x86_64-linux-gnu/liboping.so.0.2.10
7f0c813e9000-7f0c813ea000 r--p 00003000 ca:01 3424331
 /usr/lib/x86_64-linux-gnu/liboping.so.0.2.10
7f0c813ea000-7f0c813eb000 rw-p 00004000 ca:01 3424331
 /usr/lib/x86_64-linux-gnu/liboping.so.0.2.10
7f0c813eb000-7f0c8140b000 r-xp 00000000 ca:01 450677
/lib/x86_64-linux-gnu/ld-2.19.so
7f0c815fc000-7f0c815ff000 rw-p 00000000 00:00 0
7f0c81607000-7f0c8160b000 rw-p 00000000 00:00 0
7f0c8160b000-7f0c8160c000 r--p 00020000 ca:01 450677
/lib/x86_64-linux-gnu/ld-2.19.so
7f0c8160c000-7f0c8160d000 rw-p 00021000 ca:01 450677
/lib/x86_64-linux-gnu/ld-2.19.so
7f0c8160d000-7f0c8160e000 rw-p 00000000 00:00 0
7fff40c1b000-7fff40c3c000 rw-p 00000000 00:00 0
 [stack]
7fff40d13000-7fff40d14000 r-xp 00000000 00:00 0
 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
 [vsyscall]
Aborted
<<<<<<<<<<<<<<<

The second computer would complain about too many open files:

$ sudo ./testcase
[sudo] password for jesse:
No apparent arguments offered.
Testing 2000 simultaneous hosts.
Received an error on iteration 1022.: -1 (Too many open files)

Until I upped the ulimit, and then it also just buffer overflowed.
(incidentally, my first computer simply already had a larger ulimit.)

>>>>>>>>>>>>>>>
$ ulimit -n 40960
$ ulimit -n
40960
$ sudo ./testcase 1022
Testing 1022 simultaneous hosts.
Testing send...[
*** buffer overflow detected ***: ./testcase terminated
======= Backtrace: =========
/lib/i386-linux-gnu/libc.so.6(+0x67377)[0xb7604377]
/lib/i386-linux-gnu/libc.so.6(__fortify_fail+0x68)[0xb7694548]
/lib/i386-linux-gnu/libc.so.6(+0xf5738)[0xb7692738]
/lib/i386-linux-gnu/libc.so.6(+0xf743a)[0xb769443a]
/usr/lib/i386-linux-gnu/liboping.so.0(+0x15ce)[0xb77545ce]
./testcase[0x80489c8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf7)[0xb75b5637]
./testcase[0x80486e1]
======= Memory map: ========
08048000-08049000 r-xp 00000000 09:7f 2491072
 /home/jesse/dev/pingcore_v7/testcase
08049000-0804a000 r--p 00000000 09:7f 2491072
 /home/jesse/dev/pingcore_v7/testcase
0804a000-0804b000 rw-p 00001000 09:7f 2491072
 /home/jesse/dev/pingcore_v7/testcase
08c11000-08c72000 rw-p 00000000 00:00 0          [heap]
b759c000-b759d000 rw-p 00000000 00:00 0
b759d000-b774c000 r-xp 00000000 09:7f 12589310   /lib/i386-linux-gnu/
libc-2.23.so
b774c000-b774d000 ---p 001af000 09:7f 12589310   /lib/i386-linux-gnu/
libc-2.23.so
b774d000-b774f000 r--p 001af000 09:7f 12589310   /lib/i386-linux-gnu/
libc-2.23.so
b774f000-b7750000 rw-p 001b1000 09:7f 12589310   /lib/i386-linux-gnu/
libc-2.23.so
b7750000-b7753000 rw-p 00000000 00:00 0
b7753000-b7757000 r-xp 00000000 09:7f 10748919
/usr/lib/i386-linux-gnu/liboping.so.0.2.11
b7757000-b7758000 r--p 00003000 09:7f 10748919
/usr/lib/i386-linux-gnu/liboping.so.0.2.11
b7758000-b7759000 rw-p 00004000 09:7f 10748919
/usr/lib/i386-linux-gnu/liboping.so.0.2.11
b775a000-b7776000 r-xp 00000000 09:7f 12583781
/lib/i386-linux-gnu/libgcc_s.so.1
b7776000-b7777000 rw-p 0001b000 09:7f 12583781
/lib/i386-linux-gnu/libgcc_s.so.1
b7777000-b777a000 rw-p 00000000 00:00 0
b777a000-b777c000 r--p 00000000 00:00 0          [vvar]
b777c000-b777d000 r-xp 00000000 00:00 0          [vdso]
b777d000-b779f000 r-xp 00000000 09:7f 12589302   /lib/i386-linux-gnu/
ld-2.23.so
b779f000-b77a0000 rw-p 00000000 00:00 0
b77a0000-b77a1000 r--p 00022000 09:7f 12589302   /lib/i386-linux-gnu/
ld-2.23.so
b77a1000-b77a2000 rw-p 00023000 09:7f 12589302   /lib/i386-linux-gnu/
ld-2.23.so
bf8b7000-bf8d8000 rw-p 00000000 00:00 0          [stack]
Aborted (core dumped)
<<<<<<<<<<<<<<<

Does this represent a limitation in the library, or an instance where my
environment needs more room built out somewhere, like a larger stack or
heap or something analogous to the ulimit?

Is this a limit that I might be able to raise if I simply try to hack the
liboping source and rebuild, and/or what kind of scaleability tradeoffs
will I encounter if I try? For my purposes it is not strictly out of the
question to limit my broad ping swaths to batches of 1,000 or so, but on
the other hand I am looking to target only about 3,000 hosts as it is, so
batching things sounds relatively premature to me unless there are some
important scaleability concerns I need to be keeping in mind. *shrugs*

Let me know your thoughts, and thanks again for building a library that all
of my testing suggests is working rather awesomely in the 1,000 host range
at least. :D

- - Jesse
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.verplant.org/pipermail/liboping/attachments/20170416/f2e999f1/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: testcase.c
Type: text/x-csrc
Size: 2506 bytes
Desc: not available
URL: <http://mailman.verplant.org/pipermail/liboping/attachments/20170416/f2e999f1/attachment.c>