Ah! I'm pretty sure I've found the issue.<br><br>In csnmp_read_table, there is the following comparison:<br><br> if ((value_table_ptr[i] != NULL)<br> && (vb->name[vb->name_length - 1] <= value_table_ptr[i]->subid))<br>
{<br> DEBUG ("snmp plugin: host = %s; data = %s; i = %i; "<br> "SUBID is not increasing.",<br> host->name, data->name, i);<br> continue;<br> }<br><br>
This makes the (what appears to be invalid) assumption that while walking a subtree, the last entry of the SUBID will always be incremented to indicate the SUBID is increasing. This is not necessarily the case. In fact, I don't think you can even make an assumption about which value of the OID will always be increasing. An SNMP walk of the example below shows the OID increasing first at SUBID 7, then 8, 18, back to 8, then 8, then 30.<br>
<br>snmpwalk -v 2c -c communitystring target SNMPv2-SMI::enterprises.5951.4.1.3.1.1.43<br>SNMPv2-SMI::enterprises.5951.4.1.3.1.1.43.16.83.78.86.45.82.84.66.45.66.105.100.45.72.84.84.80 = STRING: "30569"<br>SNMPv2-SMI::enterprises.5951.4.1.3.1.1.43.16.83.78.86.45.82.84.66.45.66.117.115.45.72.84.84.80 = STRING: "30585"<br>
SNMPv2-SMI::enterprises.5951.4.1.3.1.1.43.19.83.78.86.45.82.84.66.45.66.117.115.45.72.84.84.80.45.67.83 = STRING: "30585"<br>SNMPv2-SMI::enterprises.5951.4.1.3.1.1.43.23.83.78.86.45.82.84.66.45.83.97.110.100.98.111.120.45.72.84.84.80.45.67.83 = STRING: "0"<br>
SNMPv2-SMI::enterprises.5951.4.1.3.1.1.43.23.83.78.86.45.82.84.66.45.83.97.110.100.98.111.120.45.72.84.84.80.45.86.83 = STRING: "0"<br><br>That's where snmpwalk stops. An snmpgetnext on that last OID gives the following response:<br>
<br>SNMPv2-SMI::enterprises.5951.4.1.3.1.1.44.16.83.78.86.45.82.84.66.45.66.105.100.45.72.84.84.80 = STRING: "478439694"<br><br>_Now_ we're in a different SUBID because of the change on 7 from "43" to "44". However, since the code is checking for vb->name[vb->name_length - 1] <= value_table_ptr[i]->subid, we never get past the second and third snmpgetnext calls because they both end in the same subid, "80". In other words, we can definitely assert that should vb->name[vb->name_length - 1] > value_table_ptr[i]->subid, then the SUBID has changed. However, we can't conversely assume that just because vb->name[vb->name_length - 1] == value_table_ptr[i]->subid, the SUBID has not changed.<br>
<br>So instead of checking for only the last subid, it seems like we should be doing something like (pseudo-code):<br><div class="gmail_quote"><br>increasing = false;<br>for(i = 0; i < length_of_oid && !increasing; i++){<br>
if(new_oid[i] > old_oid[i]){<br> increasing = true;<br> }<br>}<br><br>if(!increasing){<br> WARNING ("snmp plugin: host = %s; data = %s; i = %i; "<br> "SUBID is not increasing.",<br>
host->name, data->name, i);<br> continue;<br>}<br><br><br>The biggest problem I see is that we're only saving "subid" which is the last value of a very long string we need to check. I'm not comfortable changing the entire value_table_ptr structure so we can save the required OID information.<br>
<br>Mark<br><br><br>On Tue, Sep 11, 2012 at 8:52 AM, Mark <span dir="ltr"><<a href="mailto:gajillion@gmail.com" target="_blank">gajillion@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Sorry to reply to my own thread. I've been digging into this further and there seems to be something wrong in the way that the Instance/Subtree iterator is working. Apologies for the std_traffic example - it works fine now - but it highlights the issue.<br>
<br>Here's a dump of the example below of std_traffic (truncated for space), and the plugin working properly:<br><br>Q: interfaces.ifTable.ifEntry.ifInOctets interfaces.ifTable.ifEntry.ifOutOctets interfaces.ifTable.ifEntry.ifDescr<br>
R: interfaces.ifTable.ifEntry.ifInOctets.1=3774870817 interfaces.ifTable.ifEntry.ifOutOctets.1=3774870817 interfaces.ifTable.ifEntry.ifDescr.1="lo"<br>Q: interfaces.ifTable.ifEntry.ifInOctets.1 interfaces.ifTable.ifEntry.ifOutOctets.1 interfaces.ifTable.ifEntry.ifDescr.1<br>
R: interfaces.ifTable.ifEntry.ifInOctets.2=1268816808 interfaces.ifTable.ifEntry.ifOutOctets.2=1391460740 interfaces.ifTable.ifEntry.ifDescr.2="eth0"<br>Q: interfaces.ifTable.ifEntry.ifInOctets.2 interfaces.ifTable.ifEntry.ifOutOctets.2 interfaces.ifTable.ifEntry.ifDescr.2<br>
R: interfaces.ifTable.ifEntry.ifInOctets.3=0 interfaces.ifTable.ifEntry.ifOutOctets.3=0 interfaces.ifTable.ifEntry.ifDescr.3="eth1"<br>Q: interfaces.ifTable.ifEntry.ifInOctets.3 interfaces.ifTable.ifEntry.ifOutOctets.3 interfaces.ifTable.ifEntry.ifDescr.3<br>
R: interfaces.ifTable.ifEntry.ifInUcastPkts.1=32813168 interfaces.ifTable.ifEntry.ifOutUcastPkts.1=32813168 interfaces.ifTable.ifEntry.ifType.1=24<br><br>As you can see, all three values iterate through their trees until they reach ifInUcastPkts, ifOutUcastPkts, and ifType respectively. The plugin correctly recognizes that the subtree has been exited and quits.<br>
<br>Now here's the netscaler configuration:<div><br><br> <Data "ns_vservers"><br> Type "ns_vserver"<br> Table true<br></div> Instance "SNMPv2-SMI::enterprises.5951.4.1.3.1.1.1" # Indexed by virtual server name<br>
Values "SNMPv2-SMI::enterprises.5951.4.1.3.1.1.43" # traffic per vserver<br> </Data><br><br>And here's what I see from the dump (truncated)<br><br>Q: E:5951.4.1.3.1.1.43 E:5951.4.1.3.1.1.1<br>
R: E:5951.4.1.3.1.1.43.16.83.78.86.45.82.84.66.45.66.105.100.45.72.84.84.80="27178" E:5951.4.1.3.1.1.1.16.83.78.86.45.82.84.66.45.66.105.100.45.72.84.84.80="VSERVER-1"<br>Q: E:5951.4.1.3.1.1.43.16.83.78.86.45.82.84.66.45.66.105.100.45.72.84.84.80 E:5951.4.1.3.1.1.1.16.83.78.86.45.82.84.66.45.66.105.100.45.72.84.84.80<br>
R: E:5951.4.1.3.1.1.43.16.83.78.86.45.82.84.66.45.66.117.115.45.72.84.84.80="27204" E:5951.4.1.3.1.1.1.16.83.78.86.45.82.84.66.45.66.117.115.45.72.84.84.80="VSERVER-2"<br>Q: E:5951.4.1.3.1.1.43.16.83.78.86.45.82.84.66.45.66.105.100.45.72.84.84.80 E:5951.4.1.3.1.1.1.16.83.78.86.45.82.84.66.45.66.117.115.45.72.84.84.80<br>
R: E:5951.4.1.3.1.1.43.16.83.78.86.45.82.84.66.45.66.117.115.45.72.84.84.80="27204" E:5951.4.1.3.1.1.1.19.83.78.86.45.82.84.66.45.66.117.115.45.72.84.84.80.45.67.83="VSERVER-3"<br>Q: E:5951.4.1.3.1.1.43.16.83.78.86.45.82.84.66.45.66.105.100.45.72.84.84.80 E:5951.4.1.3.1.1.1.19.83.78.86.45.82.84.66.45.66.117.115.45.72.84.84.80.45.67.83<br>
<br>As you can see, the "5951.4.1.3.1.1.1" values are being iterated through, but the "5951.4.1.3.1.1.43" receives its first response:<br><br>R: E:5951.4.1.3.1.1.43.16.83.78.86.45.82.84.66.45.66.105.100.45.72.84.84.80="27178"<br>
<br>Queries with it, receives a different response (as expected)<br><br>R: E:5951.4.1.3.1.1.43.16.83.78.86.45.82.84.66.45.66.117.115.45.72.84.84.80="27204"<br><br>But then, goes back and continues to requery with the first response:<br>
<br>Q: E:5951.4.1.3.1.1.43.16.83.78.86.45.82.84.66.45.66.105.100.45.72.84.84.80<br><br>This continues, well... forever. The Instance MIB keeps iterating and iterating until it's way out if its tree, but the Value MIB insists on querying the initial value it received. The plugin gets stuck in a loop and never logs any data.<br>
<br>Any help would be appreciated.<span><font color="#888888"><br><br>Mark</font></span><div><div><br><br><br><br><div class="gmail_quote">On Fri, Sep 7, 2012 at 2:21 PM, Mark <span dir="ltr"><<a href="mailto:gajillion@gmail.com" target="_blank">gajillion@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thanks Octo - I'm still having issues though and I'm beginning to wonder if it's my implementation or a larger issue. I reduced my config to the standard table example in the default configuration:<br>
<br> <Data "std_traffic"><br>
Type "if_octets"<br> Table true<br> Instance "IF-MIB::ifDescr"<br> Values "IF-MIB::ifInOctets" "IF-MIB::ifOutOctets"<br> </Data><br><br> <Host "myhost"><br>
Address "myhost"<br> Version 2<br> Community "communitystring"<br> Collect "std_traffic"<br> </Host><br><br>This simple configuration doesn't work. If my understanding is correct, "Instance" provides an iterator of sorts which is then used to get the values listed in "Values". tcpdump tells me all the calls are being done at once:<br>
<br>11:07:07.416822 IP collectd_server.48042 > myhost.snmp: C=communitystring GetNextRequest(59) interfaces.ifTable.ifEntry.ifInOctets interfaces.ifTable.ifEntry.ifOutOctets interfaces.ifTable.ifEntry.ifDescr<br><br>
As expected, this returns the first interface's information (as verified by a manual snmpgetnext):<br><br>IF-MIB::ifInOctets.1 = Counter32: 3740191310<br><br>Since the box has 3 interfaces, (lo, eth0, eth1), in my thinking this process should repeat 2 more times, which it does:<br>
<br>11:07:07.420375 IP myhost.snmp > collectd_server.48042: C=communitystring GetResponse(74) interfaces.ifTable.ifEntry.ifInOctets.1=3739077099 interfaces.ifTable.ifEntry.ifOutOctets.1=3739077099 interfaces.ifTable.ifEntry.ifDescr.1="lo"<br>
11:07:07.420816 IP collectd_server.48042 > myhost.snmp: C=communitystring GetNextRequest(62) interfaces.ifTable.ifEntry.ifInOctets.1 interfaces.ifTable.ifEntry.ifOutOctets.1 interfaces.ifTable.ifEntry.ifDescr.1<br>11:07:07.422980 IP myhost.snmp > collectd_server.48042: C=communitystring GetResponse(76) interfaces.ifTable.ifEntry.ifInOctets.2=3561114835 interfaces.ifTable.ifEntry.ifOutOctets.2=<a href="tel:4032830435" value="+14032830435" target="_blank">4032830435</a> interfaces.ifTable.ifEntry.ifDescr.2="eth0"<br>
11:07:07.423126 IP collectd_server.48042 > myhost.snmp: C=communitystring GetNextRequest(62) interfaces.ifTable.ifEntry.ifInOctets.2 interfaces.ifTable.ifEntry.ifOutOctets.2 interfaces.ifTable.ifEntry.ifDescr.2<br>11:07:07.425329 IP myhost.snmp > collectd_server.48042: C=communitystring GetResponse(68) interfaces.ifTable.ifEntry.ifInOctets.3=0 interfaces.ifTable.ifEntry.ifOutOctets.3=0 interfaces.ifTable.ifEntry.ifDescr.3="eth1"<br>
<br><br>Now, when I do an snmpwalk on those MIBs, that's where it ends:<br><br># snmpwalk -v 2c -c communitystring myhost IF-MIB::ifInOctets<br>IF-MIB::ifInOctets.1 = Counter32: 3740192294<br>IF-MIB::ifInOctets.2 = Counter32: 3775550612<br>
IF-MIB::ifInOctets.3 = Counter32: 0<br>#<br><br>I understand that GetNextRequest has no way of knowing that this is the end of the MIB so the next call is expected:<br><br>11:07:07.425426 IP collectd_server.48042 > myhost.snmp: C=communitystring GetNextRequest(62) interfaces.ifTable.ifEntry.ifInOctets.3 interfaces.ifTable.ifEntry.ifOutOctets.3 interfaces.ifTable.ifEntry.ifDescr.3<br>
11:07:07.427585 IP myhost.snmp > collectd_server.48042: C=communitystring GetResponse(71) interfaces.ifTable.ifEntry.ifInUcastPkts.1=32690775 interfaces.ifTable.ifEntry.ifOutUcastPkts.1=32690775 interfaces.ifTable.ifEntry.ifType.1=24<br>
<br>However, it seems to me that once it receives that data, it should stop because we're out of the ifInOctets subtree altogether now. Instead, it keeps going:<br><br>11:07:07.427790 IP collectd_server.48042 > myhost.snmp: C=communitystring GetNextRequest(62) interfaces.ifTable.ifEntry.ifInOctets.3 interfaces.ifTable.ifEntry.ifOutOctets.3 interfaces.ifTable.ifEntry.ifType.1<br>
11:07:07.429960 IP myhost.snmp > collectd_server.48042: C=communitystring GetResponse(71) interfaces.ifTable.ifEntry.ifInUcastPkts.1=32690775 interfaces.ifTable.ifEntry.ifOutUcastPkts.1=32690775 interfaces.ifTable.ifEntry.ifType.2=6<br>
11:07:07.430062 IP collectd_server.48042 > myhost.snmp: C=communitystring GetNextRequest(62) interfaces.ifTable.ifEntry.ifInOctets.3 interfaces.ifTable.ifEntry.ifOutOctets.3 interfaces.ifTable.ifEntry.ifType.2<br>11:07:07.432313 IP myhost.snmp > collectd_server.48042: C=communitystring GetResponse(71) interfaces.ifTable.ifEntry.ifInUcastPkts.1=32690775 interfaces.ifTable.ifEntry.ifOutUcastPkts.1=32690775 interfaces.ifTable.ifEntry.ifType.3=6<br>
<br>It keeps crawling the tree and never returns so I get no data. If I wait long enough, I can see it climbing up the processor tree...<br><br>11:09:01.595140 IP myhost.snmp > collectd_server.60949: C=communitystring GetResponse(115) interfaces.ifTable.ifEntry.ifInUcastPkts.1=32694434 interfaces.ifTable.ifEntry.ifOutUcastPkts.1=32694434 25.3.2.1.3.768="GenuineIntel: Intel(R) Xeon(TM) CPU 3.20GHz"<br>
11:09:01.595335 IP collectd_server.60949 > myhost.snmp: C=communitystring GetNextRequest(64) interfaces.ifTable.ifEntry.ifInOctets.3 interfaces.ifTable.ifEntry.ifOutOctets.<a href="tel:3%2025.3.2.1.3.768" value="+13253213768" target="_blank">3 25.3.2.1.3.768</a><br>
11:09:01.596967 IP myhost.snmp > collectd_server.60949: C=communitystring GetResponse(115) interfaces.ifTable.ifEntry.ifInUcastPkts.1=32694434 interfaces.ifTable.ifEntry.ifOutUcastPkts.1=32694434 25.3.2.1.3.769="GenuineIntel: Intel(R) Xeon(TM) CPU 3.20GHz"<br>
11:09:01.597084 IP collectd_server.60949 > myhost.snmp: C=communitystring GetNextRequest(64) interfaces.ifTable.ifEntry.ifInOctets.3 interfaces.ifTable.ifEntry.ifOutOctets.<a href="tel:3%2025.3.2.1.3.769" value="+13253213769" target="_blank">3 25.3.2.1.3.769</a><br>
11:09:01.598697 IP myhost.snmp > collectd_server.60949: C=communitystring GetResponse(115) interfaces.ifTable.ifEntry.ifInUcastPkts.1=32694434 interfaces.ifTable.ifEntry.ifOutUcastPkts.1=32694434 25.3.2.1.3.770="GenuineIntel: Intel(R) Xeon(TM) CPU 3.20GHz"<br>
11:09:01.598890 IP collectd_server.60949 > myhost.snmp: C=communitystring GetNextRequest(64) interfaces.ifTable.ifEntry.ifInOctets.3 interfaces.ifTable.ifEntry.ifOutOctets.<a href="tel:3%2025.3.2.1.3.770" value="+13253213770" target="_blank">3 25.3.2.1.3.770</a><br>
<br>Any ideas what might be going on here? Thanks,<br>
<br>Mark<div><div><br><br><br><br><br><div class="gmail_quote">On Fri, Sep 7, 2012 at 1:46 AM, Florian Forster <span dir="ltr"><<a href="mailto:octo@collectd.org" target="_blank">octo@collectd.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Mark,<br>
<br>
thanks for your detailed information. I've added an appropriate check to<br>
the SNMP plugin, see <<a href="http://octo.cx/07739da" target="_blank">http://octo.cx/07739da</a>>. I've committed this to<br>
the collectd-4.10 branch, so it'll appear in the next bugfix releases.<br>
<br>
Best regards,<br>
—octo<br>
<span><font color="#888888">--<br>
collectd – The system statistics collection daemon<br>
Website: <a href="http://collectd.org" target="_blank">http://collectd.org</a><br>
Google+: <a href="http://collectd.org/+" target="_blank">http://collectd.org/+</a><br>
GitHub: <a href="https://github.com/collectd" target="_blank">https://github.com/collectd</a><br>
Twitter: <a href="http://twitter.com/collectd" target="_blank">http://twitter.com/collectd</a><br>
</font></span></blockquote></div><br>
</div></div></blockquote></div><br>
</div></div></blockquote></div><br>