[collectd] lua plugin

[Florian Forster]

Hi Julien,

On Mon, Nov 22, 2010 at 11:03:14PM +0100, Schmurfy wrote:
> this discussion started with octo as private messages on github but he
> wisely asked me to post it on the mailing list so anyone can take part
> in it so here I am !

thanks for taking this here :) And sorry for not looking closer at your
branch yet. I certainly hope to make some time later this week.

> […] the problem comes from the fact that the official lua distribution
> is a static library which cannot be embedded in the shared library
> built for collectd (I tried but failed miserably)

The problem is that in order to be able to link static libraries into
shared objects (dynamic libraries), the static library has be be
compiled with some special linker settings (-fPIC). This is an annoying
problem, because many people, including many maintainers of packages of
libraries, don't know this and have a hard time reproducing this bug,
because it doesn't show up on a x86 system.

About the two options:

> The first one is to link to a shared library release of lua […]
> The second one is to include the lua sources themselves into the collectd
> plugin, […]

In my opinion, the second option is out of question. We've done this
with liboping and libiptc in the past and nothing good came out of this.
You have all the problems you have with shared objects [*] but in an
even more severe variant [**]. We still include libltdl because that's
the way libtool works, but with the same result [0].

I'd go for a variant of the first option: Simply do a normal check for
the library using autoconf and link with libtool. This will
automatically get you the shared library if it is available and fall
back to the static library if necessary. Being the default behavior this
is what most people, including package maintainers, expect and know how
to deal with.

I'd like a comment on the (apparently common) "other languages should be
statically linked" misconception: Shared objects have a system to cope
with incompatible binary versions. A dynamic libraries implementing a
scripting language is nothing special and certainly no exception. If the
language itself changes, this is either transparent to the application
(annoying for the user, but not our problem) or it's an API change, for
which numerous detection mechanisms and solutions exist. All the reasons
for dynamic linking apply to scripting languages and DSLs just as well.

In the worst case, the API provided to scripts depends on the scripting
language's version. If so, that's how it is. Period.

To back my claims up by facts:

 * The Python plugin supports Python 2 *and* Python 3, which has a
   substantially changed concept of what a "string" is. Changes to a
   scripting languages hardly ever get any more sever than this.
 * The Java plugin has been tested with Java 1.4 through 1.6.
 * The Perl plugin works with Perl 5.8 and 5.10. I'm not sure about
   version 5.6, but I *think* that worked, too.

If we can handle these complex and mainly "stand alone" languages and
they manage to maintain mechanisms to work with changes made to them,
I'm confident a language *intended to be integrated* will manage to keep
up, too.

> I was also thinking about something else but linked to lua: have
> anyone thought about using a higher level language to configure
> filters ?

That does sound like an interesting though. It'll be a lot of work to
implement though. Be sure to toss around some ideas on the list before
implementing anything.

Regards,
—octo

[*]  In particular: You need to rebuild collectd and other binaries the
     library has been linked in when there's a security problem in the
     library. 
[**] You cannot determine that collectd uses liblua using the package's
     build dependencies, making it extra hard to realize there's a need
     to rebuild collectd.
[0]  <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=559801>
-- 
Florian octo Forster
Hacker in training
GnuPG: 0x0C705A15
http://octo.it/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: Digital signature
URL: <http://mailman.verplant.org/pipermail/collectd/attachments/20101123/288a3b37/attachment.pgp>