liblo-devel Mailing List for LibLO

On Mon, Nov 30, 2009 at 2:15 AM, Dominic Sacré <dom...@gm...> wrote:
> Hi Stephen,
>
> On Sunday 29 of November 2009 06:03:58 Stephen Sinclair wrote:
>> I couldn't reproduce this problem and then finally after some work
>> found that I was able to crash a slightly older version in the way
>> you suggested, so I started bisecting and found that in fact the top
>> commit on the svn (fixing TCP closed connections, Oct. 2) seems to
>> have fixed this problem.  Were you testing against the latest build,
>> or against 0.26?
>>
>> If it was that serious a problem maybe it is time for a release, to
>> make that bug fix more available.
>
> The more serious bug that actually caused the segfault was in pyliblo,
> which failed to acquire Python's global interpreter lock when the error
> callback was called from the server thread. I've released a bugfix
> version (pyliblo 0.8.1) that should work just fine with liblo 0.26.
>
> Of course liblo shouldn't report an error in the first place, but I
> don't think this issue is important enough to warrant a new release.
>

Okay, thanks for the info.  I'm going to try and more thoroughly test
the TCP implementation in the weeks to come anyways.

cheers,
Steve

Hi Dominic,

Thanks for the update, I will give it a try asap.

Best, Dirk
> Hi Stephen,
>
> On Sunday 29 of November 2009 06:03:58 Stephen Sinclair wrote:
>   
>> I couldn't reproduce this problem and then finally after some work
>> found that I was able to crash a slightly older version in the way
>> you suggested, so I started bisecting and found that in fact the top
>> commit on the svn (fixing TCP closed connections, Oct. 2) seems to
>> have fixed this problem.  Were you testing against the latest build,
>> or against 0.26?
>>
>> If it was that serious a problem maybe it is time for a release, to
>> make that bug fix more available.
>>     
>
> The more serious bug that actually caused the segfault was in pyliblo, 
> which failed to acquire Python's global interpreter lock when the error 
> callback was called from the server thread. I've released a bugfix 
> version (pyliblo 0.8.1) that should work just fine with liblo 0.26.
>
> Of course liblo shouldn't report an error in the first place, but I 
> don't think this issue is important enough to warrant a new release.
>
>
> Cheers,
>
> Dominic
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
> trial. Simplify your report design, integration and deployment - and focus on 
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> liblo-devel mailing list
> lib...@li...
> https://lists.sourceforge.net/lists/listinfo/liblo-devel
>   

Hi Stephen,

I used liblo 0.26.

With the little fix suggested by Dominic it now runs fine. (I will have 
a try with top svn too).

Thanks for the answers!

Kind regards, Dirk
> Hi!
>
> On Wed, Nov 25, 2009 at 11:32 AM, Dirk Griffioen
> <dir...@ba...> wrote:
>   
>> Hi Stephen,
>>     
>>> Okay, thanks for the bug report!
>>>
>>>
>>>
>>>       
>> No prob.
>>     
>>>> Would it help if I try the fix and send you a patch?
>>>>
>>>>         
>>> Sure, I will also take a look at this and test it.  Thanks, please
>>> keep me updated if you try this and it works.  I'm happy someone is
>>> testing the TCP implementation. :)
>>>
>>>
>>>       
>> You'll find it attached. Sofar it works fine for me.
>>
>> I will let you know if I find something else.
>>     
>
> Thanks,
>
> I couldn't reproduce this problem and then finally after some work
> found that I was able to crash a slightly older version in the way you
> suggested, so I started bisecting and found that in fact the top
> commit on the svn (fixing TCP closed connections, Oct. 2) seems to
> have fixed this problem.  Were you testing against the latest build,
> or against 0.26?
>
> If it was that serious a problem maybe it is time for a release, to
> make that bug fix more available.
>
> cheers,
> Steve
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
> trial. Simplify your report design, integration and deployment - and focus on 
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> liblo-devel mailing list
> lib...@li...
> https://lists.sourceforge.net/lists/listinfo/liblo-devel
>   

Hi Stephen,

On Sunday 29 of November 2009 06:03:58 Stephen Sinclair wrote:
> I couldn't reproduce this problem and then finally after some work
> found that I was able to crash a slightly older version in the way
> you suggested, so I started bisecting and found that in fact the top
> commit on the svn (fixing TCP closed connections, Oct. 2) seems to
> have fixed this problem.  Were you testing against the latest build,
> or against 0.26?
>
> If it was that serious a problem maybe it is time for a release, to
> make that bug fix more available.

The more serious bug that actually caused the segfault was in pyliblo, 
which failed to acquire Python's global interpreter lock when the error 
callback was called from the server thread. I've released a bugfix 
version (pyliblo 0.8.1) that should work just fine with liblo 0.26.

Of course liblo shouldn't report an error in the first place, but I 
don't think this issue is important enough to warrant a new release.


Cheers,

Dominic

Hi!

On Wed, Nov 25, 2009 at 11:32 AM, Dirk Griffioen
<dir...@ba...> wrote:
> Hi Stephen,
>>
>> Okay, thanks for the bug report!
>>
>>
>>
>
> No prob.
>>>
>>> Would it help if I try the fix and send you a patch?
>>>
>>
>> Sure, I will also take a look at this and test it.  Thanks, please
>> keep me updated if you try this and it works.  I'm happy someone is
>> testing the TCP implementation. :)
>>
>>
>
> You'll find it attached. Sofar it works fine for me.
>
> I will let you know if I find something else.

Thanks,

I couldn't reproduce this problem and then finally after some work
found that I was able to crash a slightly older version in the way you
suggested, so I started bisecting and found that in fact the top
commit on the svn (fixing TCP closed connections, Oct. 2) seems to
have fixed this problem.  Were you testing against the latest build,
or against 0.26?

If it was that serious a problem maybe it is time for a release, to
make that bug fix more available.

cheers,
Steve

Hi Stephen,
> Okay, thanks for the bug report!
>
>
>   
No prob.
>>
>> Would it help if I try the fix and send you a patch?
>>     
>
> Sure, I will also take a look at this and test it.  Thanks, please
> keep me updated if you try this and it works.  I'm happy someone is
> testing the TCP implementation. :)
>
>   

You'll find it attached. Sofar it works fine for me.

I will let you know if I find something else.

Thanks for the great work on liblo!

Best, Dirk

On Wed, Nov 25, 2009 at 4:18 AM, Dirk Griffioen
<dir...@ba...> wrote:
> Hi,
>
> Yesterday I reported a small problem to Dominic Sacre, the pyliblo developer
> and he suggested I report it here as well.
>
> Liblo and pyliblo work great, but there is a small problem: if a client
> disconnects with an open tcp connection to the server, the server will
> segfault (!).
>
> I use liblo 0.26.
>
> (The code is python, but the segfault originates in liblo (see the valgrind
> output below))

Okay, thanks for the bug report!


> Attached 2 pieces of code.
> - start the consumer
> - start the producer
> - exit the producer (ctrl-c)
> * the consumer will segfault ...
>
> Dominic said the following (the second bug is related to pyliblo and of no
> interest here):
>
> Interesting... There seem to be two different bugs involved here. The first
> one is actually in liblo, which doesn't handle the closing of the connection
> correctly, and raises an error instead. Basically, liblo doesn't check for
> recv() returning zero, and then fails to parse the (nonexistent) received
> data.
> A possible solution is to change liblo-0.26/src/server.c, line 797 from
> "if (!data) {" to
> "if (!data || !size) {".
>
>
> Would it help if I try the fix and send you a patch?

Sure, I will also take a look at this and test it.  Thanks, please
keep me updated if you try this and it works.  I'm happy someone is
testing the TCP implementation. :)


Steve

Hi,

Yesterday I reported a small problem to Dominic Sacre, the pyliblo 
developer and he suggested I report it here as well.

Liblo and pyliblo work great, but there is a small problem: if a client 
disconnects with an open tcp connection to the server, the server will 
segfault (!).

I use liblo 0.26.

(The code is python, but the segfault originates in liblo (see the 
valgrind output below))

Attached 2 pieces of code.
- start the consumer
- start the producer
- exit the producer (ctrl-c)
* the consumer will segfault ...

Dominic said the following (the second bug is related to pyliblo and of 
no interest here):

Interesting... There seem to be two different bugs involved here. The 
first one is actually in liblo, which doesn't handle the closing of the 
connection correctly, and raises an error instead. Basically, liblo 
doesn't check for recv() returning zero, and then fails to parse the 
(nonexistent) received data.
A possible solution is to change liblo-0.26/src/server.c, line 797 from
"if (!data) {" to
"if (!data || !size) {".


Would it help if I try the fix and send you a patch?

Best, Dirk

Valgrind:

received message '/start' with arguments: ['hello']

(here the ctrl-c happens in the producer)

==30837==
==30837== Thread 2:
==30837== Invalid read of size 4
==30837==    at 0x80610F6: PyObject_Call (in /usr/bin/python2.6)
==30837==    by 0x4654568: __pyx_f_5liblo__err_handler (liblo.c:2441)
==30837==    by 0x40392B5: lo_throw (server.c:1396)
==30837==    by 0x403A3ED: lo_server_dispatch_data (server.c:860)
==30837==    by 0x403ACBD: lo_server_recv (server.c:800)
==30837==    by 0x403AD36: lo_server_recv_noblock (server.c:707)
==30837==    by 0x403BBD1: thread_func (server_thread.c:168)
==30837==    by 0x404E4FE: start_thread (in /lib/tls/i686/cmov/libpthread-2.9.so)
==30837==    by 0x418A49D: clone (in /lib/tls/i686/cmov/libc-2.9.so)
==30837==  Address 0xc is not stack'd, malloc'd or (recently) free'd
==30837==
==30837== Process terminating with default action of signal 11 (SIGSEGV)
==30837==  Access not within mapped region at address 0xC
==30837==    at 0x80610F6: PyObject_Call (in /usr/bin/python2.6)
==30837==    by 0x4654568: __pyx_f_5liblo__err_handler (liblo.c:2441)
==30837==    by 0x40392B5: lo_throw (server.c:1396)
==30837==    by 0x403A3ED: lo_server_dispatch_data (server.c:860)
==30837==    by 0x403ACBD: lo_server_recv (server.c:800)
==30837==    by 0x403AD36: lo_server_recv_noblock (server.c:707)
==30837==    by 0x403BBD1: thread_func (server_thread.c:168)
==30837==    by 0x404E4FE: start_thread (in /lib/tls/i686/cmov/libpthread-2.9.so)
==30837==    by 0x418A49D: clone (in /lib/tls/i686/cmov/libc-2.9.so)
==30837==  If you believe this happened as a result of a stack overflow in your
==30837==  program's main thread (unlikely but possible), you can try to increase
==30837==  the size of the main thread stack using the --main-stacksize= flag.
==30837==  The main thread stack size used in this run was 8388608.

The IP header has two fields for source and destination port.

In most cases you don't specify a source port, just a destination.  In  
this case the operating system automatically assigns one to your  
connection, typically with a "high number" (> 15000).  When you  
traverse a NAT device, the source port gets changed and the NAT router  
maintains a table of which address to forward packets based on their  
source port.

When your server receives a packet, it simply swaps the source and  
destination ports to make a packet that will travel in the other  
direction.  This packet will arrive at the client regardless of how  
many NAT devices are in between.  This works for both UDP and TCP.   
Services like Skype (and also Ross Bencinas' oscgroups server) use a  
special technique called NAT hole punching that uses an intermediary  
to relay what the NAT holes are between to clients.  Unless you have  
an ultra-restrictive firewall this will pretty much always work.

When you use the function recv() you can't make a reply packet because  
the source port isn't provided.  The function recvfrom() is identical  
but also fills a structure (passed by reference) with extra  
information about the source of the packet, including the address and  
source port.  Meanwhile the client will need to find out what port the  
OS assigned to them, and start listening for a reply packet on that  
port.

Many OSC libraries have "forgotten" about bidirectional UDP, but this  
is actually the basis of how the internet works (DNS would never work  
without it, for example).  Anyways I don't know how its done in liblo,  
but hopefully this helps....



Andy.


On Nov 12, 2009, at 12:07 PM, IOhannes m zmoelnig wrote:

> hi all.
>
> sorry if this comes through twice; i keep using the wrong address for
> sending emails...
>
> for my midi-over-osc project i have to establish a bidirectional
> communication over WAN.
>
> i wanted to use liblo for this, but have trouble finding the relevant
> information in the API-documentation.
>
> so my question is:
> - how do tell my server to send back a message to a connected client?
> it would be sufficient to broadcast to all connected clients (only one
> is supposed to be connected)
>
> i understand the problem when using UDP, but really i am using TCP/IP.
> the client will be behind a natted firewall, without the possibility  
> to
> open any ports.
>
> there seems to be a way to get the source of an incoming message with
> "lo_message_get_source()", but how on earth do i get lo_messages out  
> of
> the server?



---

Andy W. Schmeder
andy [at] cnmat.berkeley.edu

Programmer/Analyst II
Research Group
Center for New Music and Audio Technologies
University of California at Berkeley
http://cnmat.berkeley.edu

On Thu, Nov 12, 2009 at 3:49 PM, Andy W. Schmeder
<an...@cn...> wrote:
> The IP header has two fields for source and destination port.
>
> In most cases you don't specify a source port, just a destination.  In
> this case the operating system automatically assigns one to your
> connection, typically with a "high number" (> 15000).  When you
> traverse a NAT device, the source port gets changed and the NAT router
> maintains a table of which address to forward packets based on their
> source port.
>
> When your server receives a packet, it simply swaps the source and
> destination ports to make a packet that will travel in the other
> direction.  This packet will arrive at the client regardless of how
> many NAT devices are in between.  This works for both UDP and TCP.
> Services like Skype (and also Ross Bencinas' oscgroups server) use a
> special technique called NAT hole punching that uses an intermediary
> to relay what the NAT holes are between to clients.  Unless you have
> an ultra-restrictive firewall this will pretty much always work.
>
> When you use the function recv() you can't make a reply packet because
> the source port isn't provided.  The function recvfrom() is identical
> but also fills a structure (passed by reference) with extra
> information about the source of the packet, including the address and
> source port.  Meanwhile the client will need to find out what port the
> OS assigned to them, and start listening for a reply packet on that
> port.
>
> Many OSC libraries have "forgotten" about bidirectional UDP, but this
> is actually the basis of how the internet works (DNS would never work
> without it, for example).  Anyways I don't know how its done in liblo,
> but hopefully this helps....


liblo uses recvfrom() for UDP, and recv() for TCP.  I'll have to test
sometime to see if it's possible to "reply" on a TCP connection...

It should perhaps be added as a unit test, now that TCP is better supported.

Steve

On Thu, Nov 12, 2009 at 3:07 PM, IOhannes m zmoelnig <zmo...@ie...> wrote:
> hi all.
>
> sorry if this comes through twice; i keep using the wrong address for
> sending emails...
>
> for my midi-over-osc project i have to establish a bidirectional
> communication over WAN.
>
> i wanted to use liblo for this, but have trouble finding the relevant
> information in the API-documentation.

I'm afraid it's not directly supported.  The problem is liblo has two
different data structures for sending and receiving (lo_address and
lo_server).  When you send a message it creates a socket, but there's
currently no way to tell a server to use the same socket as a given
lo_address.

I just looked up the last time someone asked me about this (off-list),
we came up with a suggestion that one should be able to specify this
relationship, like:

s = lo_server_new_with_proto(...);
a = lo_address_new_with_proto(...);
lo_server_set_address(s, a);

But this hasn't been implemented.. I'm trying to remember now why, but
for some reason we determined that lo_send_from() wasn't a solution to
this problem.

In any case, as things stand you have to create two uni-directional
TCP connections.

Note that I've never received feedback from anyone telling me whether
the TCP implementation works well or not, so beyond my limited testing
I'd love to see any reports of bad behaviour.


> so my question is:
> - how do tell my server to send back a message to a connected client?
> it would be sufficient to broadcast to all connected clients (only one
> is supposed to be connected)
>
> i understand the problem when using UDP, but really i am using TCP/IP.
> the client will be behind a natted firewall, without the possibility to
> open any ports.

Okay, that's a pretty good reason to need this feature...


> there seems to be a way to get the source of an incoming message with
> "lo_message_get_source()", but how on earth do i get lo_messages out of
> the server?

I'm not sure I understand this question.. the server calls a callback
when it gets a message, you don't "get messages out" of a server.


> PS: the project is due tomorrow, so i'm a bit stressed  :-)

That sucks!  Good luck, you might have to hack it for now!  (For
instance, modify the liblo code internally for a temporary fix.. like,
add a function to pass around the socket fd.)

By the way, speaking of hacks, I should mention that liblo 0.26
supports parsing and dispatching messages from memory, as well as
serializing messages to memory, so you could actually use this
functionality to send and receive the data using your own networking
code.

Look at lo_message_serialise() and lo_server_dispatch_data().


> PPS: i'm not on the liblo-devel list as i fail to find the subscribe
> info on sourceforge. please CC me.
> i _am_ subscribed on osc_dev@create)

It's linked on the liblo homepage:

https://lists.sourceforge.net/lists/listinfo/liblo-devel


Steve

On Thu, Nov 12, 2009 at 8:27 AM, Albert Graef <Dr....@t-...> wrote:

> But maybe a routine could be added to register a method with a server
> (thread) which reports the beginning and end of each bundle? I don't
> know whether that's feasible, but this would maintain backward
> compatibility with previous liblo versions and would allow an
> application to reconstruct the (possibly recursive) bundle structure if
> it needs to.
>

I think this is a great idea!


Some explanations for Stephen:

Until today loosing the bundle structure was not a big deal, but now that I
work with MIDI over OSC packets, I want to make everything possible to keep
the messages in the right order, and with the same time semantics.  The
latter is not only true for MIDI but also for any OSC application that
defines a timetag.

It strikes me that there are a lot of requests for features related to

forwarding messages from one place to another.  Why not just send the

messages to the right destination to begin with? ;-)


I think people would use a middle-man application to route their message
from their devices to their destinations, because it is easier to manage.
 You can make changes, try other input devices, swap destinations, without
having to spend time reconfiguring things.  It also makes the setup
configuration more stable since it tends to change not less often than in a
device to device configuration.

Stephen Sinclair wrote:
> The only way I can think to do it would be to queue up messages that
> have the same timestamp and then send them as a bundle if a new
> timestamp is encountered or a timer expires.  That doesn't sound great
> though, since it would certainly add latency.

And it wouldn't preserve the structure either if one or more
(sub)bundles are sent at the same time.

But maybe a routine could be added to register a method with a server
(thread) which reports the beginning and end of each bundle? I don't
know whether that's feasible, but this would maintain backward
compatibility with previous liblo versions and would allow an
application to reconstruct the (possibly recursive) bundle structure if
it needs to.

Albert

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikinformatik.uni-mainz.de/ag

On Mon, Nov 9, 2009 at 6:44 PM, Camille Troillard <ca...@os...> wrote:
> Dear list,
> I would like to have some advice on the way to handle bundle messages.
> My application receives a bundle and must re-route it to another
> destination, but must conserve the original bundle structure of the received
> data.
> I understand liblo calls my callbacks by decoding the bundle, thus calling
> the callback with a message argument for each message in the bundle.  I was
> wondering how can I know if these messages are part of a bundle.  There is
> the timetag for that, but unfortunately, it does not tell me if there are
> messages remaining in the bundle.
> It is not a big deal to reconstruct the bundle in order to send it to
> another destination, but I would need to know at least if the received
> messages are part of a bundle and if yes, if the current message is the last
> in the bundle (which would trigger the routing action).


I can't really think of a way to do that within the current API.
(Providing a bundle/nobundle flag is especially complicated by the
fact that bundles can be nested, if you think about it.)
Currently liblo parses a bundle one message at a time, so it doesn't
even determine that a message is the last one of a bundle until it
gets to the end.

It strikes me that there are a lot of requests for features related to
forwarding messages from one place to another.  Why not just send the
messages to the right destination to begin with? ;-)

(I jest.. I'm sure there's a good reason.)

More seriously though, I'm not sure it's good practice to depend on
bundle integrity being preserved.  Usually the receiver doesn't care,
except to check the timestamp.  Being an OSC "router" seems to add a
few extra requirements on functionality.

The only way I can think to do it would be to queue up messages that
have the same timestamp and then send them as a bundle if a new
timestamp is encountered or a timer expires.  That doesn't sound great
though, since it would certainly add latency.  I guess the ideal
solution would be to make available the 'remain' variable from
lo_server_dispatch_data to the handlers, which says how much data
remains to be processed in the bundle memory.  This doesn't sound like
a very eloquent solution to me, though.


Steve

Dear list,

I would like to have some advice on the way to handle bundle messages.
My application receives a bundle and must re-route it to another
destination, but must conserve the original bundle structure of the received
data.

I understand liblo calls my callbacks by decoding the bundle, thus calling
the callback with a message argument for each message in the bundle.  I was
wondering how can I know if these messages are part of a bundle.  There is
the timetag for that, but unfortunately, it does not tell me if there are
messages remaining in the bundle.

It is not a big deal to reconstruct the bundle in order to send it to
another destination, but I would need to know at least if the received
messages are part of a bundle and if yes, if the current message is the last
in the bundle (which would trigger the routing action).


Best Regards,
Camille