liblo-devel Mailing List for LibLO

On Thu, May 22, 2008 at 4:17 PM, Michael Taht <m...@te...> wrote:
> Stephen Sinclair wrote:
>
>> I know what you mean, it is a common issue.  You have to think of
>> multicast as being a common bus that everyone sees.  If you listen in
>> on it, you have to deal with listening to your own messages.
>> Designing the address namespace carefully can help with this a great
>> deal.
>>
> Not at 6.4MB/sec. I guess the source of my confusion is that
> I thought that if you held a multicast udp port open, you didn't hear
> yourself.

I'm pretty sure that's not the case... you hear anything that goes out
on a multicast port.  At least from my experience that seems to be
true.  Mostly I've played with this using Max/MSP and Pd.  (I
implemented the multicast stuff in Pd though, so that's not a very
good test case.  In Max/MSP, I have tested with the mxj.udp.send and
mxj.udp.recv objects.)


> I will look deeper at the code... multicast is new territory.

I don't have _that_ much experience with it, but it seemed not too
difficult to implement so I did it.  However, there could easily be
known issues I haven't considered.

> No. I meant that the controller -> ardour messages in this case are
> infrequent and small (and kind of need acks), and the ack could be
> either on the multicast port or on the unicast port.
>
> Acks are a problem with OSC.

Hm, well the concept of acking is kind of a transport issue, i.e.,
being able to verify that the message you sent was received.   OSC
takes the approach that transport issues should be entirely left to
the transport layer, which is purposely not specified... so, in other
words I believe that Matt Wright assumed that people needing ack-style
functionality would use TCP, which is reliable, while stream-like
messages not needing the reliability of TCP would make use of UDP.  In
practice most people use UDP, but I think that's mainly because TCP
hasn't been widely implemented.

Of course that doesn't stop you from implementing an OSC-level /ack
message.  I know at least one person my lab has done something like
that.


>>> Similar symmetry problem with setting the RTT.
>>>
>>
>> Isn't the RTT a measurement?
>>
> Sorry, meant "TTL". I flatlined a couple wireless bridges on my
> neighbor's net even though I had a router in the way...
>
> Fixed at the router now.

Ah, I thought the default TTL was 1, so that messages wouldn't leave
the subnet.  Maybe that's not the case.  There should probably be some
code in lo_address_new() that detects multicast addresses and sets the
TTL to a safe level.

Whoops, I just looked it up:
http://secfr.nerim.net/docs/fingerprint/en/ttl_default.html

Looks like the default TTL is 64, so this is definitely an issue.
I'll try to fix it a.s.a.p.


>>  It's probably
>> something that should go in the testlo program.
>> Usually UDP (and OSC) is intended for short messages with a single
>> purpose, but of course it's possible to use it in the way you're
>> talking about and it should support this.  But perhaps UDP is not the
>> best protocol for your purposes..  though liblo's TCP implementation
>> still needs a little work, you might want to think about trying it
>> instead.  What do you think?  Currently I think it closes the port
>> after one message, which isn't too useful, so the code would need some
>> attention.
>>
> TCP's ack problem/latency bugs me. SCTP?

I understand latency, but could you elaborate on "TCP's ack problem"?


> Keeping enough state around to implement a fast protocol around UDP is
> painful though.

I agree, these things are best left to the transport layer.



> I will get back on it shortly, I'm trying to get the alphatrack to speak
> the protocol....

Hadn't heard of these controllers, thanks.


Steve

Stephen Sinclair wrote:
> Hi,
>
> Sorry for the slow reply, I've been away from email for a few days.
>   
Thanks for getting back to me.
> On Fri, May 16, 2008 at 4:16 AM, Michael Taht <m...@te...> wrote:
>   
>> I was delighted to see liblo in svn support multicast and also add the
>> lo_bundle_free_messages call.
>>     
>
> I'm very glad to hear it's come in useful for someone!
>
>   
>> 0) lo_server_get_url returns the hostname of the system in the case of a
>> multicast address. It should return the multicast address, as it is
>> unlikely to be in dns, and incorrect to return the hostname.
>>     
>
> That's probably a bug, I'll look at it.
> I've filled in an item on the sourceforge bug tracker.
>
>
>   
>> It's actually kind of unclear to me how multicast should work, anyway,
>> as I would like to both receive multicast messages sent by controllers,
>> and send multicast messages that include lots of state information, and
>> don't wish to "listen to myself" having sent them.
>>     
>
> I know what you mean, it is a common issue.  You have to think of
> multicast as being a common bus that everyone sees.  If you listen in
> on it, you have to deal with listening to your own messages.
> Designing the address namespace carefully can help with this a great
> deal.
>   
Not at 6.4MB/sec. I guess the source of my confusion is that
I thought that if you held a multicast udp port open, you didn't hear
yourself.

I will look deeper at the code... multicast is new territory.
> In an application I am working on, we use multicast only to publicly
> announce which unicast ports we're listening on, and then all further
> communication is performed peer-to-peer.
>
>
>   
That is a helpful idea.
>> Alternatively a sane approach would be to accept messages on a unicast
>> addr/port and dispense them on the multicast addr/port and vice versa.
>>     
>
> Hm, I'm not sure I fully understand you here.
> You mean forward received messages to a multicast port?  I'm not sure
> I see the point.
>   
No. I meant that the controller -> ardour messages in this case are
infrequent and small (and kind of need acks), and the ack could be
either on the multicast port or on the unicast port.

Acks are a problem with OSC.
>
>   
>> 1) It would be useful to have a symmetric
>>
>> lo_address lo_server_get_address(lo_server s);
>>
>> for use in the lo_send_bundle_from and lo_send_message_from functions.
>> (this to me kind of distinguishes sending a multicast from receiving
>> one. If you send it from the existing server, it should do the right
>> thing, sending it from the already registered socket)
>>     
>
> Perhaps I'm misunderstanding, but lo_send_message_from() takes a
> lo_server argument to specify that messages should be sent from that
> server's socket.  Is that not what you mean?
>
>   
I don't frankly remember what I was getting to here, I think
it had something to do with wanting to ensure I didn't hear
myself. Or wanting the lo_server's lo_address to stick into poll
or something like that.
>   
>> Similar symmetry problem with setting the RTT.
>>     
>
> Isn't the RTT a measurement?
>   
Sorry, meant "TTL". I flatlined a couple wireless bridges on my
neighbor's net even though I had a router in the way...

Fixed at the router now.

>> 2) It's not obvious that you can't call lo_bundle_free_messages(), and
>> then lo_bundle_free().  (it became painfully obvious one segvio later)
>>     
>
> The documentation says, "Frees the memory taken by a bundle object and
> messages in the bundle."  Can this be made clearer?
>   
Nope. But I could be made smarter!
> I'm not sure there'd be any point having a function that frees the
> messages without free the bundle's memory.
>
>
>   
No, it's a good function, I'm happy.
>> 3) I am pushing liblo to new extremes, trying to send one major bundle
>> consisting of potentially hundreds of messages, (one per track), each
>> containing 10+ arguments, every 10ms. From my minimal glance at the
>> liblo code, it looks like, for 64 tracks, that looks to be a lot of
>> calls to realloc per second.
>>
>> Perhaps supplying some sort of hint to lo_bundle_new or lo_message_new
>> would be useful. The actual number of items (and types) in the messages
>> and bundle can be generally determined before making the jump to OSC.
>>
>> lo_message_new_with_hint(,"iiifffsss")
>> lo_bundle_new_with_hint(,number_of_messages);
>>
>> The structure of a message is generally known at compile time however,
>> perhaps parsing out that string would be a bit much.
>> LO_MSG_HINT(num_of_floats,num_of_ints,num_of_strings,etc), maybe.
>>     
>
> That's interesting.  I agree that some form of hinting might be
> appropriate, but I'm not sure specifying the number of messages is the
> right way, since those messages can then by any size at all.  
Which was why I proposed two calls (but blew the second one's definition,
because I wasn't sure of lo_bundle's internals, I figured it was doing
an inplace copy, hadn't thought about it...)

One to get the probable message size, the other to take
the number of those kinds of messages and that message...

lo_bundle_new_with_hint(,lo_message example, number_of_messages);

A nastier version might do varargs

You grokked that my intent was to avoid major allocations
in a RT process. One day these could be pre-allocated
> The
> realloc algorithm doubles the size each time rather than growing by a
> constant, so hopefully that helps to reduce the number of needed
> calls.  I think it would be nice, in the long-run, to be able to
> allocate a block of memory yourself and tell it to use that instead of
> allocating its own memory.
>
> (Particularly this would be useful for real-time code that can't do
> memory allocation at all.)
>
>   
Si.
>   
>> 4) I know that somewhere along the line I'll hit the max udp packet
>> size... I figure I should get that error back on the send call?
>>     
>
> Please feel free to test this and tell me the result. 
Segvio'd somewhere while testing 128 tracks but I wasn't
checking the return value at the time.

I will find a test case.
>  It's probably
> something that should go in the testlo program.
> Usually UDP (and OSC) is intended for short messages with a single
> purpose, but of course it's possible to use it in the way you're
> talking about and it should support this.  But perhaps UDP is not the
> best protocol for your purposes..  though liblo's TCP implementation
> still needs a little work, you might want to think about trying it
> instead.  What do you think?  Currently I think it closes the port
> after one message, which isn't too useful, so the code would need some
> attention.
>   
TCP's ack problem/latency bugs me. SCTP?

Keeping enough state around to implement a fast protocol around UDP is
painful though.
>   
>> Anyway, right now I have ardour happily sending 220 rather large and
>> fragmented packets a second (one group multicast, one unicast) and am
>> working on a receiver.... current ardour patch is up at:
>>
>> http://tracker.ardour.org/view.php?id=2239
>>     
>
> That's totally awesome, I'm so glad it works well.
>
>   
I will get back on it shortly, I'm trying to get the alphatrack to speak
the protocol....
> thanks for the feedback!
>
> Steve
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> liblo-devel mailing list
> lib...@li...
> https://lists.sourceforge.net/lists/listinfo/liblo-devel
>   

Hi,

Sorry for the slow reply, I've been away from email for a few days.

On Fri, May 16, 2008 at 4:16 AM, Michael Taht <m...@te...> wrote:
> I was delighted to see liblo in svn support multicast and also add the
> lo_bundle_free_messages call.

I'm very glad to hear it's come in useful for someone!

> 0) lo_server_get_url returns the hostname of the system in the case of a
> multicast address. It should return the multicast address, as it is
> unlikely to be in dns, and incorrect to return the hostname.

That's probably a bug, I'll look at it.
I've filled in an item on the sourceforge bug tracker.


> It's actually kind of unclear to me how multicast should work, anyway,
> as I would like to both receive multicast messages sent by controllers,
> and send multicast messages that include lots of state information, and
> don't wish to "listen to myself" having sent them.

I know what you mean, it is a common issue.  You have to think of
multicast as being a common bus that everyone sees.  If you listen in
on it, you have to deal with listening to your own messages.
Designing the address namespace carefully can help with this a great
deal.

In an application I am working on, we use multicast only to publicly
announce which unicast ports we're listening on, and then all further
communication is performed peer-to-peer.


> Alternatively a sane approach would be to accept messages on a unicast
> addr/port and dispense them on the multicast addr/port and vice versa.

Hm, I'm not sure I fully understand you here.
You mean forward received messages to a multicast port?  I'm not sure
I see the point.


> 1) It would be useful to have a symmetric
>
> lo_address lo_server_get_address(lo_server s);
>
> for use in the lo_send_bundle_from and lo_send_message_from functions.
> (this to me kind of distinguishes sending a multicast from receiving
> one. If you send it from the existing server, it should do the right
> thing, sending it from the already registered socket)

Perhaps I'm misunderstanding, but lo_send_message_from() takes a
lo_server argument to specify that messages should be sent from that
server's socket.  Is that not what you mean?


> Similar symmetry problem with setting the RTT.

Isn't the RTT a measurement?


> 2) It's not obvious that you can't call lo_bundle_free_messages(), and
> then lo_bundle_free().  (it became painfully obvious one segvio later)

The documentation says, "Frees the memory taken by a bundle object and
messages in the bundle."  Can this be made clearer?

I'm not sure there'd be any point having a function that frees the
messages without free the bundle's memory.


> 3) I am pushing liblo to new extremes, trying to send one major bundle
> consisting of potentially hundreds of messages, (one per track), each
> containing 10+ arguments, every 10ms. From my minimal glance at the
> liblo code, it looks like, for 64 tracks, that looks to be a lot of
> calls to realloc per second.
>
> Perhaps supplying some sort of hint to lo_bundle_new or lo_message_new
> would be useful. The actual number of items (and types) in the messages
> and bundle can be generally determined before making the jump to OSC.
>
> lo_message_new_with_hint(,"iiifffsss")
> lo_bundle_new_with_hint(,number_of_messages);
>
> The structure of a message is generally known at compile time however,
> perhaps parsing out that string would be a bit much.
> LO_MSG_HINT(num_of_floats,num_of_ints,num_of_strings,etc), maybe.

That's interesting.  I agree that some form of hinting might be
appropriate, but I'm not sure specifying the number of messages is the
right way, since those messages can then by any size at all.  The
realloc algorithm doubles the size each time rather than growing by a
constant, so hopefully that helps to reduce the number of needed
calls.  I think it would be nice, in the long-run, to be able to
allocate a block of memory yourself and tell it to use that instead of
allocating its own memory.

(Particularly this would be useful for real-time code that can't do
memory allocation at all.)


> 4) I know that somewhere along the line I'll hit the max udp packet
> size... I figure I should get that error back on the send call?

Please feel free to test this and tell me the result.  It's probably
something that should go in the testlo program.
Usually UDP (and OSC) is intended for short messages with a single
purpose, but of course it's possible to use it in the way you're
talking about and it should support this.  But perhaps UDP is not the
best protocol for your purposes..  though liblo's TCP implementation
still needs a little work, you might want to think about trying it
instead.  What do you think?  Currently I think it closes the port
after one message, which isn't too useful, so the code would need some
attention.

> Anyway, right now I have ardour happily sending 220 rather large and
> fragmented packets a second (one group multicast, one unicast) and am
> working on a receiver.... current ardour patch is up at:
>
> http://tracker.ardour.org/view.php?id=2239

That's totally awesome, I'm so glad it works well.

thanks for the feedback!

Steve

I was delighted to see liblo in svn support multicast and also add the
lo_bundle_free_messages call.

I am trying to improve ardour's OSC support to be able to support
complex surfaces like the tranzport and alphatrack.

So I started playing with liblo from svn today.

some comments:

0) lo_server_get_url returns the hostname of the system in the case of a
multicast address. It should return the multicast address, as it is
unlikely to be in dns, and incorrect to return the hostname.

Maybe.

It's actually kind of unclear to me how multicast should work, anyway,
as I would like to both receive multicast messages sent by controllers,
and send multicast messages that include lots of state information, and
don't wish to "listen to myself" having sent them.

Alternatively a sane approach would be to accept messages on a unicast
addr/port and dispense them on the multicast addr/port and vice versa.

1) It would be useful to have a symmetric

lo_address lo_server_get_address(lo_server s);

for use in the lo_send_bundle_from and lo_send_message_from functions.
(this to me kind of distinguishes sending a multicast from receiving
one. If you send it from the existing server, it should do the right
thing, sending it from the already registered socket)

Similar symmetry problem with setting the RTT.


2) It's not obvious that you can't call lo_bundle_free_messages(), and
then lo_bundle_free().  (it became painfully obvious one segvio later)

3) I am pushing liblo to new extremes, trying to send one major bundle
consisting of potentially hundreds of messages, (one per track), each
containing 10+ arguments, every 10ms. From my minimal glance at the
liblo code, it looks like, for 64 tracks, that looks to be a lot of
calls to realloc per second.

Perhaps supplying some sort of hint to lo_bundle_new or lo_message_new
would be useful. The actual number of items (and types) in the messages
and bundle can be generally determined before making the jump to OSC.

lo_message_new_with_hint(,"iiifffsss")
lo_bundle_new_with_hint(,number_of_messages);

The structure of a message is generally known at compile time however,
perhaps parsing out that string would be a bit much.
LO_MSG_HINT(num_of_floats,num_of_ints,num_of_strings,etc), maybe.

4) I know that somewhere along the line I'll hit the max udp packet
size... I figure I should get that error back on the send call?

...

Anyway, right now I have ardour happily sending 220 rather large and
fragmented packets a second (one group multicast, one unicast) and am
working on a receiver.... current ardour patch is up at:

http://tracker.ardour.org/view.php?id=2239

On Mon, May 5, 2008 at 9:31 PM, Stephen Sinclair <rad...@gm...>
wrote:

> Hi Camille,
>
> On Sat, May 3, 2008 at 12:34 PM, Camille Troillard
> <ca...@os...> wrote:
> > If the answer is no, do you folks have suggestions about how to delete
> all
> > methods in a server without stopping and restarting the server?
> > My goal is to switch a context of methods with the fewest overhead
> possible.
>
> I believe that right now, manipulating methods is not actually thread
> safe, in that there are no locks around the operation.
> The quickest safe way to delete methods, I think, is to do it inside a
> message handler, therefore in the same thread that is checking for new
> messages.
>
> Is that possible for you?


Yes, but the only plausible way I see of doing this is to send a OSC message
to myself, which is not bad after all.

Thanks for the answer!

Cam

On Sat, May 3, 2008 at 12:30 PM, Camille Troillard
<ca...@os...> wrote:
> Hi,
>
> I was searching a way to delete all method handlers without stopping and
> restarting a server thread.
> I tried to do this:
>
> lo_server_thread_del_method(_st, "*", NULL);
>
> and the debugger stopped in lo_pattern match (,) because of a null pointer
> exception.
>
> the cure has been to replace the code that goes:
> if ((it->path == path) ||
>
>    (path && it->path && !strcmp(path, it->path)) ||
>    (pattern && lo_pattern_match(it->path, path))) {
>  by
>
> if ((it->path == path) ||
>    (path && it->path && !strcmp(path, it->path)) ||
>    (pattern && it->path && lo_pattern_match(it->path, path))) {
>
> My suggestion would be to actually put this test in lo_pattern_match (,).
> This fix follows the test pattern found on line 2.

Thanks, I committed this.
The only other case where lo_pattern_match is called, it->path is
already checked, so I'll leave it for now.


Steve

Hi Camille,

On Sat, May 3, 2008 at 12:34 PM, Camille Troillard
<ca...@os...> wrote:
> Hi,
>
> I would like to know if it is thread safe to call
> lo_server_thread_del_method.
> I guess from the name of the method that I can safely call it and delete a
> method while liblo is processing incoming OSC, right?
>
> If the answer is no, do you folks have suggestions about how to delete all
> methods in a server without stopping and restarting the server?
> My goal is to switch a context of methods with the fewest overhead possible.

I believe that right now, manipulating methods is not actually thread
safe, in that there are no locks around the operation.
The quickest safe way to delete methods, I think, is to do it inside a
message handler, therefore in the same thread that is checking for new
messages.

Is that possible for you?
I can see some situations where external events want to add and delete
methods, but it's probably not ideal for you to have external threads
wanting to manipulate the server_thread.  Any one else have thoughts
on this?

Steve

Hi,
I would like to know if it is thread safe to call
lo_server_thread_del_method.
I guess from the name of the method that I can safely call it and delete a
method while liblo is processing incoming OSC, right?

If the answer is no, do you folks have suggestions about how to delete all
methods in a server without stopping and restarting the server?
My goal is to switch a context of methods with the fewest overhead possible.

Thank you.

Best Regards,
Camille

Hi,
I was searching a way to delete all method handlers without stopping and
restarting a server thread.
I tried to do this:

lo_server_thread_del_method(_st, "*", NULL);

and the debugger stopped in lo_pattern match (,) because of a null pointer
exception.

the cure has been to replace the code that goes:
if ((it->path == path) ||
   (path && it->path && !strcmp(path, it->path)) ||
   (pattern && lo_pattern_match(it->path, path))) {by
if ((it->path == path) ||
   (path && it->path && !strcmp(path, it->path)) ||
   (pattern && it->path && lo_pattern_match(it->path, path))) {

My suggestion would be to actually put this test in lo_pattern_match (,).
This fix follows the test pattern found on line 2.

Sorry, I could have sent a diff, but for just one small bunch of characters,
I thought it was not worth it.


Best,
Cam

Thanks Steve!

That patch seems to do the trick. I can now have wildcards in the 
handlers... cool :)

-Mike


Stephen Sinclair wrote:
> Hi Mike, welcome to the list.. :)
>
>
> On Wed, Apr 30, 2008 at 4:40 PM, Steve Harris
> <S.W...@ec...> wrote:
>   
>> It's tricky because OSC clients are allowed to send paths with
>>  wildcards, like:
>>     /channel/*/volume 1.0
>>  and you obviously can't deliver one wildcard into another.
>>     
>
> I think one could perhaps argue that if two wildcards hit each other
> there is always a match.  (Anything matches anything..)  However I
> guess this is more complicated for non-* patterns, like letter ranges.
>
>
>   
>>  On 30 Apr 2008, at 21:20, Mike Wozniewski wrote:
>>
>>  > Hi,
>>  >
>>  > I'm wondering if it's possible to create a handler that will be called
>>  > for a 'partial path' of a message?
>>  >
>>  > That is, when /foo/bar, or /foo/sheefa, or /foo/anything is received,
>>  > one unique method is called. This could almost be thought of as a
>>  > handler with wildcards, where you would declare it something like
>>  > this:
>>  >
>>  > lo_server_thread_add_method(st, "/foo/*", NULL, foo_handler, NULL);
>>     
>
> Maybe try the following patch, it's a pretty minimal change to the
> LibLo sources, re-using the pattern matching code that's already
> there.  As Steve mentions above, there are likely cases where this
> doesn't give you the desired results.  For example, if you register a
> handler /foo/b*, and send it a message /foo/[a-d]efg, theoretically it
> should match but it won't work.
>
> However, maybe this simple change will at least suit your purposes..
>
>
> Steve
>
>
> diff --git a/src/server.c b/src/server.c
> index 079ea34..fb0fa9b 100644
> --- a/src/server.c
> +++ b/src/server.c
> @@ -763,7 +763,8 @@ static void dispatch_method(lo_server s, const char *path,
>      for (it = s->first; it; it = it->next) {
>         /* If paths match or handler is wildcard */
>         if (!it->path || !strcmp(path, it->path) ||
> -           (pattern && lo_pattern_match(it->path, path))) {
> +           (pattern && lo_pattern_match(it->path, path)) ||
> +        (!pattern && lo_pattern_match(path, it->path))) {
>             /* If types match or handler is wildcard */
>             if (!it->typespec || !strcmp(types, it->typespec)) {
>                 /* Send wildcard path to generic handler, expanded path
> @@ -957,9 +958,11 @@ lo_method lo_server_add_method(lo_server s, const
> char *path,
>      lo_method m = calloc(1, sizeof(struct _lo_method));
>      lo_method it;
>
> +/*
>      if (path && strpbrk(path, " #*,?[]{}")) {
>         return NULL;
>      }
> +*/
>
>      if (path) {
>         m->path = strdup(path);
>
>