Mike: Kudos to Spencer for a nice bit of code. Unless I'm reading it wrong, it does exactly the opposite of what I need: the main Forkable blocks until all of its children have finished. As for

...solace in programming as if the memory leaks and performance issues don't exist and I get more real work done this way.

Trust me -- I am NOT one to do "premature optimization". Rather, I was forced find a real solution once I realized I was consistently filling up memory during live performances. (Do you know how long it takes ChucK to respond even to ^C when that happens? The rest of the band was through a verse and the final chorus before I could regain control of my laptop!) I'll explain my approach in the next mail... - Rob On 8 Jun 2009, at 12:40, mike clemow wrote:

Hey Robert,

...

...

... The problem with killing the thread is that you leak 88K with each thread (!!!).

Normally, I feel like I'm the guy complaining about this kind of thing. ;-) I've decided that, despite the memory and performance issues surrounding shreds, sporking them, & cetera, I have decided that the ChucKian way of solving these types of problems is better than trying to program around them in ChucK. Programming around ChucK's issues in any other language would not be terribly hard, but in ChucK, it is hard. The thing is that ChucK provides a way of doing this stuff in a small amount of code and it takes an amount of code multiple orders of magnitude larger to solve the same problem in an performance/memory-optimized way.

The attached piece of code is something that Spencer Salazar whipped up for me one day in response to my gripe about shreds dying without finishing. It combines the idea of a shred and an event and I've based a lot of my frameworks around this one piece of code (thank you Spencer!). To use it, you subclass it and override the run() method with your sporkable shred code, instantiate your subclass and spawn the shred with yourForkable.start(). It would not be difficult to subclass this in such a way as to incorporate the concept of "wait for a signal (e.g. event.broadcast()) OR for a specific time to elapse, whichever comes first." Or it might be that your code wouldn't need such a thing, if you use this--I honestly don't know.

While this implementation doesn't explicitly reuse shreds, I have found solace in programming as if the memory leaks and performance issues don't exist and I get more real work done this way. One day, these problems will no longer exist and all our code will run much smoother. Until then, I've used this code as a basis for a granular synthesis implementation in which every grain is at least one (if not more) shred(s) and there's still a lot it can do before it breaks. I've tried programming around these issues, however, and I end up with code I can't even read, let alone understand or debug.

It's like there's a path-of-least-resistance that ChucK's system provides. Yes, if you travel that path you will run into certain issues, however, if you don't travel that path, you might as well be using something else, because you're missing out on the revolution that this path represents.

This is just my 2 cents.

_mike

On Mon, Jun 8, 2009 at 1:59 PM, dan trueman wrote:

...

jah, i've requested something like this before:

myEvent || 1::minute => now; //time advances to whichever comes first....

i've programmed around it using secondary timing shreds, but it would be much nicer to be able to do something like the above...

dt

On Jun 8, 2009, at 1:53 PM, Robert Poor wrote:

...

Gang:

In real-time music making, sometimes you want to wait for a signal (e.g. event.broadcast()) OR for a specific time to elapse, whichever comes first. I've implemented ways to do this, but I'm not really satisfied with the code.

Here's the problem: Lets say that your music is slaved to a metronome, and the metronome is allowed to change speed. You want your music to stay sync'd to the metronome. If you simply do: now + (1/tempo)::second => time next_beat; next_beat => now; play_note(); you'll be in trouble if the metronome speeds up while you're waiting: your note will be late. The fundamental problem is that once you execute "next_beat => now;", you're committed to waiting and there's now way to break out of it, short of killing the thread. The problem with killing the thread is that you leak 88K with each thread (!!!).

So here's the programming challenge: how would you implement a "wait for signal with timeout" that will block until it gets a signal OR a specified time has elapsed? Since each thread costs 88K in non-reclaimed memory, you may create threads, but your solution must re-use them.

As I said, I have implementations which I'm happy to share, but I'd like to see how you would do this first. Most of all, this is likely to turn into a feature request for the next version of ChucK.

- Rob _______________________________________________ chuck-users mailing list chuck-users@lists.cs.princeton.edu https://lists.cs.princeton.edu/mailman/listinfo/chuck-users

---

chuck-users mailing list chuck-users@lists.cs.princeton.edu https://lists.cs.princeton.edu/mailman/listinfo/chuck-users -- http://michaelclemow.com http://semiotech.org _______________________________________________ chuck-users mailing list chuck-users@lists.cs.princeton.edu https://lists.cs.princeton.edu/mailman/listinfo/chuck-users

On Mon, Jun 8, 2009 at 5:34 PM, Robert Poor wrote:

Trust me -- I am NOT one to do "premature optimization".  Rather, I was forced find a real solution once I realized I was consistently filling up memory during live performances.  (Do you know how long it takes ChucK to respond even to ^C when that happens?  The rest of the band was through a verse and the final chorus before I could regain control of my laptop!)

Haha, I'm eating my words: my little cluster machines are all frozen now because ChucK ate all the memory! ;-) -mc

Hans, Actually, these machines explicitly complain about non-reclaimable memory due a process named "chuck," which has eaten it all. Then either the OS will kill chuck, or I will have to manually kill the system by rebooting the machine. I went through my code, however, and have been finding some objects that are unnecessarily created every time a certain method (spawning a grain) is called, which will definitely contribute to the memory leaks. I'm fixing those as I find them. Wasn't there a trick to deallocating memory for objects? I thought you could do something like, ... null => myObject; ... however, this crashes chuck on my machines. _mc On Tue, Jun 9, 2009 at 4:22 PM, Hans Aberg wrote:

On 9 Jun 2009, at 21:38, mike clemow wrote:

...

Haha, I'm eating my words:  my little cluster machines are all frozen now because ChucK ate all the memory!

It may be due CPU overload, rather than memory exhaustion that might cause segmentation fault instead. This happened when I ran Robert's example, but the machine came back after a while.

 Hans

_______________________________________________ chuck-users mailing list chuck-users@lists.cs.princeton.edu https://lists.cs.princeton.edu/mailman/listinfo/chuck-users

-- http://michaelclemow.com http://semiotech.org

Hi Hans, On Tue, Jun 9, 2009 at 5:54 PM, Hans Aberg wrote:

Aren't you on a POSIX machine, so you can kill it by 'kill -9 <pid>' where <pid> is what you get from  ps -x | grep 'chuck'

Can't even get a console! :(

Kassen suggested  null @=> foo;

Oh, the at-chuck! Of course! **slaps forehead** I'll definitely give that a try. _mike

 Hans

_______________________________________________ chuck-users mailing list chuck-users@lists.cs.princeton.edu https://lists.cs.princeton.edu/mailman/listinfo/chuck-users

-- http://michaelclemow.com http://semiotech.org

Linux 2.6 w/ realtime-patched kernel. I run a cluster of headless AMD Athlon machines with ChucK on each. Each machine only has 256mb of RAM, however, and I'm not surprised about the fact that ChucK can use it all. When it happens, the machine starts spewing error messages to the console (which I never saw until I looked at the output of the monitor--i usually login via ssh), which prevents me from logging in and killing the process. It also prevents me from logging in via ssh. I want to send some of this output, which specifically mentions the process named "chuck" and un-reclaimable memory, to you folks, but it's going to be difficult to capture. I wonder if there's a way to specify a memory quota for a process in Linux. If it tries to use more, it gets killed by the OS... I'm speculating here. -mike On Wed, Jun 10, 2009 at 3:45 AM, Hans Aberg wrote:

On 10 Jun 2009, at 04:58, mike clemow wrote:

...

...

Aren't you on a POSIX machine, so you can kill it by 'kill -9 <pid>' where <pid> is what you get from  ps -x | grep 'chuck'

Can't even get a console!  :(

What might this mean? What platform are you on?

 Hans

-- http://michaelclemow.com http://semiotech.org

Hans, ulimit is the ticket. I'm going to try this tomorrow. Thanks for the tip--I'll let you know how it goes. Cheers, Mike On Wed, Jun 10, 2009 at 10:54 AM, Hans Aberg wrote:

On 10 Jun 2009, at 16:14, mike clemow wrote:

...

Linux 2.6 w/ realtime-patched kernel.  I run a cluster of headless AMD Athlon machines with ChucK on each.  Each machine only has 256mb of RAM, however, and I'm not surprised about the fact that ChucK can use it all.  When it happens, the machine starts spewing error messages to the console (which I never saw until I looked at the output of the monitor--i usually login via ssh), which prevents me from logging in and killing the process.  It also prevents me from logging in via ssh. I want to send some of this output, which specifically mentions the process named "chuck" and un-reclaimable memory, to you folks, but it's going to be difficult to capture.

I wonder if there's a way to specify a memory quota for a process in Linux.  If it tries to use more, it gets killed by the OS...  I'm speculating here.

Yes, ulimit. It is tied to the shell. The stuff below is form 'man bash'. If you do not specify -S it sets a hard limit which can only be adjusted downwards. So -v -s and -d might be to set (in 1 kb increments).

 Hans

---- ulimit[−SHacdefilmnpqrstuvx[limit]] Provides control overthe resources available to the shell and to processes started by it, on systems that allowsuch control. The−Hand−Soptions specify that the hard or soft limit is set for the givenresource. Ahard limit cannot be increased once it is set; a soft limit may be increased up to the value of the hard limit. If neither−Hnor−Sis specified, both the soft and hard limits are set. The value oflimit can be a number in the unit specified for the resource or one of the special val- ueshard, soft, orunlimited, which stand for the current hard limit, the current soft limit, and no limit, respectively. Iflimit is omitted, the current value of the soft limit of the resource is printed, unless the−Hoption is given. Whenmore than one resource is specified, the limit name and unit are printed before the value. Otheroptions are interpreted as follows: −a All current limits are reported −c The maximum size of core files created −d The maximum size of a process’sdata segment −e The maximum scheduling priority ("nice") −f The maximum size of files written by the shell and its children −i The maximum number of pending signals −l The maximum size that may be locked into memory −m The maximum resident set size −n The maximum number of open file descriptors (most systems do not allowthis value to be set) −p The pipe size in 512-byte blocks (this may not be set) −q The maximum number of bytes in POSIX message queues −r The maximum real-time scheduling priority −s The maximum stack size −t The maximum amount of cpu time in seconds −u The maximum number of processes available to a single user −v The maximum amount of virtual memory available to the shell −x The maximum number of file locks Iflimit is given, it is the newvalue of the specified resource (the−aoption is display only). If no option is given, then−fis assumed. Values are in 1024-byte increments, except for−t, which is in seconds, −p, which is in units of 512-byte blocks, and−nand−u, which are unscaled values. The return status is 0 unless an invalid option or argument is supplied, or an error occurs while setting anew limit. ----

-- http://michaelclemow.com http://semiotech.org

Hans: You approach looks just fine for situations where you generate notes at a "musically sensible" rate. My system isn't musically sensible: I'm generating dozens of "I just modified the end time of your event" messages every second, so sporking a shred and letting it die is not an option -- I'd be burning a megabyte of non-recoverable memory every second. I suspect Dan Trueman is doing the same thing I'm doing, namely spawning secondary timing shreds as needed, which can be kept around and re-used after they've done their duty. - Rob On 8 Jun 2009, at 12:24, Hans Aberg wrote:

On 8 Jun 2009, at 19:53, Robert Poor wrote:

...

In real-time music making, sometimes you want to wait for a signal (e.g. event.broadcast()) OR for a specific time to elapse, whichever comes first. I've implemented ways to do this, but I'm not really satisfied with the code.

Here's the problem: Lets say that your music is slaved to a metronome, and the metronome is allowed to change speed. You want your music to stay sync'd to the metronome. If you simply do: now + (1/tempo)::second => time next_beat; next_beat => now; play_note(); you'll be in trouble if the metronome speeds up while you're waiting: your note will be late. The fundamental problem is that once you execute "next_beat => now;", you're committed to waiting and there's now way to break out of it, short of killing the thread. The problem with killing the thread is that you leak 88K with each thread (!!!).

So here's the programming challenge: how would you implement a "wait for signal with timeout" that will block until it gets a signal OR a specified time has elapsed? Since each thread costs 88K in non-reclaimed memory, you may create threads, but your solution must re-use them.

You might check out the noteoff() function I implemented here: https://lists.cs.princeton.edu/pipermail/chuck-users/2009-May/004183.html

There is a similar problem of future event rescheduling: a future disconnect, that should be canceled, if there has been a new note- on. So the future event gets a thread of its own, and when it awakens, it checks if there has been a rescheduling - if so, it just cancels the event.

As 'chuck' does not have event scheduling and kill-threads was not intended for massive use, this seems to be the only way. But it works just fine.

Hans

--: Robert Poor e: robert.poor@nbt-ventures.com p: +1 617 818 5115 b: http://blog.nbt-ventures.com --: This message and the information it contains are the proprietary and confidential property of NBT Ventures and may be privileged. If you are not the intended recipient, please do not read, copy, disclose or distribute its contents to any party, and notify the sender immediately. --:

Hans:

I am not sure exactly how 'chuck' does memory allocations, but I avoid allocate any reference types in the threads. I have had the program running for long time. Are you saying the threads themselves are leaking?

Essentially, yes: sporking a shred eats up 88.3K of non-recovered memory. It's easy to show with this bit of code: ================================= Std.system("ps gauxw | head -1"); // warm up the VM for (0=>int i; i<10; i++) { 10 => int x; } <<< "on startup" >>>; Std.system("ps gauxw | grep chuck | grep -v grep"); 10000 => int ntest; fun void testShred() { } for (0=>int i; i>>; Std.system("ps gauxw | grep chuck | grep -v grep"); for (0=>int i; i>>; Std.system("ps gauxw | grep chuck | grep -v grep"); ================================= which produces this output: ================================= [poorbook15-9:~/Desktop] r% chuck --caution-to-the-wind alloc_test1.ck USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND "on startup" : (string) r 207 2.5 0.5 82160 4928 p0 S+ 3:58PM 0:00.52 chuck --caution-to-the-wind alloc_test1.ck "round 1: spork ~ testShred(); me.yield();" : (string) r 207 0.1 82.8 966832 868412 p0 S+ 3:58PM 0:12.20 chuck --caution-to-the-wind alloc_test1.ck "round 2: spork ~ testShred(); me.yield();" : (string) r 207 0.0 81.8 1850480 857960 p0 S+ 3:58PM 0:52.08 chuck --caution-to-the-wind alloc_test1.ck ================================= The thing to notice is the VSZ column -- that's "virtual size" in kbytes. It grows by (966832-82160)=884672 kbytes on round 1 and (1850480-966832)=883648 kbytes on round 2. Since there's 100000 sporks per round, that's a consistent 88.3 kBytes per spork, even though the testShred is as simple as can be. - Rob On 8 Jun 2009, at 15:00, Hans Aberg wrote:

On 8 Jun 2009, at 23:20, Robert Poor wrote:

...

You approach looks just fine for situations where you generate notes at a "musically sensible" rate.

My system isn't musically sensible: I'm generating dozens of "I just modified the end time of your event" messages every second, so sporking a shred and letting it die is not an option -- I'd be burning a megabyte of non-recoverable memory every second.

I am not sure exactly how 'chuck' does memory allocations, but I avoid allocate any reference types in the threads. I have had the program running for long time. Are you saying the threads themselves are leaking?

...

I suspect Dan Trueman is doing the same thing I'm doing, namely spawning secondary timing shreds as needed, which can be kept around and re-used after they've done their duty.

One idea I have, but haven't tried, is making a thread an event scheduler.

Hans

On 9 Jun 2009, at 01:22, Robert Poor wrote:

...

Are you saying the threads themselves are leaking?

Essentially, yes: sporking a shred eats up 88.3K of non-recovered memory. It's easy to show with this bit of code: ================================= ... ================================= which produces this output: ================================= [poorbook15-9:~/Desktop] r% chuck --caution-to-the-wind alloc_test1.ck USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND "on startup" : (string) r 207 2.5 0.5 82160 4928 p0 S+ 3:58PM 0:00.52 chuck --caution-to-the-wind alloc_test1.ck "round 1: spork ~ testShred(); me.yield();" : (string) r 207 0.1 82.8 966832 868412 p0 S+ 3:58PM 0:12.20 chuck --caution-to-the-wind alloc_test1.ck "round 2: spork ~ testShred(); me.yield();" : (string) r 207 0.0 81.8 1850480 857960 p0 S+ 3:58PM 0:52.08 chuck --caution-to-the-wind alloc_test1.ck =================================

The thing to notice is the VSZ column -- that's "virtual size" in kbytes. It grows by (966832-82160)=884672 kbytes on round 1 and (1850480-966832)=883648 kbytes on round 2. Since there's 100000 sporks per round, that's a consistent 88.3 kBytes per spork, even though the testShred is as simple as can be.

I think you may have not waited enough for the threads to clean up. I first ran your code verbatim, and got: $ chuck --caution-to-the-wind thread_timing.ck USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND "on startup" : (string) haberg 673 9.2 0.5 84720 4920 s000 S+ 6:32PM 0:00.37 chuck --caution-to-the-wind thread_timing.ck "round 1: spork ~ testShred(); me.yield();" : (string) haberg 673 0.1 77.5 968368 812132 s000 S+ 6:32PM 0:14.92 chuck --caution-to-the-wind thread_timing.ck "round 2: spork ~ testShred(); me.yield();" : (string) haberg 673 0.2 81.3 1853040 852524 s000 S+ 6:32PM 0:53.13 chuck --caution-to-the-wind thread_timing.ck The figures in the VSZ columen are almost identical to yours. But the I added a 5 minute wait before writing the stuff out, to give time for any cleanups. Then I got chuck --caution-to-the-wind thread_allocation.ck USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND "on startup" : (string) haberg 703 0.7 0.5 84720 4920 s000 S+ 6:40PM 0:00.48 chuck --caution-to-the-wind thread_allocation.ck "round 1: spork ~ testShred(); me.yield();" : (string) haberg 703 0.6 0.5 89416 4904 s000 S+ 6:40PM 0:45.87 chuck --caution-to-the-wind thread_allocation.ck "round 2: spork ~ testShred(); me.yield();" : (string) haberg 703 0.8 0.4 89416 4680 s000 S+ 6:40PM 1:31.24 chuck --caution-to-the-wind thread_allocation.ck As you see, now the VSZ column stabilizes and does not grow at all. Hans

Tom: Very clean!! And probably not too hard to modify for re-usable shreds. I'll report back. - Rob On 8 Jun 2009, at 15:18, Tom Lieber wrote:

On Mon, Jun 8, 2009 at 1:53 PM, Robert Poor wrote:

...

In real-time music making, sometimes you want to wait for a signal (e.g. event.broadcast()) OR for a specific time to elapse, whichever comes first.

I don't re-use shreds, but I think this is useful anyway:

---

class NoteEvent extends Event { 0 => int sent; }

public class Timeout { fun Event either(Event ev, dur d) { NoteEvent real; spork ~ time_alert(real, d); spork ~ event_alert(real, ev); return real; }

fun void time_alert(NoteEvent real, dur d) { d => now; if(!real.sent) real.broadcast(); 1 => real.sent; }

fun void event_alert(NoteEvent real, Event ev) { ev => now; if(!real.sent) real.broadcast(); 1 => real.sent; } }

---

It's used like:

Timeout tout; tout.either(some_event, 3::second) => now;

-- Tom Lieber http://AllTom.com/ _______________________________________________ chuck-users mailing list chuck-users@lists.cs.princeton.edu https://lists.cs.princeton.edu/mailman/listinfo/chuck-users