Hi Julien,
I work on Linux as well, using Jack as audio core. there's a specific setup I've got that uses ChucK as slave (an external hardware sends MIDI clock to it and Pure Data). I have to say I've never experienced anything like this, but honestly this specific ChucK script I'm using is in sync with everything, but doesn't need to be super accurate. So I've never measured if performance in this situation. could you provide some info more about your settings? how are you setting latency? buffer size on Jack, ChucK?