I think I figured out why the traffic shaping does not work:
tcpdump -vvv -i eth0 'udp and not src net 192.168.0.0/16 and port not 22 and port not 53'
shows me RTP packets coming in from the SIP provider to the machine asterisk is running on like this:
03:36:16.109587 IP (tos 0x0, ttl 249, id 12088, offset 0, flags [DF], proto UDP (17), length 200)<src ip-address>.36520 > host.example.com.12254: [udp sum ok] UDP, length 172
The DSCP/TOS bit is not set 
I could make the traffic limiter (or shaper) match by the source IP, but that address might change.
What would be the best way to handle this?