The QoS feature is not supported by hardware offloading so whereas you will see line speeds with QoS disabled, once you enable QoS (and thus disable hardware offloading for that traffic) your throughput figures will plummet.
I recently tested an ERL with QoS enabled and it maxed out at around 90-120Mbps in a relatively simple setup. See here:
So yes, what you are seeing is normal. You will not be able to achieve anything like gigabit speeds with QoS enabled.