On a case-by-case basis, yes it can be done (and it's fine). The trouble comes in when you're trying to set this across an entire network -- not only will you impact the downstream traffic, but any and all upstream traffic you're generating (emails, pictures to facebook, youtube, trying to visit websites, anything).
If one circuit is that limited in bandwidth, you may be better served with two (or more) circuits and PBR to force VoIP / generic data across one or the other.