I was able to get replies back. I had my modify rule on the wrong interface. It was on switch0 instead of vtun0. Now onto making sure the traffic is on the right interface.
Edit: This appears to only be relevant for the interface as a source to be able to pass traffic.