Ran into a problem when using VTI's on Edgerouter-lites connecting to a Fortigate firewall. The behavior manifests as unexpectedly slow throughput and wildly variable numbers when testing. Speeds swing rapidly by as much as a thousand percent over the course of several seconds, in an almost wave-like pattern, but only on traffic coming from the Fortigate device. Traffic from the EdgeOS device through the Fortigate is fine (as long as it's not going back out another tunnel to an Edgerouter).
Long story short, there is an apparent difference in the way EdgeOS and Fortinet handle large packets. It appears the Ubiquiti fragments the payload, while Fortinet fragments the whole darn ESP packet, post-encapsulation. In other words, ubiquiti sends consistent envelops with peices of the letter, while fortinet sends one big envelope in several pieces that then has to be reassembled before it can decrypt. Before long, what's coming out of the Edgerouter starts to resemble this:
https://www.youtube.com/watch?v=8NPzLBSBzPI
What we're curious about is whether the need to reassemble the fragmented ESP packets is breaking offload. How do we check for that, and where do we go from here?