I've been using ER-X at home for a few weeks. It's my first Edgerouter. Very happy with it so far
With EdgeOS v1.9.0, we know ER-X supports HW crypto. I've seen impressive numbers from earlier tests e.g. this thread in Beta forum. This time around I put it under my own tests and the results excite me.
Test setup:
Macbook Pro <---> ER-X <---> iMac
[on WAN side] [on LAN side]
ER-X was configured as IKEv2 IPsec server. HWNAT, IPsec offload, firewall and NAT were turned on. No QoS applied in either direction. Mackbook Pro was the IKEv2 client on WAN side. For the IPsec tunnel, I used AES-128/SHA1 for ESP and AES-256/SHA256 for IKEv2 keying. The tests used iperf3 (in TCP mode) to send multiple concurrent (i.e. -P option) streams between MBP and iMac.
Here are the results of four concurrent streams:
(Note that the latency between WAN and LAN <5ms. YMMV)
The packet sizes are in MSS. Download means transferring from iMac to MBP. Upload means from MBP to iMac.
Even though I was told of good numbers before, I was very much suprised to see 377Mbit/s! Not only that but consistently reproducible. As a comparison, a single iperf3 stream at 1460-byte, download and upload were 244Mbit/s and 128Mbit/s respectively.
I saw concurrent streams utilized further performance in download direction. But in upload direction it didn't change at all. Looking a bit further, I could see in ER-X processes that in download tests, all four ksoftirqd were working (with one or two near full load) but in upload tests, only one or two ksoftirqd were working (only one near full load IIRC).
I also notice in v1.9.0 both the switch and HW crypto are configured to interrupt in CPU0. I wonder if putting HW crypto on a different CPU will make a difference. Frankly I'm not sure if that's changeable.