HWNAT was not loaded, but loading it didn't change my download speed.
Yes, I'm using the same device for both speedtests.
Another interesting thing is that my ping times are unaffected. Speedtest consistently shows <20ms pings and the largest ping time during a tracert to 8.8.8.8 was 37ms.