The interesting part here is that rules 1-3 work as is. I first applied them without the packet marking. But as that alone didn't seem to work I additionaly went for the packet marks. The odd thing is that ip rule show actually shows 0 hits for rule 2, but it doesn't work without that rule. I will have to investigate further and possibly add "accept" to the modify rules in order to force cancel further rule processing so packets don't get further processed by the load balancer.
As per your instructions I will stick with packet instead of connection source/destination.
I'm aware that "live" modifying firewall rules is far from a clean approach - but yet that shouldn't break things after a reboot -at least not when considering the config style approach of EdgeOS/VyOS, which should clear and reapply everything on reboot.
I'll post an update tomorrow or so... don't know if I have enough time today to start from scratch...