Probably, you don't have access to ISP device, so you can't do a thing on that side (like lowering ARP timeout)
On ER side, there's not much you can do. Or ugly: ditch vrrp on WAN link, and use on LAN a VRRP transition script so that active/passiv VRRP router brings up/down its WAN port (and make their MAC addresses identical)