Thanks for your response. The pfSense unit is the one that will be offline every now and then, and I don't really worry about a pure router-unit such as the ER-PoR failing. At least that has been my experience with dedicated router units for home use.
The switch in my sketch has only four GbE ports (the other 24 ports are SFP+), and at least three of them are already in use. So unfortunately I don't have three available GbE ports. I may have to get another switch for other reasons, hope to avoid it. That's why I am wondering if my suggestion may work.
I guess perhaps it boils down to the question of whether eth4, eth3 and switch0 are topologically equivalent to each other. If they are, then I should be able to route traffic from eth1 to either switch0 or eth3 (via pfsense on eth2) and connect the modem to eth4. But if the switch-part of the ER-PoR is supposed to aggregate connections before routing, i.e. switch0 has to be the uplink, then it will not work.