The ER-X has an internal switch chip. Eth2/3/4 are included on the switch chip by the wizard, so you can operate a small network without needing a separate switch.
Yes, you can remove one or more more interfaces from the switch chip. There will be no performance penalty. To use an interface that you remove from the switch chip, you will need to assign a new distinct subnet to that interface.