You can also use rate-limiting rules in your firewall. This for SSH brute force attacks:
rule 4 { action drop description "SSH Bruteforce prevention" destination { port 22 } log disable protocol tcp recent { count 3 time 600 } state { established disable invalid disable new enable related disable } }
You could adapt it for mailserver by changing it to refer to port 25 (and any other ports your mailserver uses/is being attacked on).