To make a long story short. My Site to Site has been working for a long time between my two sites. With a bone headed move, both routers got clobbered and I had to restore both routers from a backup. Yes, the backups were recent.
Everything works, however both sites IPSEC Status shows CONNECTING. Here is what I know:
1) I validated both internet facing IP's. Both Ping and Traceroute shows that they work. I have also done this via the Toolbox on EdgeMax OS.
2) H Site is an ER-4 running v1.10.9 (I did note some changes on the subnets on this version)
3) C Site is an ER-Line running v1.9.7+hotfix.2
4) I have attached the configs for both H and C.
5) I did watch sudo swanctl --log for both sites and it seems that the issue is that it is doing a lot of retransmits. Like this:
06[IKE] received retransmit of request with ID 0, retransmitting response
06[NET] sending packet: from 192.225.176.79[500] to 98.202.96.65[500] (136 bytes)
14[IKE] sending retransmit 4 of request message ID 0, seq 1
14[NET] sending packet: from 192.225.176.79[500] to 98.202.96.65[500] (156 bytes)
10[JOB] deleting half open IKE_SA after timeout
01[NET] received packet: from 98.202.96.65[500] to 192.225.176.79[500] (156 bytes)
01[ENC] parsed ID_PROT request 0 [ SA V V V V ]
01[IKE] received XAuth vendor ID
01[IKE] received DPD vendor ID
01[IKE] received NAT-T (RFC 3947) vendor ID
01[IKE] received draft-ietf-ipsec-nat-t-ike-02\n vendor ID
01[IKE] 98.202.96.65 is initiating a Main Mode IKE_SA
01[ENC] generating ID_PROT response 0 [ SA V V V ]
01[NET] sending packet: from 192.225.176.79[500] to 98.202.96.65[500] (136 bytes)
05[IKE] sending retransmit 5 of request message ID 0, seq 1
05[NET] sending packet: from 192.225.176.79[500] to 98.202.96.65[500] (156 bytes)
02[JOB] deleting half open IKE_SA after timeout
15[NET] received packet: from 98.202.96.65[500] to 192.225.176.79[500] (156 bytes)
15[ENC] parsed ID_PROT request 0 [ SA V V V V ]
15[IKE] received XAuth vendor ID
15[IKE] received DPD vendor ID
15[IKE] received NAT-T (RFC 3947) vendor ID
15[IKE] received draft-ietf-ipsec-nat-t-ike-02\n vendor ID
15[IKE] 98.202.96.65 is initiating a Main Mode IKE_SA
15[ENC] generating ID_PROT response 0 [ SA V V V ]
15[NET] sending packet: from 192.225.176.79[500] to 98.202.96.65[500] (136 bytes)
14[JOB] deleting half open IKE_SA after timeout
03[IKE] giving up after 5 retransmits
6) Here is the IPSEC Status All for both sites:
HSite IPSEC StatusAll:
Status of IKE charon daemon (strongSwan 5.2.2, Linux 3.10.107-UBNT, mips64):
uptime: 96 minutes, since Apr 27 07:35:08 2019
malloc: sbrk 376832, mmap 0, used 289416, free 87416
worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 1
loaded plugins: charon ldap sqlite pkcs11 aes des sha1 sha2 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs8 pem openssl agent xcbc cmac ctr ccm gcm curl attr kernel-netlink resolve socket-default stroke vici updown eap-identity eap-md5 eap-mschapv2 eap-radius eap-tls xauth-generic xauth-eap addrblock
Listening IP addresses:
98.202.96.65
172.22.22.1
172.22.23.1
Connections:
peer-ccc.mycompany.com-tunnel-1: %any...ccc.mycompany.com IKEv1
peer-ccc.mycompany.com-tunnel-1: local: [ccc.mycompany.com] uses public key authentication
peer-ccc.mycompany.com-tunnel-1: cert: "ccc.mycompany.com"
peer-ccc.mycompany.com-tunnel-1: remote: [hhh.mycompany.com] uses public key authentication
peer-ccc.mycompany.com-tunnel-1: cert: "hhh.mycompany.com"
peer-ccc.mycompany.com-tunnel-1: child: 172.22.22.0/24 === 172.22.26.0/24 TUNNEL
peer-ccc.mycompany.com-tunnel-2: child: 172.22.23.0/24 === 172.22.26.0/24 TUNNEL
peer-ccc.mycompany.com-tunnel-3: child: 172.22.22.0/24 === 172.22.25.0/24 TUNNEL
peer-ccc.mycompany.com-tunnel-4: child: 172.22.23.0/24 === 172.22.25.0/24 TUNNEL
remote-access: 98.202.96.65...%any IKEv1, dpddelay=15s
remote-access: local: [98.202.96.65] uses pre-shared key authentication
remote-access: remote: uses pre-shared key authentication
remote-access: child: dynamic[udp/l2f] === dynamic[udp] TRANSPORT, dpdaction=clear
Routed Connections:
peer-ccc.mycompany.com-tunnel-4{8}: ROUTED, TUNNEL
peer-ccc.mycompany.com-tunnel-4{8}: 172.22.23.0/24 === 172.22.25.0/24
peer-ccc.mycompany.com-tunnel-3{7}: ROUTED, TUNNEL
peer-ccc.mycompany.com-tunnel-3{7}: 172.22.22.0/24 === 172.22.25.0/24
peer-ccc.mycompany.com-tunnel-2{6}: ROUTED, TUNNEL
peer-ccc.mycompany.com-tunnel-2{6}: 172.22.23.0/24 === 172.22.26.0/24
peer-ccc.mycompany.com-tunnel-1{5}: ROUTED, TUNNEL
peer-ccc.mycompany.com-tunnel-1{5}: 172.22.22.0/24 === 172.22.26.0/24
Security Associations (1 up, 0 connecting):
peer-ccc.mycompany.com-tunnel-1[1]: CONNECTING, 98.202.96.65[%any]...192.225.176.79[%any]
peer-ccc.mycompany.com-tunnel-1[1]: IKEv1 SPIs: 26cb498f48c9cdc5_i* 0000000000000000_r
peer-ccc.mycompany.com-tunnel-1[1]: Tasks queued: QUICK_MODE QUICK_MODE QUICK_MODE QUICK_MODE
peer-ccc.mycompany.com-tunnel-1[1]: Tasks active: ISAKMP_VENDOR ISAKMP_CERT_PRE MAIN_MODE ISAKMP_CERT_POST ISAKMP_NATD
CSITE IPSEC StatusAll:
Status of IKE charon daemon (strongSwan 5.2.2, Linux 3.10.20-UBNT, mips64):
uptime: 63 minutes, since Apr 27 07:35:23 2019
malloc: sbrk 373904, mmap 0, used 285528, free 88376
worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 1
loaded plugins: charon ldap sqlite pkcs11 aes des sha1 sha2 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs8 pem openssl agent xcbc cmac ctr ccm gcm curl attr kernel-netlink resolve socket-default stroke vici updown eap-identity eap-md5 eap-mschapv2 eap-radius eap-tls xauth-generic xauth-eap addrblock
Listening IP addresses:
172.22.26.1
172.22.25.1
192.225.176.79
Connections:
peer-hhh.mycompany.com-tunnel-1: %any...hhh.mycompany.com IKEv1
peer-hhh.mycompany.com-tunnel-1: local: [hhh.mycompany.com] uses public key authentication
peer-hhh.mycompany.com-tunnel-1: cert: "hhh.mycompany.com"
peer-hhh.mycompany.com-tunnel-1: remote: [ccc.mycompany.com] uses public key authentication
peer-hhh.mycompany.com-tunnel-1: cert: "ccc.mycompany.com"
peer-hhh.mycompany.com-tunnel-1: child: 172.22.26.0/24 === 172.22.22.0/24 TUNNEL
peer-hhh.mycompany.com-tunnel-2: child: 172.22.26.0/24 === 172.22.23.0/24 TUNNEL
peer-hhh.mycompany.com-tunnel-3: child: 172.22.25.0/24 === 172.22.22.0/24 TUNNEL
peer-hhh.mycompany.com-tunnel-4: child: 172.22.25.0/24 === 172.22.23.0/24 TUNNEL
remote-access: 192.225.176.79...%any IKEv1, dpddelay=15s
remote-access: local: [192.225.176.79] uses pre-shared key authentication
remote-access: remote: uses pre-shared key authentication
remote-access: child: dynamic[udp/l2f] === dynamic[udp] TRANSPORT, dpdaction=clear
Routed Connections:
peer-hhh.mycompany.com-tunnel-4{8}: ROUTED, TUNNEL
peer-hhh.mycompany.com-tunnel-4{8}: 172.22.25.0/24 === 172.22.23.0/24
peer-hhh.mycompany.com-tunnel-3{7}: ROUTED, TUNNEL
peer-hhh.mycompany.com-tunnel-3{7}: 172.22.25.0/24 === 172.22.22.0/24
peer-hhh.mycompany.com-tunnel-2{6}: ROUTED, TUNNEL
peer-hhh.mycompany.com-tunnel-2{6}: 172.22.26.0/24 === 172.22.23.0/24
peer-hhh.mycompany.com-tunnel-1{5}: ROUTED, TUNNEL
peer-hhh.mycompany.com-tunnel-1{5}: 172.22.26.0/24 === 172.22.22.0/24
Security Associations (1 up, 0 connecting):
peer-hhh.mycompany.com-tunnel-1[1]: CONNECTING, 192.225.176.79[%any]...98.202.96.65[%any]
peer-hhh.mycompany.com-tunnel-1[1]: IKEv1 SPIs: 7b3dd9469bfe2980_i* 0000000000000000_r
peer-hhh.mycompany.com-tunnel-1[1]: Tasks queued: QUICK_MODE QUICK_MODE
peer-hhh.mycompany.com-tunnel-1[1]: Tasks active: ISAKMP_VENDOR ISAKMP_CERT_PRE MAIN_MODE ISAKMP_CERT_POST ISAKMP_NATD