How can you Analyse VPN IPSec Log?

Here we take an example with brief description to teach you how to read the IPSec log of Vigor router, so that you may be able to do some basic troubleshooting by yourself.

The IPSec protocol is complicated and it is hard to explain clearly with simple words. Therefore, if you have problems on resolving an IPSec issue by yourself, please do not hesitate to contact us and offer the VPN log.

VPN is initiated from Vigor5500 to Vigor2820

Please connect VPN. Type the command “log -wt” by using Telnet. You may get the following output.

Please note that:

++++>                 indicates connection direction (data transmission) is from local to remote
<++++                   indicates connection direction (data transmission) is from remote to local

Password: ********************
Type ? for command help
> log -wt
0:00:44.840 ++++>IKE Len = 296
I Cookie=0xb9 f0 0c 1a a2 e6 89 db, R Cookie=0x00 00 00 00 00 00 00 00
Next Payload = ISAKMP_NEXT_SA
Exchange Type = 0x2
Message ID = 0x0
Payload Type = SA
Next Payload = ISAKMP_NEXT_VID
Payload Length = 0x94
DOI = 0x1
Situation = 0x1
Proposal #0x0, Protocol Id = 0x1, SPI Size = 0x0, Number of Transforms = 0x4
Transform #0x0, Transform ID = 0x1, Length = 0x18
80 0b 00 01
80 0c 03 84
80 01 00 01
80 02 00 01
80 03 00 01
80 04 00 01
Transform #0x1, Transform ID = 0x1, Length = 0x18
80 0b 00 01
80 0c 03 84
80 01 00 01
80 02 00 02
80 03 00 01
80 04 00 01
Transform #0x2, Transform ID = 0x1, Length = 0x18
80 0b 00 01
80 0c 03 84
80 01 00 05
80 02 00 01
80 03 00 01
80 04 00 01
Transform #0x3, Transform ID = 0x1, Length = 0x18
80 0b 00 01
80 0c 03 84
80 01 00 05
80 02 00 01
80 03 00 01
80 04 00 02
Payload Type= VENDOR ID
Next Payload = ISAKMP_NEXT_VID
Payload Length = 0x14
VID Data = 0xaf ca d7 13 68 a1 f1 c9 6b 86 96 fc 77 57 01 00
Payload Type= VENDOR ID
Next Payload = ISAKMP_NEXT_VID
Payload Length = 0x14
VID Data = 0x4a 13 1c 81 07 03 58 45 5c 57 28 f2 0e 95 45 2f
Payload Type= VENDOR ID
Next Payload = ISAKMP_NEXT_VID
Payload Length = 0x14
VID Data = 0x7d 94 19 a6 53 10 ca 6f 2c 17 9d 92 15 52 9d 56
Payload Type= VENDOR ID
Next Payload = ISAKMP_NEXT_VID
Payload Length = 0x14
VID Data = 0x90 cb 80 91 3e bb 69 6e 08 63 81 b5 ec 42 7b 1f
Payload Type= VENDOR ID
Next Payload = ISAKMP_NEXT_VID
Payload Length = 0x14
VID Data = 0xcd 60 46 43 35 df 21 f8 7c fd b2 fc 68 b6 a4 48
Payload Type= VENDOR ID
Next Payload = ISAKMP_NEXT_NONE
Payload Length = 0x14
VID Data = 0x44 85 15 2d 18 b6 bb cd 0b e8 a8 46 95 79 dd cc
0:00:44.970 <++++IKE Len = 120
I Cookie=0xb9 f0 0c 1a a2 e6 89 db , R Cookie=0x28 04 b5 7f b8 39 77 3d
Next Payload = ISAKMP_NEXT_SA
Exchange Type = 0x2
Message ID = 0x0
Payload Type = SA
Next Payload = ISAKMP_NEXT_VID
Payload Length = 0x34
DOI = 0x1
Situation = 0x1
Proposal #0x0, Protocol Id = 0x1, SPI Size = 0x0, Number of Transforms = 0x1
Transform #0x0, Transform ID = 0x1, Length = 0x18
80 0b 00 01
80 0c 03 84
80 01 00 01
80 02 00 01
80 03 00 01
80 04 00 01
Payload Type= VENDOR ID
Next Payload = ISAKMP_NEXT_VID
Payload Length = 0x14
VID Data = 0xaf ca d7 13 68 a1 f1 c9 6b 86 96 fc 77 57 01 00
Payload Type= VENDOR ID
Next Payload = ISAKMP_NEXT_NONE
Payload Length = 0x14
VID Data = 0x4a 13 1c 81 07 03 58 45 5c 57 28 f2 0e 95 45 2f
0:00:45.000 ++++>IKE Len = 188
I Cookie=0xb9 f0 0c 1a a2 e6 89 db , R Cookie=0x28 04 b5 7f b8 39 77 3d
Next Payload = ISAKMP_NEXT_KE
Exchange Type = 0x2
Message ID = 0x0
Payload Type = KEY EX
Next Payload = ISAKMP_NEXT_NONCE
Payload Length = 0x64
Key = 0x30 da 16 b0 e0 50 5f 90 51 7c ce 8e 0c 42 2c 59
73 35 98 83 bd 96 7e b7 29 e1 7d b5 16 e2 73 fe
11 01 44 23 d4 6d 35 78 68 a9 de 89 12 72 4c f3
71 5c a5 3d 2f 18 e3 1c 7e 83 75 02 fa 09 b4 3d
9f 52 05 7d ac d2 2e 70 37 21 54 4c 55 e8 34 04
b8 0c 32 c9 8c 05 9a eb 72 c9 e3 2a 3f 06 96 57
Payload Type= NONCE
Next Payload = ISAKMP_NEXT_NAT-D
Payload Length = 0x14
Nonce = 0x33 42 4a 4e d1 13 b4 05 ae 83 6e 64 60 5e 5f 60
Payload Type= NAT-D
Next Payload = ISAKMP_NEXT_NAT-D
NAT-D Length = 0x14
NAT-D = 0xf5 33 e5 65 ef d4 e8 4e da 2a 88 09 e8 c1 10 cc
Payload Type= NAT-D
Next Payload = ISAKMP_NEXT_NONE
NAT-D Length = 0x14
NAT-D = 0x3f bd 25 13 76 12 81 b9 1e 37 fd a7 a2 41 a7 85
0:00:45.200 <++++IKE Len = 188
I Cookie=0xb9 f0 0c 1a a2 e6 89 db , R Cookie=0x28 04 b5 7f b8 39 77 3d
Next Payload = ISAKMP_NEXT_KE
Exchange Type = 0x2
Message ID = 0x0
Payload Type = KEY EX
Next Payload = ISAKMP_NEXT_NONCE
Payload Length = 0x64
Key = 0x33 cb 5a bf 6b 3b 49 4d 32 af 60 2f 9e 8f 9c 86
f3 b9 ce 55 9e e5 a8 6a 9f 3d 3c 25 d8 2a a7 de
21 df f0 31 aa 6d 22 c5 57 49 b0 4f ba d0 ca 97
98 6f cb d6 74 c6 06 d9 0e ce bc 02 a7 0a fa 49
ad 99 75 32 c5 3f b0 a7 ed ed 4e 9d 19 40 ec 82
23 17 13 69 9e 4b b0 04 64 50 36 d6 82 f9 f9 d9
Payload Type= NONCE
Next Payload = ISAKMP_NEXT_NAT-D
Payload Length = 0x14
Nonce = 0x24 48 5a 64 e9 2c 4e 60 e9 ae 91 03 3d 5a 69 f1
Payload Type= NAT-D
Next Payload = ISAKMP_NEXT_NAT-D
NAT-D Length = 0x14
NAT-D = 0x3f bd 25 13 76 12 81 b9 1e 37 fd a7 a2 41 a7 85
Payload Type= NAT-D
Next Payload = ISAKMP_NEXT_NONE
NAT-D Length = 0x14
NAT-D = 0xf5 33 e5 65 ef d4 e8 4e da 2a 88 09 e8 c1 10 cc
0:00:45.240 ++++>IKE Len = 88
I Cookie=0xb9 f0 0c 1a a2 e6 89 db , R Cookie=0x28 04 b5 7f b8 39 77 3d
Next Payload = ISAKMP_NEXT_ID
Exchange Type = 0x2
Message ID = 0x0
Payload Type = ID
Next Payload = ISAKMP_NEXT_HASH
Payload Length = 0xc
ID Type = 0x01
Protocol ID = 0x0
Port = 0x0
ID = 0xda f2 82 12
Payload Type = HASH
Next Payload = ISAKMP_NEXT_N
Payload Length = 0x14
Hash = 0x9e dc ff 64 f7 26 fa 72 58 0e 8b f0 9c ca 6c 40
Payload Type = NOTIFICATION
Next Payload = ISAKMP_NEXT_NONE
Payload Length = 0x1c
DOI = 0x1
Protocol ID = 0x01, SPI SIZE = 0x10, Message Type = 0x6002
SPI = b9 f0 0c 1a a2 e6 89 db 28 04 b5 7f b8 39 77 3d
Notification Data =
0:00:45.330 <++++IKE Len = 92
I Cookie=0xb9 f0 0c 1a a2 e6 89 db , R Cookie=0x28 04 b5 7f b8 39 77 3d
Next Payload = ISAKMP_NEXT_ID
Exchange Type = 0x2
Message ID = 0x0
Payload Type = ID
Next Payload = ISAKMP_NEXT_HASH
Payload Length = 0xc
ID Type = 0x01
Protocol ID = 0x0
Port = 0x0
ID = 0xdc 80 e6 79
Payload Type = HASH
Next Payload = ISAKMP_NEXT_N
Payload Length = 0x14
Hash = 0x15 97 0f c0 3e 20 eb fa 6a 9f 76 43 82 10 6f f9
Payload Type = NOTIFICATION
Next Payload = ISAKMP_NEXT_NONE
Payload Length = 0x1c
DOI = 0x1
Protocol ID = 0x01, SPI SIZE = 0x10, Message Type = 0x6002
SPI = b9 f0 0c 1a a2 e6 89 db 28 04 b5 7f b8 39 77 3d
Notification Data =
0:00:45.330 ++++>IKE Len = 172
I Cookie=0xb9 f0 0c 1a a2 e6 89 db , R Cookie=0x28 04 b5 7f b8 39 77 3d
Next Payload = ISAKMP_NEXT_HASH
Exchange Type = 0x20
Message ID = 0xeca88777
Payload Type = HASH
Next Payload = ISAKMP_NEXT_SA
Payload Length = 0x14
Hash = 0x90 fc 3b 5d 7e 7f 8f 5d 34 24 9a 29 ac d9 3b 1c
Payload Type = SA
Next Payload = ISAKMP_NEXT_NONCE
Payload Length = 0x48
DOI = 0x1
Situation = 0x1
Proposal #0x0, Protocol Id = 0x3, SPI Size = 0x4, Number of Transforms = 0x2
SPI = f0 ac 8b 7b
Transform #0x0, Transform ID = 0x2, Length = 0x10
80 04 00 01
80 01 00 01
80 02 02 58
80 05 00 02
Transform #0x1, Transform ID = 0x2, Length = 0x10
80 04 00 01
80 01 00 01
80 02 02 58
80 05 00 01
Payload Type= NONCE
Next Payload = ISAKMP_NEXT_ID
Payload Length = 0x14
Nonce = 0xf4 b0 8f 7f f7 34 d3 23 cb a0 8b 81 7c 7a 7b fc
Payload Type = ID
Next Payload = ISAKMP_NEXT_ID
Payload Length = 0x10
ID Type = 0x04
Protocol ID = 0x0
Port = 0x0
ID = 0xac 11 01 00 ff ff ff 00
Payload Type = ID
Next Payload = ISAKMP_NEXT_NONE
Payload Length = 0x10
ID Type = 0x04
Protocol ID = 0x0 Port = 0x0
ID = 0xac 10 02 00 ff ff ff 00
0:00:45.430 <++++IKE Len = 148
I Cookie=0xb9 f0 0c 1a a2 e6 89 db , R Cookie=0x28 04 b5 7f b8 39 77 3d
Next Payload = ISAKMP_NEXT_HASH
Exchange Type = 0x20
Message ID = 0xeca88777
Payload Type = HASH
Next Payload = ISAKMP_NEXT_SA
Payload Length = 0x14
Hash = 0xa9 03 b5 1a f2 21 c6 fe 90 01 87 ab 9a 5d ed 65
Payload Type = SA
Next Payload = ISAKMP_NEXT_NONCE
Payload Length = 0x30
DOI = 0x1
Situation = 0x1
Proposal #0x0, Protocol Id = 0x3, SPI Size = 0x4, Number of Transforms = 0x1
SPI = 31 4b 59 2d
Transform #0x0, Transform ID = 0x2, Length = 0x10
80 04 00 01
80 01 00 01
80 02 02 58
80 05 00 02
Payload Type= NONCE
Next Payload = ISAKMP_NEXT_ID
Payload Length = 0x14
Nonce = 0xc6 a1 8f 87 03 42 62 72 fb c0 a3 15 4e 6b 7a 02
Payload Type = ID
Next Payload = ISAKMP_NEXT_ID
Payload Length = 0x10
ID Type = 0x04
Protocol ID = 0x0
Port = 0x0
ID = 0xac 11 01 00 ff ff ff 00
Payload Type = ID
Next Payload = ISAKMP_NEXT_NONE
Payload Length = 0x10
ID Type = 0x04
Protocol ID = 0x0
Port = 0x0
ID = 0xac 10 02 00 ff ff ff 00
0:00:45.430 ++++>IKE Len = 48
I Cookie=0xb9 f0 0c 1a a2 e6 89 db, R Cookie=0x28 04 b5 7f b8 39 77 3d
Next Payload = ISAKMP_NEXT_HASH
Exchange Type = 0x20
Message ID = 0xeca88777
Payload Type = HASH
Next Payload = ISAKMP_NEXT_NONE
Payload Length = 0x14
Hash = 0x19 2c 30 c1 26 86 83 d0 e0 64 a0 16 de ac 56 11
>

IPSec SA Creation Phases

There are two phases on the IPsec SA creation. Phase 1 is to create IKE-SA, and phase 2 is to create IPSEC-SA. Phase 1 creates a security tunnel to protect phase2. Phase 2 is protected by phase 1.

Phase 1: Create IKE-SA. There are two modes on this phase, the major is main mode, which includes six messages;

  • 1&2: to negotiate the security policy, 1. Initiator sends all type of policies supported to remote end, and if remote end searches any one of them that support too, it will respond to the initiator. The policies include authentication method, PSK or MD5, hash- algorithm, MD5 or SHA, encryption algorithm: DES or 3DES; SA life time (duration) x seconds;
  • 3&4: to exchange the DH and key and create the key
  • 5&6: two messages have been protected by key ID for authentication for each other;

Phase 2: create IPSEC-SA.

  1. negotiate the IPSEC-protocol: ESP or AH; IPSec-mode: tunnel or transport; hash-algorithm: MD5 or SHA;
  2. ACK and ACK too.

Example

An example of an IPSec exchange using NAT-Traversal in Main Mode is shown as below:

Explanation:

1st Log:

0:00:44.840 ++++>IKE Len = 296
I Cookie=0xb9 f0 0c 1a a2 e6 89 db , R Cookie=0x00 00 00 00 00 00 00 00
Next Payload = ISAKMP_NEXT_SA
Exchange Type = 0x2
Message ID = 0x0
Payload Type = SA
Next Payload = ISAKMP_NEXT_VID
Payload Length = 0x94
DOI = 0x1
Situation = 0x1
Proposal #0x0, Protocol Id = 0x1, SPI Size = 0x0, Number of Transforms = 0x4
Transform #0x0, Transform ID = 0x1, Length = 0x18
80 0b 00 01
80 0c 03 84
80 01 00 01
80 02 00 01
80 03 00 01
80 04 00 01
Transform #0x1, Transform ID = 0x1, Length = 0x18
80 0b 00 01
80 0c 03 84
80 01 00 01
80 02 00 02
80 03 00 01
80 04 00 01
Transform #0x2, Transform ID = 0x1, Length = 0x18
80 0b 00 01
80 0c 03 84
80 01 00 05
80 02 00 01
80 03 00 01
80 04 00 01
Transform #0x3, Transform ID = 0x1, Length = 0x18
80 0b 00 01
80 0c 03 84
80 01 00 05
80 02 00 01
80 03 00 01
80 04 00 02
Payload Type= VENDOR ID
Next Payload = ISAKMP_NEXT_VID
Payload Length = 0x14
VID Data = 0xaf ca d7 13 68 a1 f1 c9 6b 86 96 fc 77 57 01 00
Payload Type= VENDOR ID
Next Payload = ISAKMP_NEXT_VID
Payload Length = 0x14
VID Data = 0x4a 13 1c 81 07 03 58 45 5c 57 28 f2 0e 95 45 2f
Payload Type= VENDOR ID
Next Payload = ISAKMP_NEXT_VID
Payload Length = 0x14
VID Data = 0x7d 94 19 a6 53 10 ca 6f 2c 17 9d 92 15 52 9d 56
Payload Type= VENDOR ID
Next Payload = ISAKMP_NEXT_VID
Payload Length = 0x14
VID Data = 0x90 cb 80 91 3e bb 69 6e 08 63 81 b5 ec 42 7b 1f
Payload Type= VENDOR ID
Next Payload = ISAKMP_NEXT_VID
Payload Length = 0x14
VID Data = 0xcd 60 46 43 35 df 21 f8 7c fd b2 fc 68 b6 a4 48
Payload Type= VENDOR ID
Next Payload = ISAKMP_NEXT_NONE
Payload Length = 0x14
VID Data = 0x44 85 15 2d 18 b6 bb cd 0b e8 a8 46 95 79 dd cc

In which, ++++>indicates connection direction is from local to remote
“I Cookie=0xb9 f0 0c 1a a2 e6 89 db , R Cookie=0x00 00 00 00 00 00 00 00”
“R Cookie=0x00 00 00 00 00 00 00 00” indicates it is the first message sent by the initiator.

80 0b 00 01
80 0c 03 84
80 01 00 01
80 02 00 01
80 03 00 01
80 04 00 01

Above is a proposal, which designates the following parameters:

Encryption Algorithm is DES,
Hash Algorithm is MD5,
Authentication Method is Preshared key,
DH Group 1, Lifetime is 900 seconds.

The Vendor ID Payloads indicate the following protocols are supported:

Dead Peer Detection,
NAT-T rfc 3947,
NAT-T draft 03,
NAT-T draft 02,
NAT-T draft 02,
NAT-T draft 00.

Summary: The first log with direction ++++> and R Cookie equal to all “0s” indicates that the router itself is the initiator of the connection. It brings 4 proposals, which is set up in the Advanced window.

2nd Log:

0:00:44.970 <++++IKE Len = 120
I Cookie=0xb9 f0 0c 1a a2 e6 89 db , R Cookie=0x28 04 b5 7f b8 39 77 3d
Next Payload = ISAKMP_NEXT_SA
Exchange Type = 0x2
Message ID = 0x0
Payload Type = SA
Next Payload = ISAKMP_NEXT_VID
Payload Length = 0x34
DOI = 0x1
Situation = 0x1
Proposal #0x0, Protocol Id = 0x1, SPI Size = 0x0, Number of Transforms = 0x1
Transform #0x0, Transform ID = 0x1, Length = 0x18
80 0b 00 01
80 0c 03 84
80 01 00 01
80 02 00 01
80 03 00 01
80 04 00 01
Payload Type= VENDOR ID
Next Payload = ISAKMP_NEXT_VID
Payload Length = 0x14
VID Data = 0xaf ca d7 13 68 a1 f1 c9 6b 86 96 fc 77 57 01 00
Payload Type= VENDOR ID
Next Payload = ISAKMP_NEXT_NONE
Payload Length = 0x14
VID Data = 0x4a 13 1c 81 07 03 58 45 5c 57 28 f2 0e 95 45 2f

In which, <++++ indicates connection direction is from remote to local
“I Cookie=0xb9 f0 0c 1a a2 e6 89 db , R Cookie=0x28 04 b5 7f b8 39 77 3d”
The successive messages in the same IPSec session all use the same I Cookie and R Cookie pair.

80 0b 00 01
80 0c 03 84
80 01 00 01
80 02 00 01
80 03 00 01
80 04 00 01

The initiator sends 4 proposals and the responder accepts one proposal with the following parameters:

Encryption Algorithm is DES,
Hash Algorithm is MD5,
Authentication Method is Pre-shared key,
DH Group 1, Lifetime is 900 seconds.

The Vendor ID Payloads indicate the following protocol is accepted by the responder.

Dead Peer Detection and NAT-T rfc 3947.

Summary: The second log with direction <++++ indicates that the remote VPN gateway has acknowledged one of the proposals proposed by the initiator. If you cannot see the message in the log, it might be:

  1. The responder doesn’t agree with any of the proposals. Please make sure the relevant settings in both sides match with each other.
  2. The responder doesn’t receive the proposals. Please check if the remote gateway is available and

IPSec service is activated or not.

3rd & 4th log:

0:00:45.000 ++++>IKE Len = 188
I Cookie=0xb9 f0 0c 1a a2 e6 89 db , R Cookie=0x28 04 b5 7f b8 39 77 3d
Next Payload = ISAKMP_NEXT_KE
Exchange Type = 0x2
Message ID = 0x0
Payload Type = KEY EX
Next Payload = ISAKMP_NEXT_NONCE
Payload Length = 0x64
Key = 0x30 da 16 b0 e0 50 5f 90 51 7c ce 8e 0c 42 2c 59
73 35 98 83 bd 96 7e b7 29 e1 7d b5 16 e2 73 fe
11 01 44 23 d4 6d 35 78 68 a9 de 89 12 72 4c f3
71 5c a5 3d 2f 18 e3 1c 7e 83 75 02 fa 09 b4 3d
9f 52 05 7d ac d2 2e 70 37 21 54 4c 55 e8 34 04
b8 0c 32 c9 8c 05 9a eb 72 c9 e3 2a 3f 06 96 57
Payload Type= NONCE
Next Payload = ISAKMP_NEXT_NAT-D
Payload Length = 0x14
Nonce = 0x33 42 4a 4e d1 13 b4 05 ae 83 6e 64 60 5e 5f 60
Payload Type= NAT-D
Next Payload = ISAKMP_NEXT_NAT-D
NAT-D Length = 0x14
NAT-D = 0xf5 33 e5 65 ef d4 e8 4e da 2a 88 09 e8 c1 10 cc
Payload Type= NAT-D
Next Payload = ISAKMP_NEXT_NONE
NAT-D Length = 0x14
NAT-D = 0x3f bd 25 13 76 12 81 b9 1e 37 fd a7 a2 41 a7 85
0:00:45.200 <++++IKE Len = 188
I Cookie=0xb9 f0 0c 1a a2 e6 89 db , R Cookie=0x28 04 b5 7f b8 39 77 3d
Next Payload = ISAKMP_NEXT_KE
Exchange Type = 0x2
Message ID = 0x0
Payload Type = KEY EX
Next Payload = ISAKMP_NEXT_NONCE
Payload Length = 0x64
Key = 0x33 cb 5a bf 6b 3b 49 4d 32 af 60 2f 9e 8f 9c 86
f3 b9 ce 55 9e e5 a8 6a 9f 3d 3c 25 d8 2a a7 de
21 df f0 31 aa 6d 22 c5 57 49 b0 4f ba d0 ca 97
98 6f cb d6 74 c6 06 d9 0e ce bc 02 a7 0a fa 49
ad 99 75 32 c5 3f b0 a7 ed ed 4e 9d 19 40 ec 82
23 17 13 69 9e 4b b0 04 64 50 36 d6 82 f9 f9 d9
Payload Type= NONCE
Next Payload = ISAKMP_NEXT_NAT-D
Payload Length = 0x14
Nonce = 0x24 48 5a 64 e9 2c 4e 60 e9 ae 91 03 3d 5a 69 f1
Payload Type= NAT-D
Next Payload = ISAKMP_NEXT_NAT-D
NAT-D Length = 0x14
NAT-D = 0x3f bd 25 13 76 12 81 b9 1e 37 fd a7 a2 41 a7 85
Payload Type= NAT-D
Next Payload = ISAKMP_NEXT_NONE
NAT-D Length = 0x14
NAT-D = 0xf5 33 e5 65 ef d4 e8 4e da 2a 88 09 e8 c1 10 cc

In these two messages, pre-shared key are exchanged and checked. If you cannot see the 4th message, it is probably that the pre-shared keys set in both sides don’t match with each other. The NAT-D payloads are used to detect which VPN gateway is behind a NATed device.

5th & 6th log:

0:00:45.240 ++++>IKE Len = 88
I Cookie=0xb9 f0 0c 1a a2 e6 89 db , R Cookie=0x28 04 b5 7f b8 39 77 3d
Next Payload = ISAKMP_NEXT_ID
Exchange Type = 0x2
Message ID = 0x0
Payload Type = ID
Next Payload = ISAKMP_NEXT_HASH
Payload Length = 0xc
ID Type = 0x01
Protocol ID = 0x0
Port = 0x0
ID = 0xda f2 82 12
Payload Type = HASH
Next Payload = ISAKMP_NEXT_N
Payload Length = 0x14
Hash = 0x9e dc ff 64 f7 26 fa 72 58 0e 8b f0 9c ca 6c 40
Payload Type = NOTIFICATION
Next Payload = ISAKMP_NEXT_NONE
Payload Length = 0x1c
DOI = 0x1
Protocol ID = 0x01, SPI SIZE = 0x10, Message Type = 0x6002
SPI = b9 f0 0c 1a a2 e6 89 db 28 04 b5 7f b8 39 77 3d
Notification Data =
0:00:45.330 <++++IKE Len = 92
I Cookie=0xb9 f0 0c 1a a2 e6 89 db , R Cookie=0x28 04 b5 7f b8 39 77 3d
Next Payload = ISAKMP_NEXT_ID
Exchange Type = 0x2
Message ID = 0x0
Payload Type = ID
Next Payload = ISAKMP_NEXT_HASH
Payload Length = 0xc
ID Type = 0x01
Protocol ID = 0x0
Port = 0x0
ID = 0xdc 80 e6 79
Payload Type = HASH
Next Payload = ISAKMP_NEXT_N
Payload Length = 0x14
Hash = 0x15 97 0f c0 3e 20 eb fa 6a 9f 76 43 82 10 6f f9
Payload Type = NOTIFICATION
Next Payload = ISAKMP_NEXT_NONE
Payload Length = 0x1c
DOI = 0x1
Protocol ID = 0x01, SPI SIZE = 0x10, Message Type = 0x6002
SPI = b9 f0 0c 1a a2 e6 89 db 28 04 b5 7f b8 39 77 3d
Notification Data =

In these two messages, ID payload is exchanged and checked. In main mode, the real WAN IP address of the router itself is set as local ID. If you cannot see the 6th message, it is probably that the IP address is not accepted by remote VPN gateway.
ID = 0xda f2 82 12 (Hex format) 218.242.130.18 (Decimal format)
ID = 0xdc 80 e6 79 (Hex format) 220.128.230.121 (Decimal format)
Upon seeing the 6th message, the ISAKMP SA is successfully created. Next, the connection will proceed to the Quick mode.

7th message:

0:00:45.330 ++++>IKE Len = 172
I Cookie=0xb9 f0 0c 1a a2 e6 89 db , R Cookie=0x28 04 b5 7f b8 39 77 3d
Next Payload = ISAKMP_NEXT_HASH
Exchange Type = 0x20
Message ID = 0xeca88777
Payload Type = HASH
Next Payload = ISAKMP_NEXT_SA
Payload Length = 0x14
Hash = 0x90 fc 3b 5d 7e 7f 8f 5d 34 24 9a 29 ac d9 3b 1c
Payload Type = SA
Next Payload = ISAKMP_NEXT_NONCE
Payload Length = 0x48
DOI = 0x1
Situation = 0x1
Proposal #0x0, Protocol Id = 0x3, SPI Size = 0x4, Number of Transforms = 0x2
SPI = f0 ac 8b 7b
Transform #0x0, Transform ID = 0x2, Length = 0x10
80 04 00 01
80 01 00 01
80 02 02 58
80 05 00 02
Transform #0x1, Transform ID = 0x2, Length = 0x10
80 04 00 01
80 01 00 01
80 02 02 58
80 05 00 01
Payload Type= NONCE
Next Payload = ISAKMP_NEXT_ID
Payload Length = 0x14
Nonce = 0xf4 b0 8f 7f f7 34 d3 23 cb a0 8b 81 7c 7a 7b fc
Payload Type = ID
Next Payload = ISAKMP_NEXT_ID
Payload Length = 0x10
ID Type = 0x04
Protocol ID = 0x0
Port = 0x0
ID = 0xac 11 01 00 ff ff ff 00
Payload Type = ID
Next Payload = ISAKMP_NEXT_NONE
Payload Length = 0x10
ID Type = 0x04
Protocol ID = 0x0
Port = 0x0
ID = 0xac 10 02 00 ff ff ff 00
Transform ID = 0x2

The transform ID stands for the Encryption Algorithm. 0x2 means ESP_DES.

80 04 00 01
80 01 00 01
80 02 02 58
80 05 00 02

Above is one proposal, which designates the following parameters:

Hash Algorithm is SHA1,
Encapsulation Mode is Tunnel,
Lifetime is 600 seconds. \

The setup can be modified in the Advanced window.

 

ID = 0xac 11 01 00 ff ff ff 00 Local Subnet: 172.17.1.0/255.255.255.0
ID = 0xac 10 02 00 ff ff ff 00 Remote Subnet: 172.16.2.0/255.255.255.0
The Local Subnet is defined in the LAN >> General Setup page and 1st IP Address/Subnet field.
The Remote Subnet is defined in the VPN profile. Make sure in Remote Network IP field you enter the network IP address of remote subnet, not a usable IP address within remote subnet.

8th message:

0:00:45.430 <++++IKE Len = 148
I Cookie=0xb9 f0 0c 1a a2 e6 89 db , R Cookie=0x28 04 b5 7f b8 39 77 3d
Next Payload = ISAKMP_NEXT_HASH
Exchange Type = 0x20
Message ID = 0xeca88777
Payload Type = HASH
Next Payload = ISAKMP_NEXT_SA
Payload Length = 0x14
Hash = 0xa9 03 b5 1a f2 21 c6 fe 90 01 87 ab 9a 5d ed 65
Payload Type = SA
Next Payload = ISAKMP_NEXT_NONCE
Payload Length = 0x30
DOI = 0x1
Situation = 0x1
Proposal #0x0, Protocol Id = 0x3, SPI Size = 0x4, Number of Transforms = 0x1
SPI = 31 4b 59 2d
Transform #0x0, Transform ID = 0x2, Length = 0x10
80 04 00 01
80 01 00 01
80 02 02 58
80 05 00 02
Payload Type= NONCE
Next Payload = ISAKMP_NEXT_ID
Payload Length = 0x14
Nonce = 0xc6 a1 8f 87 03 42 62 72 fb c0 a3 15 4e 6b 7a 02
Payload Type = ID
Next Payload = ISAKMP_NEXT_ID
Payload Length = 0x10
ID Type = 0x04
Protocol ID = 0x0
Port = 0x0
ID = 0xac 11 01 00 ff ff ff 00
Payload Type = ID
Next Payload = ISAKMP_NEXT_NONE
Payload Length = 0x10
ID Type = 0x04
Protocol ID = 0x0
Port = 0x0
ID = 0xac 10 02 00 ff ff ff 00

The initiator sends 2 proposals and the responder accepts one proposal with the following parameters:

ESP_DES,
Hash Algorithm is SHA1,
Encapsulation Mode is Tunnel,
Lifetime is 600 seconds.
Also, the responder sends its ID information.

Summary: If you don’t see the 8th message, or you see this message but the information contained in it shows being encrypted, it is probably the relevant parameters set in both routers don’t match with each other. For example, the PFS (Perfect Forward Secret) is enabled in one side and disabled in the other side; local ID or remote ID configuration exceeds the range allowed in the other side.

9th message

0:00:45.430 ++++>IKE Len = 48
I Cookie=0xb9 f0 0c 1a a2 e6 89 db , R Cookie=0x28 04 b5 7f b8 39 77 3d
Next Payload = ISAKMP_NEXT_HASH
Exchange Type = 0x20
Message ID = 0xeca88777
Payload Type = HASH
Next Payload = ISAKMP_NEXT_NONE
Payload Length = 0x14
Hash = 0x19 2c 30 c1 26 86 83 d0 e0 64 a0 16 de ac 56 11

Upon seeing the 9th message, the IPSec SA is successfully created. The IPSec connection is successfully established.

Note: For detailed information, please refer to documents for RFC2409.

Application Notes and References

See also https://www.draytek.com/support/knowledge-base/5292

Open VPN