With cyberattacks and data leakage threats on the rise, strong encryption and secure connections are becoming a necessity. More and more users and companies are turning to VPNs to ensure privacy and protect against interception. Among many solutions, IPSec (Internet Protocol Security) remains one of the most reliable and widespread standards.
IPSec provides end-to-end encryption, authentication and integrity of transmitted data, operating at the network layer. This makes it particularly useful for securing enterprise networks, cloud infrastructures, and mobile connections.
In this article, we’ll take a closer look at the key components and principles of IPSec, its architecture and protocols, and setting up an IPSec VPN using Servercore infrastructure.
IPSec Features
IPSec protects transmitted data using authentication, encryption, and key management mechanisms.
Authentication helps to verify the data source using PSK (Pre-Shared Key) and digital certificates. In PSK both participants exchange a secret key in advance for authentication. Digital certificates provide a higher level of security because they are based on a PKI (Public Key Infrastructure) that performs mutual verification between participants. Also, with strong authentication, IPSec implements protection mechanisms against man-in-the-middle (MITM) attacks, preventing the possibility of data interception and spoofing.
IPSec encryption protects transmitted information from unauthorized access. Symmetric algorithms such as AES provide high speed data encryption. Asymmetric algorithms (RSA) are used in the Internet Key Exchange (IKE) protocol to authenticate parties and protect the symmetric key exchange process, which is used to securely transmit information.
Key management in IPSec plays a key role in maintaining the security of the connection. The IKE protocol creates a secure channel between the participants and customizes the data encryption parameters. To enhance security, IKE uses PFS (Perfect Forward Secrecy), which ensures that compromise of one key does not affect other sessions.
What is an IPSec VPN and How it Works
IPSec VPN is a technology that uses IPSec to create secure tunnels between devices. It can operate in two modes:
- Transport mode – protects only the payload (data within IP packets), leaving the headers unchanged. It is used to encrypt communications between two hosts.
- Tunnel mode – encrypts the entire IP packet by encapsulating it in a new IP header. This mode is used in corporate VPNs and firewalls.
When choosing a technology to secure your network connections, it is important to understand the differences between IPSec VPN and SSL VPN. Both solutions provide secure data transfer, but differ in how they work, their level of protection, and their use cases. IPSec VPN works at the network layer and provides full encryption for all traffic, while SSL VPN protects only application layer communications. The table below summarizes the main differences between these protocols.
| Parameter | IPSec VPN | SSL VPN |
| Layer of operation | Network (L3) | Transport (L4) or application (L7) |
| Encryption | ESP (Encapsulating Security Payload) | TLS (Transport Layer Security) |
| Application | Connects the entire network or device | Works through a browser or client |
| Flexibility | Requires client-side and server-side configuration | Easy to use, works through a web interface |
IPSec VPNs are better for protecting all network traffic, while SSL VPNs are more commonly used for remote access via a browser.
What is IPSec
IPSec is a set of protocols that provide secure data transfer in IP networks. It operates at the network layer of the OSI model, providing authentication, integrity, and encryption of data between hosts. Unlike other security methods, IPSec can encrypt all IP traffic, making it a popular choice for VPN connections.
The first IPSec standard was published in 1995, when it was parallelly developed with IPv6. In 1998, the IETF approved it as a universal protocol for securing network communications. Today, IPSec is widely used in enterprise networks, data centers and cloud environments to protect traffic.
The main purpose of IPSec is to provide security for data transmitted over a network. The protocol suite performs encryption, authenticates senders and ensures the integrity of data, protecting it from modification during transmission. IPSec also protects against the reuse of intercepted packets through the anti-replay mechanism.
How IPSec Works
IPSec provides secure data transmission at the network layer (L3 of the OSI model). Unlike other VPN protocols that operate at the transport layer (L4), IPSec is integrated into the IP protocol, making it more versatile for building secure tunnels between devices, servers, and corporate networks.
How IPSec VPN works
The operation of IPSec is based on two main mechanisms: authentication and traffic encryption. The process is established in several stages:
- Connection initialization – performed through the IKE protocol, which includes creating a secure channel and setting security parameters for IPSec SA (Security Association) data.
- Party Authentication – uses digital signatures, certificates or Pre-Shared Keys (PSK) to authenticate the nodes.
- Data encryption and protection – all network traffic passing through the tunnel is encrypted using symmetric algorithms (AES, 3DES).
- Data integrity control – realized through ESP (encryption and authentication) or AH (authentication and integrity check only).
- Data transmission through a tunnel – packets are transmitted through an encrypted channel, and on the receiver side they are decrypted and checked for integrity.
- Connection closure – after data transfer is completed, the VPN tunnel is closed after the SA lifetime expires or after a certain amount of data has been transferred.
How IPsec Affects MSS and MTU
Using IPSec affects the MSS (Maximum Segment Size) and MTU (Maximum Transmission Unit) parameters because of the additional load associated with encryption and data authentication. When encapsulating packets, IPSec adds ESP or AH headers, which increases the overall packet size. This can cause the MTU size to exceed the allowable MTU size, causing fragmentation and potentially degrading network performance.
To minimize the impact, MSS is often reduced at the TCP layer, allowing for IPSec overhead and avoiding fragmentation. Optimal MTU and MSS tuning helps maintain a stable connection, reducing latency and improving the efficiency of data transmission over an IPSec tunnel.
Benefits of IPSec
IPSec remains one of the most reliable protocols for organizing VPNs due to several key advantages:
- High security – supports modern encryption algorithms (AES-256, ChaCha20), authentication (RSA, ECDSA) and data protection.
- Flexible and scalable – works at the IP level, organizes VPN tunnels between different types of devices and networks, including cloud solutions.
- Protects against MITM attacks and data interception – ensures traffic integrity and protects against data changes during transmission.
- Supports multiple authentication methods – passwords, digital certificates or keys enhance connectivity.
- Integration with other security systems – IPSec can be used in conjunction with firewalls, IDS/IPS and other network defenses.
- Compatible with most operating systems and devices – Windows, Linux, macOS, mobile platforms and routers are supported.
How IPSec Helps Businesses
IPSec protects corporate networks, mobile connections and cloud services. For corporate networks, the protocol suite enables secure connections between offices, remote branch offices and data centers. Tunnel mode encrypts all network traffic, preventing data leaks and attacks. This is especially important for organizations handling sensitive information such as financial data, medical records or intellectual property.
In remote work environments, IPSec provides secure access to corporate resources for employees. When connecting to public Wi-Fi networks, an encrypted connection protects data from interception.
Companies are increasingly deploying their IT infrastructure in the cloud, which requires reliable mechanisms to protect data in transit between cloud services and local networks. IPSec is often used to protect traffic between corporate networks and the cloud (site-to-site), while application protocols (HTTPS, TLS) prevail inside the cloud. This reduces the risk of data leaks and increases the reliability of communication between services.
IPSec Architecture and Protocols
The IPSec architecture includes several layers, each with its own role in securing data transmission.
Layer 1 of the IPSec architecture consists of the transport protocols. The ESP (Encapsulating Security Payload) protocol is responsible for data confidentiality, integrity, and authentication. It uses encryption to protect the transmitted information and supports transport and tunnel modes of operation. The AH (Authentication Header) protocol performs sender authentication and data integrity control. Unlike ESP, it does not encrypt the transmitted data, but protects them from spoofing.
Layer 2 provides authentication and encryption mechanisms. Pre-Shared Key, X.509 digital certificates and Kerberos system are used for authentication. Encryption is performed using symmetric algorithms such as AES and 3DES.
Layer 3 defines the rules of communication between different IPSec implementations through standardized security parameters. This component regulates the use of encryption, authentication, and key management algorithms, ensuring interoperability between different IPSec solutions.
Layer 4 includes IKE (Internet Key Exchange), which plays a key role in establishing secure connections between hosts. It manages the generation and exchange of keys to encrypt traffic, and supports dynamic updating of security settings without interrupting active connections.
Configuring an IPSec VPN with Servercore
In this chapter, we will look at running a Servercore virtual machine with an IPsec VPN to route traffic between a cloud network and a remote connection point using the strongSwan daemon. We’ll also break down a fault-tolerant scheme with two VPN IPsec servers.
Initial data
For IPsec to work properly, we need to make sure that the virtual machine has access to both networks and routes traffic correctly. To establish a connection you will need:
- A VM running on an Ubuntu 20.04 image with two ports.
- The first port of the VM is connected to the public network and is used as the default route.
- The second port is connected to the private network that needs to be routed.
It is important to keep in mind that the firewall and routing rules must be configured to allow seamless communication between the interfaces.
Configuring a single-instance VPN
Connect to the VM via SSH and start the configuration. First you need to install the strongSwan daemon and enable it in the autoloader:
sudo apt update sudo apt install -y strongswan sudo systemctl enable strongswan-starter
Next, we need to enable packet routing between interfaces, since our VM will serve as a gateway to the private network and send traffic over the public network with encryption.
sudo cat << EOF > /etc/sysctl.d/99-vpn-ipsec.conf net.ipv4.ip_forward = 1 net.ipv6.conf.all.forwarding = 1 net.ipv4.conf.all.accept_redirects = 0 net.ipv4.conf.all.send_redirects = 0 EOF sudo sysctl -p /etc/sysctl.d/99-vpn-ipsec.conf
These commands create a sysctl configuration file and apply it. Next, we need to create a basic IPsec configuration file for strongSwan:
sudo cat << EOF > /etc/ipsec.conf
config setup
charondebug="all"
uniqueids=yes
strictcrlpolicy=no
conn site-to-site-vpn
type=tunnel
authby=secret
left=%defaultroute
leftid=188.68.206.154
leftsubnet=192.168.10.0/24
right=51.250.46.166
rightsubnet=192.168.20.0/24
ike=aes256-sha2_256-modp1024!
esp=aes256-sha2_256!
keyingtries=0
ikelifetime=1h
lifetime=8h
dpddelay=30
dpdtimeout=120
dpdaction=restart
auto=start
EOF
Important options that need to be configured depending on your environment and the addresses used:
- leftid – the public address of the IPsec VM itself;
- leftsubnet – the private network created in the Servercore cloud for other VMs. From it you need to have access to VMs in another cloud;
- right – public address of the remote VM, i.e. in our case 51.250.46.166;
- rightsubnet – remote private network, which was created in another cloud and is used there for VMs.
After creating the main configuration file, you need to create a secret file that will contain the password for connection:
sudo cat << EOF > /etc/ipsec.secrets 188.68.206.154 51.250.46.166 : PSK "<SECRET>" EOF
Where <SECRET> is the password specified on both sides of the tunnel, that is, on both VPN IPsec virtual machines.
Reboot the IPsec daemon and check the connection status:
sudo systemctl restart strongswan-starter ipsec status
Configuring a fault tolerant scheme
Unlike the previous scheme, VIP addresses will be used as the gateway IP address on the private network and as the primary address for the VPN tunnel. These will be moved between VMs using VRRP.
To configure a fault tolerant scheme, we need to start another VM like this one for the VPN. We will call it vpn2 and the first VM will be called vpn1.
The second VM then reproduces the same basic settings as the first:
sudo apt update sudo apt install -y strongswan sudo systemctl enable strongswan-starter sudo cat << EOF > /etc/sysctl.d/99-vpn-ipsec.conf net.ipv4.ip_forward = 1 net.ipv6.conf.all.forwarding = 1 net.ipv4.conf.all.accept_redirects = 0 net.ipv4.conf.all.send_redirects = 0 EOF sudo sysctl -p /etc/sysctl.d/99-vpn-ipsec.conf
Before IPsec, you must configure VRRP using the keepalived daemon. Install the service:
sudo apt install -y keepalived
Set up the VRRP configuration so that both VIPs move at the same time and are in the same group. When changing the status of an instance from MASTER to BACKUP and back again, it is necessary that IPsec is turned on and off accordingly, as the tunnel should only be built from one VM at a time.
Configuring VRRP on the vpn2 virtual machine:
sudo cat << EOF > /usr/local/sbin/notify-ipsec.sh
#!/bin/bash
TYPE=\$1
NAME=\$2
STATE=\$3
case \$STATE in
"MASTER") /usr/sbin/ipsec restart
;;
"BACKUP") /usr/sbin/ipsec stop
;;
"FAULT") /usr/sbin/ipsec stop
exit 0
;;
*) /usr/bin/logger "ipsec unknown state"
exit 1
;;
esac
EOF
sudo chmod a+x /usr/local/sbin/notify-ipsec.sh
sudo cat << EOF > /etc/keepalived/keepalived.conf
vrrp_sync_group G1 {
group {
EXT
INT
}
notify "/usr/local/sbin/notify-ipsec.sh"
}
vrrp_instance INT {
state BACKUP
interface eth1
virtual_router_id 11
priority 25
advert_int 1
authentication {
auth_type PASS
auth_pass <SECRET>
}
virtual_ipaddress {
192.168.120.100/24
}
garp_master_delay 1
}
vrrp_instance EXT {
state BACKUP
interface eth0
virtual_router_id 22
priority 25
advert_int 1
authentication {
auth_type PASS
auth_pass <SECRET>
}
virtual_ipaddress {
188.68.206.158/29
}
garp_master_delay 1
}
EOF
These commands create two files:
- /usr/local/sbin/notify-ipsec.sh – script that will be called when VRRP status changes and will disable/enable IPsec;
- /etc/keepalived/keepalived.conf – keepalived configuration file for the two groups.
Next, you need to restart keepalived and enable it in autorun:
sudo systemctl restart keepalived sudo systemctl enable keepalived
You can then proceed to configure IPsec on vpn2. The configuration file will now be slightly different, as the tunnel must be built from the VIP address instead of the address on the interface.
sudo cat << EOF > /etc/ipsec.conf
config setup
charondebug="all"
uniqueids=yes
strictcrlpolicy=no
conn site-to-site-vpn
type=tunnel
authby=secret
left=%defaultroute
leftid=188.68.206.158
leftsubnet=192.168.10.0/24
right=51.250.46.166
rightsubnet=192.168.20.0/24
ike=aes256-sha2_256-modp1024!
esp=aes256-sha2_256!
keyingtries=0
ikelifetime=1h
lifetime=8h
dpddelay=30
dpdtimeout=120
dpdaction=restart
auto=start
EOF
The secret file is also created with a VIP address:
sudo cat << EOF > /etc/ipsec.secrets 188.68.206.158 51.250.46.166 : PSK "<SECRET>" EOF
Configuration of vpn2 is over. Let’s start configuring vpn1, since the first VM also needs to run VRRP and tweak the IPsec configuration.
Since this scheme is built for fault tolerance, most of the configuration files will be exactly the same as on vpn2. The exception is that the VRRP role of this instance will be MASTER. Install keepalived, create configuration and IPsec control script:
sudo apt install -y keepalived
sudo cat << EOF > /usr/local/sbin/notify-ipsec.sh
#!/bin/bash
TYPE=\$1
NAME=\$2
STATE=\$3
case \$STATE in
"MASTER") /usr/sbin/ipsec restart
;;
"BACKUP") /usr/sbin/ipsec stop
;;
"FAULT") /usr/sbin/ipsec stop
exit 0
;;
*) /usr/bin/logger "ipsec unknown state"
exit 1
;;
esac
EOF
sudo chmod a+x /usr/local/sbin/notify-ipsec.sh
sudo cat << EOF > /etc/keepalived/keepalived.conf
vrrp_sync_group G1 {
group {
EXT
INT
}
notify "/usr/local/sbin/notify-ipsec.sh"
}
vrrp_instance INT {
state MASTER
interface eth1
virtual_router_id 11
priority 50
advert_int 1
authentication {
auth_type PASS
auth_pass <SECRET>
}
virtual_ipaddress {
192.168.120.100/24
}
garp_master_delay 1
}
vrrp_instance EXT {
state MASTER
interface eth0
virtual_router_id 22
priority 50
advert_int 1
authentication {
auth_type PASS
auth_pass <SECRET>
}
virtual_ipaddress {
188.68.206.158/29
}
garp_master_delay 1
}
EOF
sudo systemctl enable keepalived
The password <SECRET>, specified in the configuration, must be the same on both VPN VMs.
Next, you need to update the secret and IPsec configuration so that the primary address is replaced by the VIP address. That is, the IPsec and secret configurations will be identical to vpn1 and vpn2.
sudo cat << EOF > /etc/ipsec.conf
config setup
charondebug="all"
uniqueids=yes
strictcrlpolicy=no
conn site-to-site-vpn
type=tunnel
authby=secret
left=%defaultroute
leftid=188.68.206.158
leftsubnet=192.168.10.0/24
right=51.250.46.166
rightsubnet=192.168.20.0/24
ike=aes256-sha2_256-modp1024!
esp=aes256-sha2_256!
keyingtries=0
ikelifetime=1h
lifetime=8h
dpddelay=30
dpdtimeout=120
dpdaction=restart
auto=start
EOF
sudo cat << EOF > /etc/ipsec.secrets
188.68.206.158 51.250.46.166 : PSK "<SECRET>"
EOF
Reboot keepalived and check the IPsec status:
sudo systemctl restart keepalived ipsec status
Conclusion
IPsec has long been recognized as one of the most reliable tools for securing data in transit. It is widely used in enterprise and cloud networks, providing secure connectivity between hosts. Modern implementations of IPsec allow.n customize the configuration for different scenarios – from remote access of employees to the protection of interconnectivity.
Servercore offers optimized server solutions that allow you to deploy IPsec VPN with high performance and configuration flexibility. With support for modern security standards, these solutions provide strong data encryption and secure connections even in scalable cloud environments.
