Protocol CARP
Introduction and history
« CARP stands for Common Address Redundancy Protocol and its basic functionality is to allow multiple hosts to share a set of IP addresses ».
CARP was developed by the OpenBSD project in 2003 as an alternative to VRRP due to patent concerns related to Cisco's HSRP technology. To avoid potential legal issues and remain compatible with open-source principles, OpenBSD implemented its own redundancy protocol with added cryptographic features. CARP was later adopted by other systems such as FreeBSD and NetBSD. A userland implementation of CARP, named ucarp, made the protocol usable on Linux without requiring kernel integration.
These patent concerns originated from the development of VRRP by the Internet Engineering Task Force (IETF). Indeed, when the Internet Engineering Task Force (IETF) developed the Virtual Router Redundancy Protocol (VRRP), Cisco Systems indicated that it owned patents related to router redundancy mechanisms used in the protocol Hot Standby Router Protocol (HSRP). This created a legal risk for developers, particularly for open-source projects that wanted to implement VRRP freely.
Sources :
https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol
https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf
Master host and backup hosts
CARP creates a redundancy group, meaning several hosts are configured to share a virtual IP address. However, at any given time, only one host uses the shared IP address. This host is called the Master host and it receives and handles all traffic destined to that virtual IP.
When the Master host becomes unavailable (it crashed, turned off, or lost its network connection), the other hosts in the same redundancy group detect the failure. Immediately, one of the Backup hosts is elected as the new Master host. That means that it will take over the shared IP address.
This switch happens automatically, ensuring service continuity without clients noticing any interruption.
Sources :
https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf
https://wxcafe.net/posts/redondance-routeurs-openbsd-freebsd/
CARP advertisements
- The Master host periodically broadcasts CARP advertisements to the backup hosts.
- The Backup hosts listen but do not send any packets.
Each CARP advertisement contains :
1. The CARP version and packet type (CARP Header).
2. The VHID (Virtual Host ID), which identifies the redundancy group (CARP Header).
3. The advertisement parameters (advbase and advskew), which determine the host’s priority CARP (CARP Header).
4. The counter
5. The signature in SHA-1
CARP Advertisement Packet Format :
Each line represent 32 bits, 4 bytes
Length : 36 bytes
/* * The CARP header layout is as follows: * * 0 1 2 3 * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * |Version| Type | VirtualHostID | AdvSkew | Auth Len | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * | Demotion | AdvBase | Checksum | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * | Counter (1) | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * | Counter (2) | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * | SHA-1 HMAC (1) | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * | SHA-1 HMAC (2) | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * | SHA-1 HMAC (3) | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * | SHA-1 HMAC (4) | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * | SHA-1 HMAC (5) | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * */
commande : tcpdump -i <interface> proto 112
Sources :
https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf
https://github.com/openbsd/src/blob/master/sys/netinet/ip_carp.h
Cryptography
All CARP advertisements are cryptographically signed :
Steps :
1. The Master host creates an advertisement.
2. It calculates a cryptographic signature, a SHA-1 hash using:
- the shared secret,
- the VHID,
- the virtual IP address,
- the advertisement parameters (advbase and advskew),
- the counter
- and other header fields.
3. It then sends the signed advertisement.
4. The Backup hosts receives the signed advertisement.
5. Each Backup host recalculates the signature using the same shared secret :
- If the calculated signature matches the received signature → the packet is considered valid because authentication and integrity are insured.
- If it does not match → the packet is rejected.
It protects against spoofing attacks : without cryptographic authentication, an attacker could send fake CARP advertisements and attempt to pretend to be or to become the Master host.
command : tcpdump -X -ni <interface> proto 112
We can observe the complete packet in the capture. Each hexadecimal value represents two bytes. The first 20 bytes correspond to the IP header, while the last 20 bytes correspond to the SHA-1 authentication hash included in the CARP advertisement.
Sources :
https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf
https://www.giac.org/paper/gsec/4031/carp-free-fail-over-protocol/106433
IP protocol number
CARP uses the same IP protocol number as VRRP (112). As a result, packet capture tools such as tcpdump interpret CARP packets as VRRP advertisements like we see above, even though the packets actually belong to the CARP protocol.
Why ?? The OpenBSD developers initially requested unique identifiers from the Internet Assigned Numbers Authority (IANA), but the request was denied because they had not followed the IETF standardization process. As a result, they chose to use protocol number 112, which was already assigned to VRRP. Despite this overlap, VRRP and CARP can coexist on the same network as long as their identifiers (VRRP group ID and CARP VHID) are different.
Sources :
https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol
Multiple VHID : Distribute traffic
Using multiple VHIDs in CARP allows several virtual IP addresses to be managed separately. This means different hosts can be Master for different services. For example, one host can manage the web service, while another manages the DNS service. Thanks to this, the hosts can be used efficiently while still ensuring high availability. It would not be practical to use a machine only when the other machine is not responding.
sources :
https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf
Split-brain
Normally:
- one Master
- one or several Backups
If the Master host stops sending advertisements, the Backup hosts know that the master is no longer reachable. However, a problem can occur if the Backup hosts stop receiving each other's advertisements because the link between them is broken. In that case, each node may assume it is the Master.
When communication is restored, CARP resolves the problem:
1. The Masters can see each other 2. The real Master send the packet 3. The hosts compare CARP priority 4. The hosts with lower priority automatically switch back to Backup
sources :
https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf
Master Election Process
1. All hosts start
All hosts begin in BACKUP state.
2. Each host starts a timer
- The delay depends on the CARP priority (advskew).
- The lower the advskew, the shorter the delay.
- The lower the advskew, the higher the priority.
3. The higher-priority host’s timer expires first
That host sends CARP advertisements when the timer expires.
It assigns the virtual IP (VIP) to its interface.
It becomes the MASTER host.
4. The other hosts receive the advertisement
They detect that a higher-priority host is active.
They remain in BACKUP state.
They do not send CARP advertisements.
In CARP, we usually choose the preferred Master by giving it a higher priority (lower advskew).
Virtual Mac Address
The protocol automatically generates a virtual MAC address based on the VHID, ending with the VHID's value (00:00:5e:00:01:XX). This MAC address is identical on both hosts and is used by the MASTER node.
We can see here that the virtual mac address associated to the vhid2 is : 00:00:5e:00:01:02 (02 at the end for vhid2)
command : tcpdump -e -ni vmx1 proto 112
✔ The CARP virtual MAC address is visible.
❌ The virtual IP address (VIP) is not visible as the source.
In CARP advertisements, the Ethernet source address corresponds to the virtual CARP MAC address derived from the VHID. However, the IP source address is the physical IP address of the router sending the advertisement. The virtual IP address (VIP) is not used as the source IP in these control packets. Instead, the VIP is used for normal traffic between clients and the virtual gateway.
sources :
https://dynfi.com/documentations/dynfi-firewall/configuration_firewall_VIP.html
Address Multicast CARP
In the Common Address Redundancy Protocol (CARP), routers communicate with each other using a multicast address. CARP advertisements are sent to the IPv4 multicast address 224.0.0.18, which allows all routers participating in the CARP group to receive the messages simultaneously. At the Ethernet level, this multicast IP address corresponds to the multicast MAC address 01:00:5e:00:00:12.
Priority vs advskew.
In the CARP protocol, the Master router is not directly determined by priority, but rather by a value called advskew. Advskew is a numerical parameter between 0 and 255: the lower its value, the more likely the router is to become the Master. Priority, on the other hand, is a more intuitive abstraction used in automation tools or logical configurations: the higher the priority, the higher the router's expected priority. To link the two, we use the formula: advskew = 255 − priority. Thus, a router with a priority of 200 will have a low advskew (55) and will become the Master, while a router with a priority of 0 will have a maximum advskew (255) and will remain in Backup mode. In short, priority is a value that is easily understood and logical for humans, while advskew is the value actually used by CARP's internal mechanism to determine the Master or Backup role. However, advskew can also be called "CARP priority".
Basic configuration
We want :
router-002 to act as the master, while router-003 should operate as the backup for Intranought network.
router-003 to act as the master, while router-002 should operate as the backup for Public network.
1) Enable CARP Support
On the file /boot/loader.conf, you need to add : carp_load="YES"
You are telling FreeBSD:
“Load the CARP kernel module automatically at boot time.”
If you want to load it now, but not permanently : kldload carp
If someone has build a custom FreeBSD kernel, they can include CARP directly by adding this line to the kernel configuration file: device carp
2) CARP configuration
In each router, on the new appropriate file /etc/rc.conf.d/netif/carp, we will have the CARP configuration
router-002 :
ifconfig_vmx0_alias0="inet vhid 1 advskew 0 pass xxxxxxx alias 172.27.27.13/27"
ifconfig_vmx1_alias0="inet vhid 2 advskew 100 pass xxxxxxx alias 51.68.252.230/32"
router-003 :
ifconfig_vmx0_alias0="inet vhid 1 advskew 100 pass xxxxxxx alias 172.27.27.13/27"
ifconfig_vmx1_alias0="inet vhid 2 advskew 0 pass xxxxxxx alias 51.68.252.230/32"
So we gave a lower advskew (default is 0) to become the master router.
In older versions of FreeBSD (9 and earlier), CARP required creating a separate virtual interface such as carp0, and the shared IP address was attached to that interface. Starting with FreeBSD 10, this was simplified. CARP can now be configured directly on the physical interface using an alias with a VHID. The newer method is easier to configure and maintain, so it is recommended to use it in modern versions of FreeBSD.
commands :
ifconfig carp0 create ifconfig carp0 vhid 1 pass xxxxxx ifconfig carp0 inet 172.27.27.X/28
sources :
Automatic blue/green deployment
In this context, “automatic Blue-Green” does not refer to a traditional application deployment strategy. Instead, it describes a network-level behavior enabled by CARP.
With CARP, two routers are deployed in parallel:
- One acts as the active router (MASTER)
- The other remains in standby (BACKUP)
Both routers are fully operational and connected to the network. Traffic is sent to a shared virtual IP address. At any given time, only the MASTER processes the traffic. However, if the MASTER fails, the BACKUP automatically takes over the virtual IP address without manual intervention.
This behavior resembles Blue-Green deployment because:
1. Two environments exist simultaneously
2. Only one actively handles traffic
3. Traffic can switch from one to the other
4. The switch is transparent to users
The difference is that in traditional Blue-Green deployment, the switch is usually performed manually during an update. With CARP, the switch happens automatically in case of failure.
ESXI and CARP
When using Common Address Redundancy Protocol in a virtualized environment such as VMware ESXi, specific security settings must be adjusted on the virtual switch. CARP relies on the use of a virtual MAC address that can move between routers depending on which device is currently the Master. However, ESXi applies strict security policies by default to prevent virtual machines from sending or receiving traffic with unexpected MAC addresses. To ensure that CARP functions correctly, three settings must be enabled on the vSwitch or port group.
First, Promiscuous Mode must be enabled. This allows a virtual machine to receive all frames circulating on the virtual network, even if the destination MAC address does not match the interface’s MAC address. This is important for CARP because routers must receive multicast advertisements sent to the CARP group.
Second, MAC Address Changes must be allowed. CARP dynamically assigns a virtual MAC address to the Master router. When a failover occurs, the Backup router must adopt this same virtual MAC address. If ESXi blocks MAC address changes, the router will not be able to assume the virtual MAC address and the failover mechanism will fail.
Finally, Forged Transmits must also be permitted. This option allows a virtual machine to send frames with a source MAC address different from the one originally assigned to the virtual network interface. Since CARP advertisements and ARP responses are sent using the virtual MAC address rather than the physical interface MAC address, this setting is required to allow those frames to be transmitted correctly.
Together, these three settings ensure that CARP redundancy mechanisms operate properly in a virtualized ESXi network environment.
Configuration to enable "Promiscuous Mode", “MAC Address Changes” and “Forged Transmits” :
https://knowledge.broadcom.com/external/article/324520/configuring-promiscuous-mode-on-a-virtua.html
Sources :
Linked to T2227
