Protocol CARP: Difference between revisions
No edit summary |
|||
| Line 141: | Line 141: | ||
https://www.giac.org/paper/gsec/4031/carp-free-fail-over-protocol/106433 | https://www.giac.org/paper/gsec/4031/carp-free-fail-over-protocol/106433 | ||
== IP protocol number == | |||
CARP uses the same IP protocol number as VRRP (112). As a result, packet capture tools such as tcpdump interpret CARP packets as VRRP advertisements like we see above, even though the packets actually belong to the CARP protocol. | |||
== Multiple VHID == | == Multiple VHID == | ||
Revision as of 16:03, 9 March 2026
Introduction and history
« CARP stands for Common Address Redundancy Protocol and its basic functionality is to allow multiple hosts to share a set of IP addresses ».
CARP was developed by the OpenBSD project in 2003 as an alternative to VRRP due to patent concerns related to Cisco's HSRP technology. To avoid potential legal issues and remain compatible with open-source principles, OpenBSD implemented its own redundancy protocol with added cryptographic features. CARP was later adopted by other systems such as FreeBSD and NetBSD. A userland implementation of CARP, named ucarp, made the protocol usable on Linux without requiring kernel integration.
These patent concerns originated from the development of VRRP by the Internet Engineering Task Force (IETF). Indeed, when the Internet Engineering Task Force (IETF) developed the Virtual Router Redundancy Protocol (VRRP), Cisco Systems indicated that it owned patents related to router redundancy mechanisms used in the protocol Hot Standby Router Protocol (HSRP). This created a legal risk for developers, particularly for open-source projects that wanted to implement VRRP freely.
Sources :
https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol
https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf
Master host and backup hosts
CARP creates a redundancy group, meaning several hosts are configured to share a virtual IP address. However, at any given time, only one host uses the shared IP address. This host is called the master host and it receives and handles all traffic destined to that virtual IP.
When the master host becomes unavailable (it crashed, turned off, or lost its network connection), the other hosts in the same redundancy group detect the failure. Immediately, one of the backup hosts is elected as the new master host. That means that it will take over the shared IP address.
This switch happens automatically, ensuring service continuity without clients noticing any interruption.
Sources :
https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf
https://wxcafe.net/posts/redondance-routeurs-openbsd-freebsd/
CARP advertisements
- The master host periodically broadcasts CARP advertisements to the backup hosts.
- Backup hosts listen but do not send any packets.
Each CARP advertisement contains :
1. The CARP version and packet type (CARP Header).
2. The VHID (Virtual Host ID), which identifies the redundancy group (CARP Header).
3. The advertisement parameters (advbase and advskew), which determine the host’s priority CARP (CARP Header).
4. The counter
5. The signature in SHA-1
CARP Advertisement Packet Format :
Each line represent 32 bits, 4 bytes
Length : 36 bytes
/* * The CARP header layout is as follows: * * 0 1 2 3 * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * |Version| Type | VirtualHostID | AdvSkew | Auth Len | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * | Demotion | AdvBase | Checksum | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * | Counter (1) | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * | Counter (2) | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * | SHA-1 HMAC (1) | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * | SHA-1 HMAC (2) | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * | SHA-1 HMAC (3) | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * | SHA-1 HMAC (4) | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * | SHA-1 HMAC (5) | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * */
commande : tcpdump -i <interface> proto 112
Sources :
https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf
https://github.com/openbsd/src/blob/master/sys/netinet/ip_carp.h
Cryptography
All CARP advertisements are cryptographically signed :
Steps :
1. The master host creates an advertisement.
2. It calculates a cryptographic signature, a SHA-1 hash using:
- the shared secret,
- the VHID,
- the virtual IP address,
- the advertisement parameters (advbase and advskew),
- the counter
- and other header fields.
3. It then sends the signed advertisement.
4. The backup hosts receives the signed advertisement.
5. Each backup host recalculates the signature using the same shared secret :
- If the calculated signature matches the received signature → the packet is considered valid because authentication and integrity are insured.
- If it does not match → the packet is rejected.
It protects against spoofing attacks : without cryptographic authentication, an attacker could send fake CARP advertisements and attempt to pretend to be or to become the master host.
command : tcpdump -X -ni <interface> proto 112
We can observe the complete packet in the capture. Each hexadecimal value represents two bytes. The first 20 bytes correspond to the IP header, while the last 20 bytes correspond to the SHA-1 authentication hash included in the CARP advertisement.
Sources :
https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf
https://www.giac.org/paper/gsec/4031/carp-free-fail-over-protocol/106433
IP protocol number
CARP uses the same IP protocol number as VRRP (112). As a result, packet capture tools such as tcpdump interpret CARP packets as VRRP advertisements like we see above, even though the packets actually belong to the CARP protocol.
Multiple VHID
Using multiple VHIDs in CARP allows several virtual IP addresses to be managed separately. This means different hosts can be master for different services. For example, one host can manage the web service, while another manages the DNS service. Thanks to this, the hosts can be used efficiently while still ensuring high availability. It would not be practical to use a machine only when the other machine is not responding.
Split-brain
Normally:
- master
- backups
If the master host stops sending advertisements, the backup hosts know that the master is no longer reachable. However, a problem can occur if the backup hosts stop receiving each other's advertisements because the link between them is broken. In that case, each node may assume it is the master.
When communication is restored, CARP resolves the problem:
1. The masters can see each other 2. They receive advertisements from each other 3. They compare priority 4. The hosts with lower priority automatically switche back to backup
Master Election Process
1. All hosts start
- All hosts begin in BACKUP state.
2. Each host starts a timer
- The delay depends on the priority (advskew).
- The lower the advskew, the shorter the delay.
- The lower the advskew, the higher the priority.
3. The higher-priority host’s timer expires first
- That host sends CARP advertisements when the timer expires.
- It assigns the virtual IP (VIP) to its interface.
- It becomes the MASTER host.
4. The other hosts receive the advertisement
- They detect that a higher-priority host is active.
- They remain in BACKUP state.
- They do not send CARP advertisements.
In CARP, we usually choose the preferred master by giving it a higher priority (lower advskew). If the master fails and two backup hosts have the same priority, they may both try to become master at the same time. This can temporarily create a split-brain situation, where both hosts start sending CARP advertisements. However, they will quickly detect each other’s advertisements. Because of small timing differences in the network, one host will effectively win and remain master, while the other will switch stay in backup mode. To avoid this situation, it is recommended to configure different priorities for each host.
Basic configuration
In this example, we consider a topology with two routers: router-002 and router-003. We want router-002 to act as the master, while router-003 should operate as the backup.
1) Enable CARP Support
On the file /boot/loader.conf, you need to add : carp_load="YES"
You are telling FreeBSD:
“Load the CARP kernel module automatically at boot time.”
If you want to load it now, but not permanently : kldload carp
If someone has build a custom FreeBSD kernel, they can include CARP directly by adding this line to the kernel configuration file: device carp
2) Hosts configuration
The hostname, management IP address, subnet mask, shared virtual IP address, and VHID are configured by adding the appropriate entries to the /etc/rc.conf file.
router-002 :
hostname="router-002"
ifconfig_vmx0="inet 172.27.27.11 netmask 255.255.255.240"
ifconfig_vmx0_alias0="inet vhid 1 pass 123piano2026% alias 172.27.27.X/28"
router-003 :
hostname="router-003"
ifconfig_vmx0="inet 172.27.27.12 netmask 255.255.255.240"
ifconfig_vmx0_alias0="inet vhid 1 pass 123piano2026% advskew 100 alias 172.27.27.X/28"
So we gave a lower advskew to the router-002 (default is 0) to become the master router.
In older versions of FreeBSD (9 and earlier), CARP required creating a separate virtual interface such as carp0, and the shared IP address was attached to that interface. Starting with FreeBSD 10, this was simplified. CARP can now be configured directly on the physical interface using an alias with a VHID. The newer method is easier to configure and maintain, so it is recommended to use it in modern versions of FreeBSD.
commands :
ifconfig carp0 create
ifconfig carp0 vhid 1 pass 123piano2026%
ifconfig carp0 inet 172.27.27.X/28
Automatic blue/green deployment
In this context, “automatic Blue-Green” does not refer to a traditional application deployment strategy. Instead, it describes a network-level behavior enabled by CARP.
With CARP, two routers are deployed in parallel:
- One acts as the active router (MASTER)
- The other remains in standby (BACKUP)
Both routers are fully operational and connected to the network. Traffic is sent to a shared virtual IP address. At any given time, only the MASTER processes the traffic. However, if the MASTER fails, the BACKUP automatically takes over the virtual IP address without manual intervention.
This behavior resembles Blue-Green deployment because:
1. Two environments exist simultaneously
2. Only one actively handles traffic
3. Traffic can switch from one to the other
4. The switch is transparent to users
The difference is that in traditional Blue-Green deployment, the switch is usually performed manually during an update. With CARP, the switch happens automatically in case of failure.
Priority vs advskew.
In the CARP protocol, the master router is not directly determined by priority, but rather by a value called advskew. Advskew is a numerical parameter between 0 and 255: the lower its value, the more likely the router is to become the master. Priority, on the other hand, is a more intuitive abstraction used in automation tools or logical configurations: the higher the priority, the higher the router's expected priority. To link the two, we use the formula: advskew = 255 − priority. Thus, a router with a priority of 200 will have a low advskew (55) and will become the master, while a router with a priority of 0 will have a maximum advskew (255) and will remain in backup mode. In short, priority is a value that is easily understood and logical for humans, while advskew is the value actually used by CARP's internal mechanism to determine the master or backup role.
Virtual Mac Address
The protocol automatically generates a virtual MAC address based on the VHID, ending with the VHID's value. This MAC address is identical on both hosts and is used by the MASTER node to advertise the VIP on the network.
Linked to T2227
Distribute traffic
The current infrastructure relies on two distinct networks at the ESXI hypervisor level:
- An internal network 172.27.27.0/28, referred to as IntraNought, used for internal communication and management.
- A public network used for public IP addresses exposed to the Internet.
The objective now is to evaluate the possibility of implementing an active/active model so that both routers would receive traffic during normal operation, while still maintaining high availability through CARP.
On the internal network (IntraNought), configuring CARP does not introduce a significant complexity. One router can operate as master and the other as backup. We will configure a first VHID.
However, on the public network, a constraint exists. Public IP addresses (IPFO) provided by OVH are associated with a specific MAC address. If both routers need to use the same public IP address, they must present the same MAC address.
By default, the ESXi vSwitch blocks situations where two virtual machines use the same MAC address. This requires enabling specific settings such as Promiscuous Mode, MAC Address Changes and Forged Transmits. It is technically feasible, but this adds complexity and may require network isolation by using a dedicated vSwitch for the routers.
Finally, even if both routers are active, they run on the same hypervisor. This means the hypervisor is a Single Point of Failure (SPOF). If the physical host fails, both routers would go down at the same time, so the redundancy only exists at the virtual level, not at the hardware level.
But at this point, a key question remains: If two VMs share the same MAC address, how does the vSwitch decide which VM should receive the traffic? Could it send the frame to the wrong VM? Or both ?
In our setup, two routers share the same virtual public IP and therefore the same virtual MAC address using CARP. However, only one router is active (the master) at a time.
When the master sends traffic, the ESXi vSwitch learns the virtual MAC address from the source MAC of the frames it receives. It then associates that MAC address with the master’s port. As a result, any incoming traffic destined for that MAC is forwarded only to the active router.
The backup router has the same virtual MAC configured, but it does not actively use it while in backup state. It does not respond to ARP requests and does not send traffic using that MAC. Therefore, the vSwitch does not associate the MAC with the backup’s port.
If a failover occurs, the backup router becomes master and starts sending traffic with the same virtual MAC. The vSwitch detects that the MAC now appears on a different port and updates its table. Traffic is then forwarded to the new master.
Security settings such as “MAC Address Changes” and “Forged Transmits” must be set to Accept to allow this MAC movement between ports and prevent ESXi from blocking the traffic as a potential MAC spoofing attempt.
Regarding Promiscuous Mode, VMware documentation states: “Promiscuous mode eliminates any reception filtering that the virtual machine adapter performs so that the guest operating system receives all traffic observed on the wire.” This means that the filtering normally performed by the virtual network adapter based on the destination MAC address is disabled. As a result, the guest operating system can receive all traffic visible on that VLAN. However, this setting does not modify the forwarding logic of the vSwitch itself. The vSwitch continues to perform standard Layer 2 MAC learning and forwards unicast traffic to a single port at a time. Therefore, even with Promiscuous Mode enabled, the vSwitch does not behave like a hub. Promiscuous mode is mainly used for traffic monitoring or packet sniffing, it is not required in our case.
Also, VMware documentation explicitly mentions legitimate scenarios where multiple adapters share the same MAC address, such as Microsoft NLB in unicast mode. This confirms that using a shared virtual MAC with CARP is a supported and valid design pattern.
Configuration to enable “MAC Address Changes” and “Forged Transmits” :
https://knowledge.broadcom.com/external/article/324520/configuring-promiscuous-mode-on-a-virtua.html Source documentation:
https://wxcafe.net/posts/redondance-routeurs-openbsd-freebsd/
https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf
https://dynfi.com/documentations/dynfi-firewall/configuration_firewall_VIP.html
