Protocol CARP: Difference between revisions

From Nasqueron Agora
← Older edit
Yousra (talk | contribs)
142 edits
Yousra (talk | contribs)
142 edits
No edit summary
Yousra (talk | contribs)
142 edits
No edit summary
 
(41 intermediate revisions by the same user not shown)
Line 15: Line 15:
== Master host and backup hosts ==
== Master host and backup hosts ==


CARP creates a redundancy group, meaning several hosts are configured to share a virtual IP address. However, at any given time, only one host uses the shared IP address. This host is called the master host and it receives and handles all traffic destined to that virtual IP.
CARP creates a redundancy group, meaning several hosts are configured to share a virtual IP address. However, at any given time, only one host uses the shared IP address. This host is called the Master host and it receives and handles all traffic destined to that virtual IP.


When the master host becomes unavailable (it crashed, turned off, or lost its network connection), the other hosts in the same redundancy group detect the failure. Immediately, one of the backup hosts is elected as the new master host. That means that it will take over the shared IP address.
When the Master host becomes unavailable (it crashed, turned off, or lost its network connection), the other hosts in the same redundancy group detect the failure. Immediately, one of the Backup hosts is elected as the new Master host. That means that it will take over the shared IP address.


This switch happens automatically, ensuring service continuity without clients noticing any interruption.
This switch happens automatically, ensuring service continuity without clients noticing any interruption.
Line 31: Line 31:
== CARP advertisements ==
== CARP advertisements ==


- The master host periodically broadcasts CARP advertisements to the backup hosts.
- The Master host periodically broadcasts CARP advertisements to the backup hosts.


- Backup hosts listen but do not send any packets.
- The Backup hosts listen but do not send any packets.




Line 101: Line 101:
Steps :  
Steps :  


1. The master host creates an advertisement.
1. The Master host creates an advertisement.


2. It calculates a cryptographic signature, a SHA-1 hash using:
2. It calculates a cryptographic signature, a SHA-1 hash using:
Line 119: Line 119:
3. It then sends the signed advertisement.
3. It then sends the signed advertisement.


4. The backup hosts receives the signed advertisement.
4. The Backup hosts receives the signed advertisement.


5. Each backup host recalculates the signature using the same shared secret :  
5. Each Backup host recalculates the signature using the same shared secret :  


  - If the calculated signature matches the received signature → the packet is considered valid because authentication and integrity are insured.
  - If the calculated signature matches the received signature → the packet is considered valid because authentication and integrity are insured.
Line 127: Line 127:
  - If it does not match → the packet is rejected.
  - If it does not match → the packet is rejected.


It protects against spoofing attacks : without cryptographic authentication, an attacker could send fake CARP advertisements and attempt to pretend to be or to become the master host.
It protects against spoofing attacks : without cryptographic authentication, an attacker could send fake CARP advertisements and attempt to pretend to be or to become the Master host.




Line 141: Line 141:


https://www.giac.org/paper/gsec/4031/carp-free-fail-over-protocol/106433
https://www.giac.org/paper/gsec/4031/carp-free-fail-over-protocol/106433


== IP protocol number  ==
== IP protocol number  ==
Line 147: Line 146:
CARP uses the same IP protocol number as VRRP (112). As a result, packet capture tools such as tcpdump interpret CARP packets as VRRP advertisements like we see above, even though the packets actually belong to the CARP protocol.
CARP uses the same IP protocol number as VRRP (112). As a result, packet capture tools such as tcpdump interpret CARP packets as VRRP advertisements like we see above, even though the packets actually belong to the CARP protocol.


== Multiple VHID ==
Why ??
The OpenBSD developers initially requested unique identifiers from the Internet Assigned Numbers Authority (IANA), but the request was denied because they had not followed the IETF standardization process. As a result, they chose to use protocol number 112, which was already assigned to VRRP. Despite this overlap, VRRP and CARP can coexist on the same network as long as their identifiers (VRRP group ID and CARP VHID) are different.


Using multiple VHIDs in CARP allows several virtual IP addresses to be managed separately. This means different hosts can be master for different services. For example, one host can manage the web service, while another manages the DNS service. Thanks to this, the hosts can be used efficiently while still ensuring high availability. It would not be practical to use a machine only when the other machine is not responding.
Sources :
 
https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol
 
== Multiple VHID : Distribute traffic ==
 
Using multiple VHIDs in CARP allows several virtual IP addresses to be managed separately. This means different hosts can be Master for different services. For example, one host can manage the web service, while another manages the DNS service. Thanks to this, the hosts can be used efficiently while still ensuring high availability. It would not be practical to use a machine only when the other machine is not responding.
 
sources :
 
https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf


== Split-brain ==
== Split-brain ==


Normally:
Normally:
- master
* one Master


- backups
* one or several Backups


If the master host stops sending advertisements, the backup hosts know that the master is no longer reachable. However, a problem can occur if the backup hosts stop receiving each other's advertisements because the link between them is broken. In that case, each node may assume it is the master.
If the Master host stops sending advertisements, the Backup hosts know that the master is no longer reachable. However, a problem can occur if the Backup hosts stop receiving each other's advertisements because the link between them is broken. In that case, each node may assume it is the Master.


When communication is restored, CARP resolves the problem:
When communication is restored, CARP resolves the problem:


  1. The masters can see each other
  1. The Masters can see each other
  2. They receive advertisements from each other
  2. The real Master send the packet
  3. They compare priority
  3. The hosts compare CARP priority
  4. The hosts with lower priority automatically switche back to backup
  4. The hosts with lower priority automatically switch back to Backup
 
sources :
 
https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf


== Master Election Process ==
== Master Election Process ==
Line 171: Line 185:
1. All hosts start
1. All hosts start


  - All hosts begin in BACKUP state.
  All hosts begin in BACKUP state.


2. Each host starts a timer
2. Each host starts a timer


  - The delay depends on the priority (advskew).
* The delay depends on the CARP priority (advskew).


  - The lower the advskew, the shorter the delay.
* The lower the advskew, the shorter the delay.


  - The lower the advskew, the higher the priority.
* The lower the advskew, the higher the priority.


3. The higher-priority host’s timer expires first
3. The higher-priority host’s timer expires first


   - That host sends CARP advertisements when the timer expires.
   That host sends CARP advertisements when the timer expires.


   - It assigns the virtual IP (VIP) to its interface.
   It assigns the virtual IP (VIP) to its interface.


   - It becomes the MASTER host.
   It becomes the MASTER host.


4. The other hosts receive the advertisement
4. The other hosts receive the advertisement


   - They detect that a higher-priority host is active.
   They detect that a higher-priority host is active.


   - They remain in BACKUP state.
   They remain in BACKUP state.


   - They do not send CARP advertisements.
   They do not send CARP advertisements.




In CARP, we usually choose the preferred master by giving it a higher priority (lower advskew). If the master fails and two backup hosts have the same priority, they may both try to become master at the same time. This can temporarily create a split-brain situation, where both hosts start sending CARP advertisements. However, they will quickly detect each other’s advertisements. Because of small timing differences in the network, one host will effectively win and remain master, while the other will switch stay in backup mode. To avoid this situation, it is recommended to configure different priorities for each host.
In CARP, we usually choose the preferred Master by giving it a higher priority (lower advskew).
 
== Virtual Mac Address ==
 
The protocol automatically generates a virtual MAC address based on the VHID, ending with the VHID's value (00:00:5e:00:01:XX). This MAC address is identical on both hosts and is used by the MASTER node.
 
We can see here that the virtual mac address associated to the vhid2 is : 00:00:5e:00:01:02 (02 at the end for vhid2)
 
'''command : tcpdump -e -ni vmx1 proto 112'''
[[ File:Virtual-mac.png ]]
 
✔ The CARP virtual MAC address is visible.
 
❌ The virtual IP address (VIP) is not visible as the source.
 
In CARP advertisements, the Ethernet source address corresponds to the virtual CARP MAC address derived from the VHID. However, the IP source address is the physical IP address of the router sending the advertisement. The virtual IP address (VIP) is not used as the source IP in these control packets. Instead, the VIP is used for normal traffic between clients and the virtual gateway.
 
sources :
 
https://dynfi.com/documentations/dynfi-firewall/configuration_firewall_VIP.html
 
== Address Multicast CARP ==
 
In the Common Address Redundancy Protocol (CARP), routers communicate with each other using a multicast address. CARP advertisements are sent to the IPv4 multicast address '''224.0.0.18''', which allows all routers participating in the CARP group to receive the messages simultaneously. At the Ethernet level, this multicast IP address corresponds to the multicast MAC address '''01:00:5e:00:00:12'''.
 
== Priority vs advskew. ==
 
In the CARP protocol, the Master router is not directly determined by priority, but rather by a value called advskew. Advskew is a numerical parameter between 0 and 255: the lower its value, the more likely the router is to become the Master. Priority, on the other hand, is a more intuitive abstraction used in automation tools or logical configurations: the higher the priority, the higher the router's expected priority. To link the two, we use the formula: advskew = 255 − priority. Thus, a router with a priority of 200 will have a low advskew (55) and will become the Master, while a router with a priority of 0 will have a maximum advskew (255) and will remain in Backup mode. In short, priority is a value that is easily understood and logical for humans, while advskew is the value actually used by CARP's internal mechanism to determine the Master or Backup role. However, advskew can also be called "CARP priority".


== Basic configuration ==
== Basic configuration ==


In this example, we consider a topology with two routers: router-002 and router-003. We want router-002 to act as the  master, while router-003 should operate as the backup.
We want :  
 
router-002 to act as the  master, while router-003 should operate as the backup for Intranought network.
 
router-003 to act as the  master, while router-002 should operate as the backup for Public network.




Line 220: Line 265:




2) Hosts configuration
2) CARP configuration
 
The hostname, management IP address, subnet mask, shared virtual IP address, and VHID are configured by adding the appropriate entries to the ''/etc/rc.conf file''.


In each router, on the new appropriate file '''/etc/rc.conf.d/netif/carp''', we will have the CARP configuration


router-002 :  
router-002 :  


'''hostname="router-002"'''
ifconfig_vmx0_alias0="inet vhid 1 advskew 0 pass xxxxxxx alias 172.27.27.13/27"


'''ifconfig_vmx0="inet 172.27.27.11 netmask 255.255.255.240"'''
ifconfig_vmx1_alias0="inet vhid 2 advskew 100 pass xxxxxxx alias 51.68.252.230/32"


'''ifconfig_vmx0_alias0="inet vhid 1 pass 123piano2026% alias 172.27.27.X/28"'''




router-003 :  
router-003 :  


'''hostname="router-003"'''
ifconfig_vmx0_alias0="inet vhid 1 advskew 100 pass xxxxxxx alias 172.27.27.13/27"
 
'''ifconfig_vmx0="inet 172.27.27.12 netmask 255.255.255.240"'''


'''ifconfig_vmx0_alias0="inet vhid 1 pass 123piano2026% advskew 100 alias 172.27.27.X/28"'''
ifconfig_vmx1_alias0="inet vhid 2 advskew 0 pass xxxxxxx alias 51.68.252.230/32"




So we gave a lower advskew to the router-002 (default is 0) to become the master router.
So we gave a lower advskew (default is 0) to become the master router.




Line 250: Line 291:
commands :  
commands :  


'''ifconfig carp0 create'''
ifconfig carp0 create
 
ifconfig carp0 vhid 1 pass xxxxxx
'''ifconfig carp0 vhid 1 pass 123piano2026%'''
ifconfig carp0 inet 172.27.27.X/28'''
 
'''ifconfig carp0 inet 172.27.27.X/28'''


sources :


https://docs-archive.freebsd.org/doc/11.0-RELEASE/usr/local/share/doc/freebsd/en/books/handbook/carp.html


== Automatic blue/green deployment ==
== Automatic blue/green deployment ==
Line 282: Line 323:
The difference is that in traditional Blue-Green deployment, the switch is usually performed manually during an update. With CARP, the switch happens automatically in case of failure.
The difference is that in traditional Blue-Green deployment, the switch is usually performed manually during an update. With CARP, the switch happens automatically in case of failure.


== Priority vs advskew. ==
== ESXI and CARP ==
 
In the CARP protocol, the master router is not directly determined by priority, but rather by a value called advskew. Advskew is a numerical parameter between 0 and 255: the lower its value, the more likely the router is to become the master. Priority, on the other hand, is a more intuitive abstraction used in automation tools or logical configurations: the higher the priority, the higher the router's expected priority. To link the two, we use the formula: advskew = 255 − priority. Thus, a router with a priority of 200 will have a low advskew (55) and will become the master, while a router with a priority of 0 will have a maximum advskew (255) and will remain in backup mode. In short, priority is a value that is easily understood and logical for humans, while advskew is the value actually used by CARP's internal mechanism to determine the master or backup role.
 
== Virtual Mac Address ==
 
The protocol automatically generates a virtual MAC address based on the VHID, ending with the VHID's value. This MAC address is identical on both hosts and is used by the MASTER node to advertise the VIP on the network.
 
 
 
Linked to {{T|2227}}
 
 
== Distribute traffic ==
 
The current infrastructure relies on two distinct networks at the ESXI hypervisor level:
 
- An internal network 172.27.27.0/28, referred to as IntraNought, used for internal communication and management.
 
- A public network used for public IP addresses exposed to the Internet.
 
The objective now is to evaluate the possibility of implementing an active/active model so that both routers would receive traffic during normal operation, while still maintaining high availability through CARP.
 
On the internal network (IntraNought), configuring CARP does not introduce a significant complexity. One router can operate as master and the other as backup. We will configure a first VHID.
 
However, on the public network, a constraint exists. Public IP addresses (IPFO) provided by OVH are associated with a specific MAC address. If both routers need to use the same public IP address, they must present the same MAC address.
 
By default, the ESXi vSwitch blocks situations where two virtual machines use the same MAC address. This requires enabling specific settings such as Promiscuous Mode, MAC Address Changes and Forged Transmits. It is technically feasible, but this adds complexity and may require network isolation by using a dedicated vSwitch for the routers.
 
Finally, even if both routers are active, they run on the same hypervisor. This means the hypervisor is a Single Point of Failure (SPOF). If the physical host fails, both routers would go down at the same time, so the redundancy only exists at the virtual level, not at the hardware level.


When using Common Address Redundancy Protocol in a virtualized environment such as VMware ESXi, specific security settings must be adjusted on the virtual switch. CARP relies on the use of a virtual MAC address that can move between routers depending on which device is currently the Master. However, ESXi applies strict security policies by default to prevent virtual machines from sending or receiving traffic with unexpected MAC addresses. To ensure that CARP functions correctly, three settings must be enabled on the vSwitch or port group.


First, '''Promiscuous Mode''' must be enabled. This allows a virtual machine to receive all frames circulating on the virtual network, even if the destination MAC address does not match the interface’s MAC address. This is important for CARP because routers must receive multicast advertisements sent to the CARP group.


'''But at this point, a key question remains: If two VMs share the same MAC address, how does the vSwitch decide which VM should receive the traffic? Could it send the frame to the wrong VM? Or both ?'''
Second, '''MAC Address Changes''' must be allowed. CARP dynamically assigns a virtual MAC address to the Master router. When a failover occurs, the Backup router must adopt this same virtual MAC address. If ESXi blocks MAC address changes, the router will not be able to assume the virtual MAC address and the failover mechanism will fail.


In our setup, two routers share the same virtual public IP and therefore the same virtual MAC address using CARP. However, only one router is active (the master) at a time.
Finally, '''Forged Transmits''' must also be permitted. This option allows a virtual machine to send frames with a source MAC address different from the one originally assigned to the virtual network interface. Since CARP advertisements and ARP responses are sent using the virtual MAC address rather than the physical interface MAC address, this setting is required to allow those frames to be transmitted correctly.


When the master sends traffic, the ESXi vSwitch learns the virtual MAC address from the source MAC of the frames it receives. It then associates that MAC address with the master’s port. As a result, any incoming traffic destined for that MAC is forwarded only to the active router.
Together, these three settings ensure that CARP redundancy mechanisms operate properly in a virtualized ESXi network environment.


The backup router has the same virtual MAC configured, but it does not actively use it while in backup state. It does not respond to ARP requests and does not send traffic using that MAC. Therefore, the vSwitch does not associate the MAC with the backup’s port.
Configuration to enable "Promiscuous Mode", “MAC Address Changes” and “Forged Transmits”  :  
 
If a failover occurs, the backup router becomes master and starts sending traffic with the same virtual MAC. The vSwitch detects that the MAC now appears on a different port and updates its table. Traffic is then forwarded to the new master.
 
Security settings such as “MAC Address Changes” and “Forged Transmits” must be set to Accept to allow this MAC movement between ports and prevent ESXi from blocking the traffic as a potential MAC spoofing attempt.
 
Regarding Promiscuous Mode, VMware documentation states: “Promiscuous mode eliminates any reception filtering that the virtual machine adapter performs so that the guest operating system receives all traffic observed on the wire.” This means that the filtering normally performed by the virtual network adapter based on the destination MAC address is disabled. As a result, the guest operating system can receive all traffic visible on that VLAN. However, this setting does not modify the forwarding logic of the vSwitch itself. The vSwitch continues to perform standard Layer 2 MAC learning and forwards unicast traffic to a single port at a time. Therefore, even with Promiscuous Mode enabled, the vSwitch does not behave like a hub. Promiscuous mode is mainly used for traffic monitoring or packet sniffing, it is not required in our case.
 
Also, VMware documentation explicitly mentions legitimate scenarios where multiple adapters share the same MAC address, such as Microsoft NLB in unicast mode. This confirms that using a shared virtual MAC with CARP is a supported and valid design pattern.
 
 
 
Configuration to enable “MAC Address Changes” and “Forged Transmits”  :  


https://knowledge.broadcom.com/external/article/324520/configuring-promiscuous-mode-on-a-virtua.html
https://knowledge.broadcom.com/external/article/324520/configuring-promiscuous-mode-on-a-virtua.html
Source documentation:


https://wxcafe.net/posts/redondance-routeurs-openbsd-freebsd/
Sources :  


https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf
https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-security/securing-vsphere-networking/securing-vsphere-standard-switches.html
 
https://docs-archive.freebsd.org/doc/11.0-RELEASE/usr/local/share/doc/freebsd/en/books/handbook/carp.html


https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-security/securing-vsphere-networking/securing-vsphere-standard-switches.html


https://dynfi.com/documentations/dynfi-firewall/configuration_firewall_VIP.html
Linked to {{T|2227}}

Latest revision as of 20:49, 9 March 2026

Introduction and history

« CARP stands for Common Address Redundancy Protocol and its basic functionality is to allow multiple hosts to share a set of IP addresses ».

CARP was developed by the OpenBSD project in 2003 as an alternative to VRRP due to patent concerns related to Cisco's HSRP technology. To avoid potential legal issues and remain compatible with open-source principles, OpenBSD implemented its own redundancy protocol with added cryptographic features. CARP was later adopted by other systems such as FreeBSD and NetBSD. A userland implementation of CARP, named ucarp, made the protocol usable on Linux without requiring kernel integration.

These patent concerns originated from the development of VRRP by the Internet Engineering Task Force (IETF). Indeed, when the Internet Engineering Task Force (IETF) developed the Virtual Router Redundancy Protocol (VRRP), Cisco Systems indicated that it owned patents related to router redundancy mechanisms used in the protocol Hot Standby Router Protocol (HSRP). This created a legal risk for developers, particularly for open-source projects that wanted to implement VRRP freely.

Sources :

https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

Master host and backup hosts

CARP creates a redundancy group, meaning several hosts are configured to share a virtual IP address. However, at any given time, only one host uses the shared IP address. This host is called the Master host and it receives and handles all traffic destined to that virtual IP.

When the Master host becomes unavailable (it crashed, turned off, or lost its network connection), the other hosts in the same redundancy group detect the failure. Immediately, one of the Backup hosts is elected as the new Master host. That means that it will take over the shared IP address.

This switch happens automatically, ensuring service continuity without clients noticing any interruption.

Sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

https://wxcafe.net/posts/redondance-routeurs-openbsd-freebsd/

CARP advertisements

- The Master host periodically broadcasts CARP advertisements to the backup hosts.

- The Backup hosts listen but do not send any packets.


Each CARP advertisement contains :

1. The CARP version and packet type (CARP Header).

2. The VHID (Virtual Host ID), which identifies the redundancy group (CARP Header).

3. The advertisement parameters (advbase and advskew), which determine the host’s priority CARP (CARP Header).

4. The counter

5. The signature in SHA-1


CARP Advertisement Packet Format :

Each line represent 32 bits, 4 bytes

Length : 36 bytes 

/*
 * The CARP header layout is as follows:
 *
 *     0                   1                   2                   3
 *     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 *    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 *    |Version| Type  | VirtualHostID |    AdvSkew    |    Auth Len   |
 *    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 *    |    Demotion   |     AdvBase   |          Checksum             |
 *    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 *    |                         Counter (1)                           |
 *    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 *    |                         Counter (2)                           |
 *    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 *    |                        SHA-1 HMAC (1)                         |
 *    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 *    |                        SHA-1 HMAC (2)                         |
 *    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 *    |                        SHA-1 HMAC (3)                         |
 *    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 *    |                        SHA-1 HMAC (4)                         |
 *    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 *    |                        SHA-1 HMAC (5)                         |
 *    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 *
 */


commande : tcpdump -i <interface> proto 112

Sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

https://github.com/openbsd/src/blob/master/sys/netinet/ip_carp.h

Cryptography

All CARP advertisements are cryptographically signed :

Steps :

1. The Master host creates an advertisement.

2. It calculates a cryptographic signature, a SHA-1 hash using:

  • the shared secret,
  • the VHID,
  • the virtual IP address,
  • the advertisement parameters (advbase and advskew),
  • the counter
  • and other header fields.

3. It then sends the signed advertisement.

4. The Backup hosts receives the signed advertisement.

5. Each Backup host recalculates the signature using the same shared secret : 

- If the calculated signature matches the received signature → the packet is considered valid because authentication and integrity are insured.

- If it does not match → the packet is rejected.

It protects against spoofing attacks : without cryptographic authentication, an attacker could send fake CARP advertisements and attempt to pretend to be or to become the Master host.


command : tcpdump -X -ni <interface> proto 112

We can observe the complete packet in the capture. Each hexadecimal value represents two bytes. The first 20 bytes correspond to the IP header, while the last 20 bytes correspond to the SHA-1 authentication hash included in the CARP advertisement.

Sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

https://www.giac.org/paper/gsec/4031/carp-free-fail-over-protocol/106433

IP protocol number

CARP uses the same IP protocol number as VRRP (112). As a result, packet capture tools such as tcpdump interpret CARP packets as VRRP advertisements like we see above, even though the packets actually belong to the CARP protocol.

Why ?? The OpenBSD developers initially requested unique identifiers from the Internet Assigned Numbers Authority (IANA), but the request was denied because they had not followed the IETF standardization process. As a result, they chose to use protocol number 112, which was already assigned to VRRP. Despite this overlap, VRRP and CARP can coexist on the same network as long as their identifiers (VRRP group ID and CARP VHID) are different.

Sources :

https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol

Multiple VHID : Distribute traffic

Using multiple VHIDs in CARP allows several virtual IP addresses to be managed separately. This means different hosts can be Master for different services. For example, one host can manage the web service, while another manages the DNS service. Thanks to this, the hosts can be used efficiently while still ensuring high availability. It would not be practical to use a machine only when the other machine is not responding.

sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

Split-brain

Normally:

  • one Master
  • one or several Backups

If the Master host stops sending advertisements, the Backup hosts know that the master is no longer reachable. However, a problem can occur if the Backup hosts stop receiving each other's advertisements because the link between them is broken. In that case, each node may assume it is the Master.

When communication is restored, CARP resolves the problem: 

1. The Masters can see each other
2. The real Master send the packet
3. The hosts compare CARP priority
4. The hosts with lower priority automatically switch back to Backup

sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

Master Election Process

1. All hosts start 

 All hosts begin in BACKUP state.

2. Each host starts a timer

  • The delay depends on the CARP priority (advskew).
  • The lower the advskew, the shorter the delay.
  • The lower the advskew, the higher the priority.

3. The higher-priority host’s timer expires first 

  That host sends CARP advertisements when the timer expires.

  It assigns the virtual IP (VIP) to its interface.

  It becomes the MASTER host.

4. The other hosts receive the advertisement 

  They detect that a higher-priority host is active.

  They remain in BACKUP state.

  They do not send CARP advertisements.


In CARP, we usually choose the preferred Master by giving it a higher priority (lower advskew).

Virtual Mac Address

The protocol automatically generates a virtual MAC address based on the VHID, ending with the VHID's value (00:00:5e:00:01:XX). This MAC address is identical on both hosts and is used by the MASTER node.

We can see here that the virtual mac address associated to the vhid2 is : 00:00:5e:00:01:02 (02 at the end for vhid2)

command : tcpdump -e -ni vmx1 proto 112

✔ The CARP virtual MAC address is visible.

❌ The virtual IP address (VIP) is not visible as the source.

In CARP advertisements, the Ethernet source address corresponds to the virtual CARP MAC address derived from the VHID. However, the IP source address is the physical IP address of the router sending the advertisement. The virtual IP address (VIP) is not used as the source IP in these control packets. Instead, the VIP is used for normal traffic between clients and the virtual gateway.

sources :

https://dynfi.com/documentations/dynfi-firewall/configuration_firewall_VIP.html

Address Multicast CARP

In the Common Address Redundancy Protocol (CARP), routers communicate with each other using a multicast address. CARP advertisements are sent to the IPv4 multicast address 224.0.0.18, which allows all routers participating in the CARP group to receive the messages simultaneously. At the Ethernet level, this multicast IP address corresponds to the multicast MAC address 01:00:5e:00:00:12.

Priority vs advskew.

In the CARP protocol, the Master router is not directly determined by priority, but rather by a value called advskew. Advskew is a numerical parameter between 0 and 255: the lower its value, the more likely the router is to become the Master. Priority, on the other hand, is a more intuitive abstraction used in automation tools or logical configurations: the higher the priority, the higher the router's expected priority. To link the two, we use the formula: advskew = 255 − priority. Thus, a router with a priority of 200 will have a low advskew (55) and will become the Master, while a router with a priority of 0 will have a maximum advskew (255) and will remain in Backup mode. In short, priority is a value that is easily understood and logical for humans, while advskew is the value actually used by CARP's internal mechanism to determine the Master or Backup role. However, advskew can also be called "CARP priority".

Basic configuration

We want : 

router-002 to act as the  master, while router-003 should operate as the backup for Intranought network.

router-003 to act as the  master, while router-002 should operate as the backup for Public network.


1) Enable CARP Support

On the file /boot/loader.conf, you need to add : carp_load="YES"


You are telling FreeBSD:

“Load the CARP kernel module automatically at boot time.”

If you want to load it now, but not permanently : kldload carp


If someone has build a custom FreeBSD kernel, they can include CARP directly by adding this line to the kernel configuration file: device carp


2) CARP configuration

In each router, on the new appropriate file /etc/rc.conf.d/netif/carp, we will have the CARP configuration

router-002 : 

ifconfig_vmx0_alias0="inet vhid 1 advskew 0 pass xxxxxxx alias 172.27.27.13/27"

ifconfig_vmx1_alias0="inet vhid 2 advskew 100 pass xxxxxxx alias 51.68.252.230/32"


router-003 : 

ifconfig_vmx0_alias0="inet vhid 1 advskew 100 pass xxxxxxx alias 172.27.27.13/27"

ifconfig_vmx1_alias0="inet vhid 2 advskew 0 pass xxxxxxx alias 51.68.252.230/32"


So we gave a lower advskew (default is 0) to become the master router.


In older versions of FreeBSD (9 and earlier), CARP required creating a separate virtual interface such as carp0, and the shared IP address was attached to that interface. Starting with FreeBSD 10, this was simplified. CARP can now be configured directly on the physical interface using an alias with a VHID. The newer method is easier to configure and maintain, so it is recommended to use it in modern versions of FreeBSD.

commands : 

ifconfig carp0 create
ifconfig carp0 vhid 1 pass xxxxxx
ifconfig carp0 inet 172.27.27.X/28

sources :

https://docs-archive.freebsd.org/doc/11.0-RELEASE/usr/local/share/doc/freebsd/en/books/handbook/carp.html

Automatic blue/green deployment

In this context, “automatic Blue-Green” does not refer to a traditional application deployment strategy. Instead, it describes a network-level behavior enabled by CARP.

With CARP, two routers are deployed in parallel: 

- One acts as the active router (MASTER)

- The other remains in standby (BACKUP)

Both routers are fully operational and connected to the network. Traffic is sent to a shared virtual IP address. At any given time, only the MASTER processes the traffic. However, if the MASTER fails, the BACKUP automatically takes over the virtual IP address without manual intervention.

This behavior resembles Blue-Green deployment because: 

1. Two environments exist simultaneously

2. Only one actively handles traffic

3. Traffic can switch from one to the other

4. The switch is transparent to users

The difference is that in traditional Blue-Green deployment, the switch is usually performed manually during an update. With CARP, the switch happens automatically in case of failure.

ESXI and CARP

When using Common Address Redundancy Protocol in a virtualized environment such as VMware ESXi, specific security settings must be adjusted on the virtual switch. CARP relies on the use of a virtual MAC address that can move between routers depending on which device is currently the Master. However, ESXi applies strict security policies by default to prevent virtual machines from sending or receiving traffic with unexpected MAC addresses. To ensure that CARP functions correctly, three settings must be enabled on the vSwitch or port group.

First, Promiscuous Mode must be enabled. This allows a virtual machine to receive all frames circulating on the virtual network, even if the destination MAC address does not match the interface’s MAC address. This is important for CARP because routers must receive multicast advertisements sent to the CARP group.

Second, MAC Address Changes must be allowed. CARP dynamically assigns a virtual MAC address to the Master router. When a failover occurs, the Backup router must adopt this same virtual MAC address. If ESXi blocks MAC address changes, the router will not be able to assume the virtual MAC address and the failover mechanism will fail.

Finally, Forged Transmits must also be permitted. This option allows a virtual machine to send frames with a source MAC address different from the one originally assigned to the virtual network interface. Since CARP advertisements and ARP responses are sent using the virtual MAC address rather than the physical interface MAC address, this setting is required to allow those frames to be transmitted correctly.

Together, these three settings ensure that CARP redundancy mechanisms operate properly in a virtualized ESXi network environment.

Configuration to enable "Promiscuous Mode", “MAC Address Changes” and “Forged Transmits”  :

https://knowledge.broadcom.com/external/article/324520/configuring-promiscuous-mode-on-a-virtua.html

Sources :

https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-security/securing-vsphere-networking/securing-vsphere-standard-switches.html


Linked to T2227

Retrieved from "https://agora.nasqueron.org/index.php?title=Protocol_CARP&oldid=2470"