Protocol CARP

Introduction

CARP stands for Common Address Redundancy Protocol and its basic functionality is to allow multiple hosts to share a set of IP addresses

CARP was first introduced in 2003 in OpenBSD as an alternative to Cisco’s VRRP protocol. It was created to avoid patent issues related to VRRP. Later, CARP was added to FreeBSD and NetBSD. A userland implementation of CARP protocol called ucarp was also developed, which allowed CARP to be used on Linux systems.

Master host and backup hosts

CARP creates a redundancy group, meaning several hosts are configured to share a virtual IP address. However, at any given time, only one host uses the shared IP address. This host is called the master host and it receives and handles all traffic destined to that virtual IP.

When the master host becomes unavailable (it crashed, turned off, or lost its network connection), the other hosts in the same redundancy group detect the failure. Immediately, one of the backup hosts is elected as the new master host. That means that it will take over the shared IP address.

This switch happens automatically, ensuring service continuity without clients noticing any interruption.

CARP advertisements

- The master host periodically broadcasts CARP advertisements to the backup hosts.

- Backup hosts listen but do not send any packets.

Each CARP advertisement contains:

1. The VHID (Virtual Host ID), which identifies the redundancy group.

2. The CARP version and packet type.

3. The advertisement parameters (advbase and advskew), which determine the host’s priority.

All CARP advertisements are cryptographically signed :

Steps :

1. The master host creates an advertisement.

2. It calculates a cryptographic signature using:

     - the shared secret,

     - the VHID,

     - the virtual IP address,

     - the advertisement parameters (advbase and advskew),

     - and other header fields.

3. It then sends the signed advertisement.

4. The backup hosts receives the signed advertisement.

5. Each backup host recalculates the signature using the same shared secret :

- If the calculated signature matches the received signature → the packet is considered valid.

- If it does not match → the packet is rejected.

It protects against spoofing attacks : without cryptographic authentication, an attacker could send fake CARP advertisements and attempt to become the master host.

Also, all hosts must have the same set of virtual IP addresses configured in order to calculate the signature (because the IP addresses are not send), but only one host actually uses these IP addresses at any given time.

Multiple VHID

Using multiple VHIDs in CARP allows several virtual IP addresses to be managed separately. This means different hosts can be master for different services. For example, one host can manage the web service, while another manages the DNS service. Thanks to this, the hosts can be used efficiently while still ensuring high availability. It would not be practical to use a machine only when the other machine is not responding.

Split-brain

Normally:

- master

- backups

If the master host stops sending advertisements, the backup nodes know that the master is no longer reachable. However, a problem can occur if the backup nodes stop receiving each other's advertisements because the link between them is broken. In that case, each node may assume it is the master.

When communication is restored, CARP resolves the problem:

1. The masters can see each other
2. They receive advertisements from each other
3. They compare priority
4. The hosts with lower priority automatically switche back to backup

Master Election Process

1. All hosts start

  - All hosts begin in BACKUP state.

2. Each host starts a timer

  - The delay depends on the priority (advskew).

  - The lower the advskew, the shorter the delay.

  - The lower the advskew, the higher the priority.

3. The higher-priority host’s timer expires first

  - That host sends CARP advertisements when the timer expires.

  - It assigns the virtual IP (VIP) to its interface.

  - It becomes the MASTER host.

4. The other hosts receive the advertisement

  - They detect that a higher-priority host is active.

  - They remain in BACKUP state.

  - They do not send CARP advertisements.

In CARP, we usually choose the preferred master by giving it a higher priority (lower advskew). If the master fails and two backup hosts have the same priority, they may both try to become master at the same time. This can temporarily create a split-brain situation, where both hosts start sending CARP advertisements. However, they will quickly detect each other’s advertisements. Because of small timing differences in the network, one host will effectively win and remain master, while the other will switch stay in backup mode. To avoid this situation, it is recommended to configure different priorities for each host.

Basic configuration

In this example, we consider a topology with two routers: router-002 and router-003. We want router-002 to act as the master, while router-003 should operate as the backup.

1) Enable CARP Support

On the file /boot/loader.conf, you need to add : carp_load="YES"

You are telling FreeBSD:

“Load the CARP kernel module automatically at boot time.”

If you want to load it now, but not permanently : kldload carp

If someone has build a custom FreeBSD kernel, they can include CARP directly by adding this line to the kernel configuration file: device carp

2) Hosts configuration

The hostname, management IP address, subnet mask, shared virtual IP address, and VHID are configured by adding the appropriate entries to the /etc/rc.conf file.

router-002 :

hostname="router-002"

ifconfig_vmx0="inet 172.27.27.11 netmask 255.255.255.240"

ifconfig_vmx0_alias0="inet vhid 1 pass 123piano2026% alias 172.27.27.X/28"

router-003 :

hostname="router-003"

ifconfig_vmx0="inet 172.27.27.12 netmask 255.255.255.240"

ifconfig_vmx0_alias0="inet vhid 1 pass 123piano2026% advskew 100 alias 172.27.27.X/28"

So we gave a lower advskew to the router-002 (default is 0) to become the master router.

In older versions of FreeBSD (9 and earlier), CARP required creating a separate virtual interface such as carp0, and the shared IP address was attached to that interface. Starting with FreeBSD 10, this was simplified. CARP can now be configured directly on the physical interface using an alias with a VHID. The newer method is easier to configure and maintain, so it is recommended to use it in modern versions of FreeBSD.

commands :

ifconfig carp0 create

ifconfig carp0 vhid 1 pass 123piano2026%

ifconfig carp0 inet 192.168.1.50/24

Distribute traffic

The current infrastructure relies on two distinct networks at the ESXI hypervisor level:

- An internal network 172.27.27.0/28, referred to as IntraNought, used for internal communication and management.

- A public network used for public IP addresses exposed to the Internet.

The initial objective is to evaluate the possibility of implementing an active/active model so that both routers would receive traffic during normal operation, while still maintaining high availability through CARP.

On the internal network (IntraNought), configuring CARP does not introduce a significant complexity. One router can operate as master and the other as backup. We will configure a first VHID.

However, on the public network, a constraint exists. Public IP addresses (IPFO) provided by OVH are associated with a specific MAC address. If both routers need to use the same public IP address, they must present the same MAC address.

By default, the ESXi vSwitch blocks situations where two virtual machines use the same MAC address. This requires enabling specific settings such as Promiscuous Mode, MAC Address Changes, and Forged Transmits. It is technically feasible, but this adds complexity and may require network isolation by using a dedicated vSwitch for the routers.

Finally, even if both routers are active, they run on the same hypervisor. This means the hypervisor is a Single Point of Failure (SPOF). If the physical host fails, both routers would go down at the same time, so the redundancy only exists at the virtual level, not at the hardware level.

When a virtual machine wants to communicate with the Internet, it sends its traffic to the default gateway. Before sending the packet, it uses ARP to resolve the gateway’s IP address into a MAC address. The packet is then sent to that MAC address, and the ESXi vSwitch forwards it based on its MAC address table.

But at this point, a key question remains: how does ESXi decide which virtual machine receives the traffic in this situation? Does the vSwitch act like a hub ?

In our setup, two routers share the same virtual public IP and therefore the same virtual MAC address using CARP. However, only one router is active (the master) at a time.

When the master sends traffic, the ESXi vSwitch learns the virtual MAC address from the source MAC of the frames it receives. It then associates that MAC address with the master’s port. As a result, any incoming traffic destined for that MAC is forwarded only to the active router.

The backup router has the same virtual MAC configured, but it does not actively use it while in backup state. It does not respond to ARP requests and does not send traffic using that MAC. Therefore, the vSwitch does not associate the MAC with the backup’s port.

If a failover occurs, the backup router becomes master and starts sending traffic with the same virtual MAC. The vSwitch detects that the MAC now appears on a different port and updates its table. Traffic is then forwarded to the new master.

Security settings such as “MAC Address Changes” and “Forged Transmits” must be set to Accept to allow this MAC movement between ports and prevent ESXi from blocking the traffic as a potential MAC spoofing attempt.

Regarding Promiscuous Mode, VMware documentation states: “Promiscuous mode eliminates any reception filtering that the virtual machine adapter performs so that the guest operating system receives all traffic observed on the wire.” This means that the filtering normally performed by the virtual network adapter based on the destination MAC address is disabled. As a result, the guest operating system can receive all traffic visible on that VLAN. However, this setting does not modify the forwarding logic of the vSwitch itself. The vSwitch continues to perform standard Layer 2 MAC learning and forwards unicast traffic to a single port at a time. Therefore, even with Promiscuous Mode enabled, the vSwitch does not behave like a hub.

Configiration to enable the settingb :

https://knowledge.broadcom.com/external/article/324520/configuring-promiscuous-mode-on-a-virtua.html

Source :

https://wxcafe.net/posts/redondance-routeurs-openbsd-freebsd/ https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf https://docs-archive.freebsd.org/doc/11.0-RELEASE/usr/local/share/doc/freebsd/en/books/handbook/carp.html https://knowledge.broadcom.com/external/article/324520/configuring-promiscuous-mode-on-a-virtua.html