Nasqueron Agora - Recent changes [en]

Operations grimoire/Operations repository

Dereckson — Sun, 07 Jun 2026 11:32:13 GMT

macOS

← Older revision		Revision as of 11:32, 7 June 2026
(One intermediate revision by the same user not shown)
Line 94:		Line 94:

	This section cover topics NOT needed on our development servers.		This section cover topics NOT needed on our development servers.

			==== macOS ====

			To run tests properly, install first:

			brew install bats
			brew install python3.13

			You can now setup the virtual environment per everywhere setup instructions.

			Salt behaves correctly on macOS, so you can install it safely in the virtual environment:

			pip install salt==3006.25

			Check the version on WindRiver or Complector with <code>salt --versions</code>.

			Finally, to run the tests, you need a recent version of make, but the BSD make currently shipped, as of Tahoe 26.5, isn't up-to-date:

			brew install gmake
			(cd _tests && gmake)

	==== Arcanist ====		==== Arcanist ====

Operations grimoire/Router

Duranzed — Wed, 20 May 2026 15:36:31 GMT

Procedure

← Older revision		Revision as of 15:36, 20 May 2026
(9 intermediate revisions by the same user not shown)
Line 1:		Line 1:
	= ~~Router~~ =		= Routers =

	== Troubleshoot ==		== Troubleshoot ==
Line 13:		Line 13:
	* access through Windriver becomes unstable		* access through Windriver becomes unstable
	* router-003 does not appear to restore SSH access properly when it becomes primary		* router-003 does not appear to restore SSH access properly when it becomes primary
	* the issue may disappear temporarily after switching the primary router ~~(mostly router-002 remains usuable),~~ but can come back later		* the issue may disappear temporarily after switching the primary router but can come back later

	Observed troubleshooting notes:		Observed troubleshooting notes:
Line 25:		Line 25:

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	ifconfig vmx1 down		ifconfig vmx1 down on router-002 / ifconfig vmx1 up on router-003
	</syntaxhighlight>		</syntaxhighlight>

	~~To re-enable~~ the ~~interface later:~~		the unused router needs to be in INIT status and not BAKCUP, as BACKUP status blocks access for an unknown reason.

	~~<syntaxhighlight lang="bash">~~		Check CARP logs after finishing the procedure.
	~~ifconfig vmx1 up~~
	~~</syntaxhighlight>~~

	~~If the switch does not restore connectivity, it may be necessary to revert the operation by disabling `vmx1` again on the new primary and re-enabling it on~~ the ~~previous router~~.

	~~==== Checks ====~~

	~~Check current routing table:~~

	~~<syntaxhighlight lang="bash">~~
	~~netstat -rn~~
	~~</syntaxhighlight>~~

	~~Check TCP connectivity:~~

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	nc -~~zv 172~~.~~27.27.7 22~~		tail -f /var/log/carp.log
	~~nc -zv 172.27.27.7 8201~~
	</syntaxhighlight>		</syntaxhighlight>

	~~Check Vault status:~~

	~~<syntaxhighlight lang="bash">~~
	~~vault status~~
	~~</syntaxhighlight>~~

	~~==== Notes ====~~

	This is currently a workaround, not a permanent fix.		This is currently a workaround, not a permanent fix.

	The network issue is still unstable and requires further investigation.		The network issue is still unstable and requires further investigation.

Operations grimoire/Router

Duranzed — Wed, 20 May 2026 15:02:53 GMT

Created page with "= Router = == Troubleshoot == === Switch primary router === This procedure can be used when the current primary router doesn't allow access and network access through the GRE/CARP setup is degraded. Typical symptoms observed: * ICMP may still work, but TCP sessions cannot actually be used * `nc` can connect to TCP ports, but SSH or Vault requests hang * Vault queries time out during TLS handshake * access through Windriver becomes unstable * router-003 does not appea..."

New page

= Router =

== Troubleshoot ==

=== Switch primary router ===

This procedure can be used when the current primary router doesn't allow access and network access through the GRE/CARP setup is degraded.

Typical symptoms observed:
* ICMP may still work, but TCP sessions cannot actually be used
* `nc` can connect to TCP ports, but SSH or Vault requests hang
* Vault queries time out during TLS handshake
* access through Windriver becomes unstable
* router-003 does not appear to restore SSH access properly when it becomes primary
* the issue may disappear temporarily after switching the primary router (mostly router-002 remains usuable), but can come back later

Observed troubleshooting notes:
* the issue can reappear after about 1 hour, but sometimes only after up to 48 hours
* GRE tunnels may remain pingable while application access times out
* when the issue occurs, routing and primary router state should be checked

==== Procedure ====

If `router-002` is currently primary and needs to be switched out temporarily, disable `vmx1` on `router-002` so that `router-003` can take over:

<syntaxhighlight lang="bash">
ifconfig vmx1 down
</syntaxhighlight>

To re-enable the interface later:

<syntaxhighlight lang="bash">
ifconfig vmx1 up
</syntaxhighlight>

If the switch does not restore connectivity, it may be necessary to revert the operation by disabling `vmx1` again on the new primary and re-enabling it on the previous router.

==== Checks ====

Check current routing table:

<syntaxhighlight lang="bash">
netstat -rn
</syntaxhighlight>

Check TCP connectivity:

<syntaxhighlight lang="bash">
nc -zv 172.27.27.7 22
nc -zv 172.27.27.7 8201
</syntaxhighlight>

Check Vault status:

<syntaxhighlight lang="bash">
vault status
</syntaxhighlight>

==== Notes ====

This is currently a workaround, not a permanent fix.

The network issue is still unstable and requires further investigation.

Operations grimoire

Duranzed — Wed, 20 May 2026 14:59:10 GMT

Network

← Older revision		Revision as of 14:59, 20 May 2026
Line 35:		Line 35:
	* [[/DNS]]		* [[/DNS]]
	* [[/Firewall]]		* [[/Firewall]]
			* [[/Router]]
	* [[/IPv6]]		* [[/IPv6]]
	* [https://netbox.nasqueron.org/ NetBox]		* [https://netbox.nasqueron.org/ NetBox]