Operations grimoire/How to add a server to the Nasqueron servers pool

From Nasqueron Agora

Naming convention

Servers names use <feature>-<counter>, for example docker-002 or web-001.

Previously, naming convention stated server names should be proper name from Iain Banks, The Algebraist.

If you want to create a server to do host one site, it's suggested the server name should still following this convention and so be different than your website subdomain.

Setup the server

NetBox

  • Create IP if needed, both for public and private networks
  • Create the device. Beware sections for VM and physical machines aren't the same
  • Declare network interfaces carefully: it's the source of truth and Salt will use that for configure network

OS installation

Nothing fancy here:

  • On VMWare ESXi, we download ISO and setup manually.
  • Dedicated bare-metal servers can be installed with provider method
  • Scaleway instances could benefit of a custom image

Some tips:

  • Timezone MUST be UTC (from tzdata etc/UTC)
  • Root password doesn't matter, as long as it's long enough ; document it to Vault under ops/secrets/infra/
  • No need to create another user, Salt will do that (correctly)
  • FreeBSD servers use zfs on root with arcology as default pool name

Bootstrap the server

You can use rOPS: utils/bootstrap/hello-FreeBSD.sh to install Salt and connect to Complector.

fetch https://windriver.nasqueron.org/~dereckson/ops/hello-FreeBSD.sh
chmod +x hello-FreeBSD.sh
./hello-FreeBSD.sh

Copy it and adapt it for other OSes (and commit what works). Tilde character lost? <alt gr> + 126

If the server isn't already on the private network, you've git to clone Ops repository to /srv/salt and use salt-call --local saltutil.sync_all ; salt-call --local state.apply roles/core.

If the server isn't on the private network at that stage, open a task on DevCentral for a new GRE tunnel or any other fancy way to get there.

Provision through Salt

Commit it:

  • rOPS: pillar/nodes/nodes.sls as long as we don't have merged our custom pillar to fetch info from NetBox
  • {{Ops file|roles/core/motd/files} can host a custom MOTD to explain this server role

Run it:

  • Go to complector, accept key with salt-key -A
  • Apply our modules: salt-call --local saltutil.sync_all
  • Apply the core role: salt-call --local state.apply roles/core
  • Report any issue on DevCentral, roles/core MUST run smoothly everywhere
  • Apply other roles as needed, one after the other

If you recreate a machine where everything is already tested fine as Salt as concerned, feel free to run the highstate.

Web guidelines

If the server run a webserver, it's suggested at least three vhosts:

  • one to catch every request not intended for one of the configured block, to provide user with an error message (like the 404 droid not found) if a domain points to our server but is not configured ;
  • one at <short hostname>, <full hostname>, localhost, <IP>, to offer a cover page for the server,
  • one for the service you host.

Checklist

  • Document the server on the wiki:
    • Create a page on the wiki, with the server's short hostname as title (e.g. Ysul). Minimal information is server name IP and maintainer.
    • Add the server addresses on the IPs page
    • If the server is public or semi public, consider to document an AUP (acceptable use policy).
  • Provision the server through Salt
  • Prepare the server cover homepage: