Operations grimoire/Operations repository
Our configuration as code is stores in the operations (rOPS) repository.
Repository layout
| Path | Description |
|---|---|
| _modules | Custom execution modules |
| _states | Custom states modules |
| _tests | Unit tests for custom modules, scripts, tests for repo |
| hotfixes | Fixes you need to run to solve a problem on the servers |
| pillar | Configuration data structures to use in the states |
| roles | The states to deploy, divided in roles and then in units |
| utils | Helper scripts to maintain the repository |
| map.jinja | As we deploy on several OS and distros, mapping of packages names or directories |
| top.sls | topfile: what we deploy where? |
| PORTS | Documentation of the ports used in the configuration |
| UIDs | Documentation of the users used in the configuration |
| GIDs | Documentation of the groups used in the configuration |
Contributions howto
Repository source
You'll find the Operations repository at https://devcentral.nasqueron.org/source/operations/
If DevCentral isn't available, a mirror of the repository can be found at https://github.com/nasqueron/operations.
Workflow of contributions
We follow the general workflow is described at How to contribute code.
Before committing a change to the main branch, you can test a deployment on a server. In that case, merge in main immediately after the deployment as the repository is the source of truth for the server state.
Log what you do on #nasqueron-ops, so it will be included at https://infra.nasqueron.org/servers-log/
Code conventions
See also: Code conventions.
- Python code: run `black` before committing
- YAML: indent with two spaces, including list bullets
id:
method:
- somekey: value
- otherkey: value
Environment setup
Use of WindRiver is recommended if you wish to get an environment already installed for your contributions.
Everywhere setup
This section cover topics needed both on our development servers or in your laptop.
Python dependencies
Create a virtual environment, then install dependencies from rOPS: requirements.txt:
mkdir -m $HOME/dev/python/ops python3 -m venv $HOME/dev/python/ops source $HOME/dev/python/ops/bin/activate pip install -r requirements.txt
Datacube
If you're in the ops group, you'll need sometimes to deploy from branches. As our deployment server Complector doesn't have PHP installed (so no arc), you need to exchange branches through our datacube.
On WindRiver:
git remote add datacube /datacube/git/operations.git
Anywhere else:
git remote add datacube ssh://windriver.nasqueron.org/datacube/git/operations.git
If you aren't in the ops groups, and have access to our devserver, you'll still be able to clone or pull from that directory, but not to write there (ie push branches).
Local computer setup
This section cover topics NOT needed on our development servers.
Arcanist
To use arc with the operations repository, our collection of patches is needed.
If arcanist isn't installed through Git repository, best is to do like on devserver role:
mkdir /opt/phabricator && cd /opt/phabricator git clone https://github.com/nasqueron/arcanist.git git checkout production
If you've already cloned Arcanist repository, switch to our branch production:
git remote add nasqueron git@github.com:nasqueron/arcanist.git git fetch --all git checkout production
As of August 2024, there is no plan to make that repository available on DevCentral.
IDE configuration
The states .sls file are Jinja2 templates producing YAML files, or "jinja2+yaml". If this case can't be handled by your IDE, it works best to use YAML syntax highlighting.
If you use PyCharm or IntelliJ IDEA Ultimate, you can configure this scheme:
- Allow YAML files to be Jinja2 templates
- Go to Settings > Languages & Frameworks > Template Languages
- Select Jinja2 as template language
- Add YAML in the list
- Open SLS as YAML
Linting
pre-commit
You can use pre-commit you can install with make to get a hook running Python code to lint the repository before a commit.
Beware if you use arc diff without a commit first: your message will be lost if pre-commit raises an error, always use git commit first.
On FreeBSD, there is an issue to build with clang 16+ the ruamel yaml clib library. You can build the FreeBSD port devel/py-ruamel.yaml.clib and copy the .whl to a stable path, then install it in your virtual environment.
On WindRiver, the .whl is available in /opt/python/py311, so you can:
$ python3 -m venv /path/to/your/virtualenv $ source /path/to/your/virtualenv/bin/activate $ pip install /opt/python/py311/ruamel.yaml.clib-0.2.8-cp311-cp311-freebsd_14_0_release_p3_amd64.whl $ pip install pre-commit $ make pre-commit install pre-commit installed at .git/hooks/pre-commit $ rehash $ pre-commit
Refactoring
Rename a lot of files
To move a directory and rename the Source file headers:
git mv roles/webserver-legacy/nginx/files roles/webserver-alkane/nginx/ cd roles/webserver-alkane/nginx/ find . -type f -name '*.conf' | xargs gsed -i s/webserver-legacy/webserver-alkane/g
