View source for Operations grimoire/DevCentral

'''DevCentral''' is the name of our Phabricator instance.

== Security ==
=== CI access ===
Our Jenkins infrastructure uses two Jenkins server:
* Jenkins CI, a more open one for continuous integration purpose
* Jenkins CD, to deploy code from the main branch, with only trusted users access to run jobs

Jenkinsfile script can arbitrarily request the node tag they want, so specify secure and non secure nodes isn't enough since the Jenkins 2.0 migration. Thence the Jenkins CD creation, isolated from Jenkins CI.

The access is still secured from the era we only had one Jenkins instance:
* we maintain a group [https://devcentral.nasqueron.org/project/members/50/ Trusted users] with users trusted not to send malicious code to CI
* Herald rules triggering job should check ''Author's projects include all of Trusted users''
* When creating a new build plan, it should only be visible to trusted users (it exposes a Jenkins token to trigger a new job)

We should consider to open more this, for example is Jenkins CI safe enough to exposes trigger new job links to anonymous access? If so, can we protect it against DoS?

== Upgrade ==

First, upgrade the source code, pulling master for libphutil and rebasing production against origin/master for phabricator:
<syntaxhighlight lang="console">
$ cd /opt/arcanist
$ git pull
$ cd /opt/phabricator
$ git fetch
$ git rebase origin/master
</syntaxhighlight>

If there is any issue, solve the merge conflict for the production branch. Don't panic, all is compiled through opcache, you can safely and calmly solve it without impacting the website.

Now stop the services, apply database migrations and restart services:

<syntaxhighlight lang="console">
$ bin/phd status
$ bin/phd stop
$ chpst -u app bin/storage upgrade -f
…
Done.                                                   
Completed applying all schema adjustments.
$ sv restart php-fpm
ok: run: php-fpm: (pid 8297) 0s
$ sv restart phd
ok: run: phd: (pid 8415) 0s
</syntaxhighlight>

Check https://devcentral.nasqueron.org/ for installation issues.

If you aren't administrator, ask an admin to check (a yellow balloon popup appears with instructions where there is something to do).

==  Troubleshoot ==
=== SSH servers aren't running ===
We need two instances of sshd (none with default configuration file):
* hosting: <code>/usr/sbin/sshd -f /etc/ssh/sshd_phabricator_hosting_config</code> (container port 22, world port 5022)
* staging: <code>/usr/sbin/sshd -f /etc/ssh/sshd_staging_config</code> (port 7022)

=== devcentral.nasqueron.org port 5022: Connection refused ===
<syntaxhighlight lang="console">
$ git push
ssh: connect to host devcentral.nasqueron.org port 5022: Connection refused
fatal: Could not read from remote repository.
</syntaxhighlight>

That requires two things:
* a SSH server launched on the port 22 of the devcentral Docker container, to serve repositories (not a staging area): http://pad.wolfplex.be/p/DevCentral
* an iptables rule to forward ports: <code> iptables -t nat -I PREROUTING -i ens192 -p TCP -d 51.255.124.10/32 --dport 5022 -j DNAT --to-destination 172.17.0.23:22</code>
* if the IP changed, check with <code>iptables -t nat -L PREROUTING</code> an old entry (5022 is "mice"):
** To remove the old: <code>iptables -t nat -D PREROUTING -i ens192 -p TCP -d 51.255.124.10/32 --dport 5022 -j DNAT --to-destination 172.17.0.139:22</code>
** To add the new: <code>iptables -t nat -I PREROUTING -i ens192 -p TCP -d 51.255.124.10/32 --dport 5022 -j DNAT --to-destination 172.17.0.23:22</code>
** To check the rules: <code>iptables -t nat -L PREROUTING | grep mice</code>

2018-11-10: current IP has been moved to 172.17.0.23

=== Database flood ===
Some tables can be heavily flooded:
* devcentral_daemon.daemon_logevent — 7M lines
* devcentral_conduit.conduit_methodcalllog — 23M lines

Just truncate them.

=== A recent SSH key change doesn't allow VCS access ===
When an user SSH key is updated in settings, the authentication script uses a cached version of the information.

To troubleshoot that, ask user to use `ssh -p 5022 vcs@devcentral.nasqueron.org`. If a permission denied is returned and keys are the same locally and on devcentral, check on the container:

<syntaxhighlight lang="console">
$ docker exec -it devcentral bash
$ scripts/ssh-auth | grep <username>
</syntaxhighlight>

If the key isn't there, purge the ''general'' cache:

<syntaxhighlight lang="console">
$ bin/cache purge --caches general
</syntaxhighlight>

It's not needed to restart phd or php-fpm (and it won't affect anything if you do, that's not an OPCache service, but an user data cache).