Operations grimoire/DevCentral

DevCentral is the name of our Phabricator instance.

Security

CI access

Our Jenkins infrastructure uses two Jenkins server:

• Jenkins CI, a more open one for continuous integration purpose
• Jenkins CD, to deploy code from the main branch, with only trusted users access to run jobs

Jenkinsfile script can arbitrarily request the node tag they want, so specify secure and non secure nodes isn't enough since the Jenkins 2.0 migration. Thence the Jenkins CD creation, isolated from Jenkins CI.

The access is still secured from the era we only had one Jenkins instance:

• we maintain a group Trusted users with users trusted not to send malicious code to CI
• Herald rules triggering job should check Author's projects include all of Trusted users
• When creating a new build plan, it should only be visible to trusted users (it exposes a Jenkins token to trigger a new job)

We should consider to open more this, for example is Jenkins CI safe enough to exposes trigger new job links to anonymous access? If so, can we protect it against DoS?

Upgrade

First, upgrade the source code, pulling master for libphutil and rebasing production against origin/master for phabricator:

$ cd /opt/arcanist
$ git pull
$ cd /opt/phabricator
$ git fetch
$ git rebase origin/master

If there is any issue, solve the merge conflict for the production branch. Don't panic, all is compiled through opcache, you can safely and calmly solve it without impacting the website.

Now stop the services, apply database migrations and restart services:

$ bin/phd status
$ bin/phd stop
$ chpst -u app bin/storage upgrade -f
…
Done.                                                   
Completed applying all schema adjustments.
$ sv restart php-fpm
ok: run: php-fpm: (pid 8297) 0s
$ sv restart phd
ok: run: phd: (pid 8415) 0s

Check https://devcentral.nasqueron.org/ for installation issues.

If you aren't administrator, ask an admin to check (a yellow balloon popup appears with instructions where there is something to do).

Troubleshoot

SSH servers aren't running

We need two instances of sshd (none with default configuration file):

• hosting: /usr/sbin/sshd -f /etc/ssh/sshd_phabricator_hosting_config (container port 22, world port 5022)
• staging: /usr/sbin/sshd -f /etc/ssh/sshd_staging_config (port 7022)

devcentral.nasqueron.org port 5022: Connection refused

$ git push
ssh: connect to host devcentral.nasqueron.org port 5022: Connection refused
fatal: Could not read from remote repository.

That requires two things:

• a SSH server launched on the port 22 of the devcentral Docker container, to serve repositories (not a staging area): http://pad.wolfplex.be/p/DevCentral
• an iptables rule to forward ports: iptables -t nat -I PREROUTING -i ens192 -p TCP -d 51.255.124.10/32 --dport 5022 -j DNAT --to-destination 172.17.0.23:22
• if the IP changed, check with iptables -t nat -L PREROUTING an old entry (5022 is "mice"):
  • To remove the old: iptables -t nat -D PREROUTING -i ens192 -p TCP -d 51.255.124.10/32 --dport 5022 -j DNAT --to-destination 172.17.0.139:22
  • To add the new: iptables -t nat -I PREROUTING -i ens192 -p TCP -d 51.255.124.10/32 --dport 5022 -j DNAT --to-destination 172.17.0.23:22
  • To check the rules: iptables -t nat -L PREROUTING | grep mice

2018-11-10: current IP has been moved to 172.17.0.23

Database flood

Some tables can be heavily flooded:

• devcentral_daemon.daemon_logevent — 7M lines
• devcentral_conduit.conduit_methodcalllog — 23M lines

Just truncate them.

A recent SSH key change doesn't allow VCS access

When an user SSH key is updated in settings, the authentication script uses a cached version of the information.

To troubleshoot that, ask user to use `ssh -p 5022 vcs@devcentral.nasqueron.org`. If a permission denied is returned and keys are the same locally and on devcentral, check on the container:

$ docker exec -it devcentral bash
$ scripts/ssh/ssh-auth.php | grep <username>

Former cache issue has been fixed 2025-10-24: cache is correctly invalidated when a new key is added.

Login support

What external provides I use?

To get the list of accounts and the way to authenticate (e.g. Google, GitHub), you can execute this query:

   $ bin/storage shell
   \u devcentral_user
   SELECT u.userName, e.accountType FROM user u, user_externalaccount e WHERE e.userPHID = u.phid ORDER BY u.userName ASC;

Or for one user, with more details (per GRPD policy, strictly only usable for support):

   $ bin/storage shell
   \u devcentral_user
   SELECT accountType accountDomain, accountID, e.username, email, accountURI FROM user_externalaccount e, user u WHERE e.userPHID = u.phid AND u.userName = "";

2FA transition

Use case: an user gets a new device and create a new token there instead of import them, adds the new token on Phabricator and, SURPRISE, BOTH are now required. A surprise after the old token has been deleted.

   $ bin/storage shell
   \u devcentral_auth
   SELECT c.id, c.factorName, c.dateCreated, c.dateModified FROM auth_factorconfig c, devcentral_user.user u WHERE c.userPHID = u.phid and u.username = "Dereckson"; -- the oldest should have the lower id, but dates are also provided
   DELETE FROM auth_factorconfig WHERE id = 0; -- the ID of the oldest

Use common sense for that kind of requests: as the addition of the second factor is authenticated in high security by the use of the first factor, the request is pretty safe as long as you keep a factor, but if the request is to remove everything, it requires extra verification, ask to meet the requester on Jitsi. If nobody knows them, ask also them to authenticate through their SSH key or something.