https://agora.nasqueron.org/index.php?action=history&feed=atom&title=Operations_grimoire%2FIncidents%2F2018-12-08-Mastodon 2026-04-25T23:52:00Z Revision history for this page on the wiki MediaWiki 1.46.0-alpha https://agora.nasqueron.org/index.php?title=Operations_grimoire/Incidents/2018-12-08-Mastodon&diff=828&oldid=prev 2018-12-09T02:34:30Z

<p><span class="autocomment">Actionables: </span> No note here</p> <table style="background-color: #fff; color: #202122;" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 02:34, 9 December 2018</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l19">Line 19:</td> <td colspan="2" class="diff-lineno">Line 19:</td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* Report issue upstream (Mastodon and ffmpeg?)</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* Report issue upstream (Mastodon and ffmpeg?)</div></td></tr> <tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* Provide a watchdog to kill ŝtuck ffmpeg process (script is done, but a cron to run it is still needed)</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* Provide a watchdog to kill ŝtuck ffmpeg process (script is done, but a cron to run it is still needed)</div></td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;"></del></div></td><td colspan="2" class="diff-side-added"></td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;">;Notes</del></div></td><td colspan="2" class="diff-side-added"></td></tr> <tr><td class="diff-marker" data-marker="−"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;">&lt;references group="Note" /&gt;</del></div></td><td colspan="2" class="diff-side-added"></td></tr> </table>

Dereckson https://agora.nasqueron.org/index.php?title=Operations_grimoire/Incidents/2018-12-08-Mastodon&diff=827&oldid=prev 2018-12-09T02:34:15Z

<p>Created page with &quot;Tracked at https://devcentral.nasqueron.org/T1492. == Incident timeline == * ffmpeg process are stuck in the queue and don&#039;t allow other jobs to run * Several people report t...&quot;</p> <p><b>New page</b></p><div>Tracked at https://devcentral.nasqueron.org/T1492.<br /> <br /> == Incident timeline ==<br /> * ffmpeg process are stuck in the queue and don&#039;t allow other jobs to run<br /> * Several people report the issue by DM, but of course, timeline isn&#039;t updated, including the notifications one<br /> * 2018-12-08 11:35 Investigation to see what happens on the queue<br /> * 2018-12-08 PM The stuck ffmpeg processes was killed<br /> * 2018-12-08 PM The queue started to resorbed<br /> * 2018-12-08 PM Workers capacity has been increased<br /> * 2018-12-08 17:17 Prepare a script to kill stuck ffmpeg:&lt;br /&gt;{{Ops file|roles/paas-docker/containers/files/mastodon/clear-video-queue.py}}<br /> * 2018-12-08 PM The queue resorbed more quickly<br /> * 2018-12-08 19:46 The situation was fixed<br /> <br /> == Analysis ==<br /> There is a clear monitoring issue here: we need to get an alert if the queue contains more than some hundred jobs.<br /> <br /> == Actionables ==<br /> * Monitor Sidekiq queue<br /> * Report issue upstream (Mastodon and ffmpeg?)<br /> * Provide a watchdog to kill ŝtuck ffmpeg process (script is done, but a cron to run it is still needed)<br /> <br /> ;Notes<br /> &lt;references group=&quot;Note&quot; /&gt;</div>

Dereckson