Your experience has been improved, citizen.
Media almost all off of local storage and into cloud blob storage. I've reclaimed almost a TB of storage and offloaded lots of bandwidth.
Status available here, and will be available even when mastodon is down: https://nastatus.com
Jiggling the handle. Web interface and images may be temporarily broken while the transition completes. Will flush cache once finished. Mobile apps will probably work.
https://noagendasocial.com/status redirects there now as well, if it's easier to remember.
Status page for the server: https://nastatus.com/status.html
Any dudes-named-ben work at a SEIM provider?
I'm jiggling the server handle a little bit. Things may be a bit slow, but should continue to function. Might be a few #glitches
I am making changes to improve your experience, citizen. Remain calm and buy toilet paper.
Thank you Fedora, for not only EOLing CoreOS with three months notice, but for disabling the docker daemon from automatically starting on reboot. If you noticed the most recent downtime, shake your fist at RedHat.
Server Status - NAS
boosted
For those who want to ping me when there are issues (outside of NAS), my Twitter is: https://twitter.com/theeriner
Going to go offline for about 15m in the next hour or so that I can perform a full backup and test the fixes. Don't worry friends, I'm at the wheel and have only had a little bourbon.
Automatic update to the latest version of mastodon failed (partially) which is what caused the issue. I've rolled back and will update manually and attempt a fix to prevent this from happening again.
Fixed. Does someone want to build a cachet-monitor config so that I can create a cachet status page and get notifications when there are 500 errors (for authenticated queries?) Current monitoring only alerts me if the un-authenticated main page returns a non-200, which isn't always the case.
Sorry about that folks, an unexpected issue occurred at a time when I was not able to deploy a fix. Thanks for your patience, er, courage.
* Add new services to NAS
Future NAS services in consideration for deployment:
* Status pages (https://cachethq.io/)
* Pubic infrastructure documentation (https://github.com/rust-lang/mdBook)
* Secure pastebin (https://github.com/sametmax/0bin)
* TOR hidden service
* Resources permitting (low priority):
* free, invitation-only NA VPN (wireguard) server
* potential to host any future No Agenda projects made by producers (in docker containers, using CI+CD)
* Move from current kubernetes cluster to new cluster
The current cluster's operational patterns were created by me over two years ago. I haven't made many operational changes since then, although I did migrate physical machine hosts once. Now two years of development and experimentation down the road, my new system is significantly more mature.
This will be a big deal and will resolve much of the current operational overhead that I am currently burdened with.
* Implement CloudFlare Argo tunnel
This is a back-end change that should go unnoticed by everyone, but I would like to enable this for various reasons. To learn what Argo tunnel is, see the link below:
* Investigating potential move to Pleroma to reduce server load
Mastodon is a fitting name, as it is as computationally heavy as a mastodon is physically heavy. I would like to move to Pleroma, but don't want to do a "fresh start" if it can be avoided.
If the migration script is not mature enough (or if Pleroma itself is not mature enough), I will put this off.
* Moving static content to S3
Sometime after the new year, I will be moving mastodon's static files to S3. Due to the moderate upload speed and the large volume of data, this will take approximately two days. During this time, I will keep the mastodon server online, but any images uploaded during this period will be lost.
Notice will be given before this migration begins.
Upcoming changes to the mastodon server:
* Moving static content to S3
* Investigating potential move to Pleroma
* Implement CloudFlare Argo tunnel
* Move from current kubernetes cluster to new cluster
* Add new service to NAS
See thread for details.
I am aware that NAS network speeds have become sluggish, steps have been taken to resolve the issue. I expect full remediation within 24-48 hours. TYFYC
