Server Status - NAS @[email protected]

Sorry about that folks, an unexpected issue occurred at a time when I was not able to deploy a fix. Thanks for your patience, er, courage.

* Add new services to NAS

Future NAS services in consideration for deployment:

* Status pages (https://cachethq.io/)
* Pubic infrastructure documentation (https://github.com/rust-lang/mdBook)
* Secure pastebin (https://github.com/sametmax/0bin)
* TOR hidden service
* Resources permitting (low priority):
* free, invitation-only NA VPN (wireguard) server
* potential to host any future No Agenda projects made by producers (in docker containers, using CI+CD)

* Move from current kubernetes cluster to new cluster

The current cluster's operational patterns were created by me over two years ago. I haven't made many operational changes since then, although I did migrate physical machine hosts once. Now two years of development and experimentation down the road, my new system is significantly more mature.

This will be a big deal and will resolve much of the current operational overhead that I am currently burdened with.

* Implement CloudFlare Argo tunnel

This is a back-end change that should go unnoticed by everyone, but I would like to enable this for various reasons. To learn what Argo tunnel is, see the link below:

https://www.cloudflare.com/products/argo-tunnel/

* Investigating potential move to Pleroma to reduce server load

Mastodon is a fitting name, as it is as computationally heavy as a mastodon is physically heavy. I would like to move to Pleroma, but don't want to do a "fresh start" if it can be avoided.

If the migration script is not mature enough (or if Pleroma itself is not mature enough), I will put this off.

* Moving static content to S3

Sometime after the new year, I will be moving mastodon's static files to S3. Due to the moderate upload speed and the large volume of data, this will take approximately two days. During this time, I will keep the mastodon server online, but any images uploaded during this period will be lost.

Notice will be given before this migration begins.

Upcoming changes to the mastodon server:

* Moving static content to S3

* Investigating potential move to Pleroma

* Implement CloudFlare Argo tunnel

* Move from current kubernetes cluster to new cluster

* Add new service to NAS

See thread for details.

Issue resolved.

I am aware that NAS network speeds have become sluggish, steps have been taken to resolve the issue. I expect full remediation within 24-48 hours. TYFYC

Issue identified, working on a fix.

Mastodon refuses to stop soft-crashing with no explication in the logs, thus it gets the sledgehammer:

!#/bin/bash
while true; do
start-msdn && sleep 14400 && stop-msdn
done

@ChrisWilson is the watchman.

After elasticsearch finishes, there will be a 15-30m downtime for a manual backup.

Elasticsearch is currently eating through statuses. Once it's finished (I have no ETA) it should be possible to search your statuses.

Anyone experiencing any bugs or issues since the upgrade?

Server mostly back. Will probably go up and down a few times over the remainder of the weekend.

Was down longer than initially planned as I'm in the process of enabling toot searching across the instance, which takes a considerable amount of time to index all the current accounts and statuses.

...and by a "a few minutes" I mean am hour or so.

Server going down for maintenance in a few minutes...

Mastodon 3.0 is out, and the instance will be taken down this weekend for extended maintenance. Will post once more before downtime begins.

NAS is moving off of some power-hungry Dell servers and onto a small-formfactor Ryzen builds with NVMe drives.

Next upstream mastodon release, I'll be taking the server down for a few hours. Need to make some hardware changes.

Will send another update before downtime. TYFYC.

Infrastructure changes have been made. Hopefully this change is transparent to everyone here and no one notices anything (except that things load a bit faster).

Please message me directly if you notice anything is broken.