Up until now I have been running 3 Raspberry Pi 4 systems all held in a metal frame with fans which makes a nice neat setup. One Pi does the home automation, one runs pi-hole (really useful!), and one is a server and has an SSD attached. Not long ago while we were out of the country (of course) the website hosted by the server failed. I did not have remote ssh access set up nor a VPN for access. When we got back home the pi had lost the filesystem on the SSD. The disk was still mounted, but not accessible. Being a server all logging was on the SSD so no errors were caught. A reboot was the only way to cure it.
I thought it was a one off until it happened again, this time while I was nearby. After that our broadband was upgraded to FTTP and with PlusNet giving a fixed IP and no blocks I moved my production websites and email server across to the Pi, saving the rental of the VPS I had been using up until then. The cost of the VPS covered the annual cost of the broadband so worked out well.
Then the Pi lost the SSD twice in three days. This time had got worse because on reboot it did not start Apache or fail2ban, even though they both started fine by hand once I had realised. So something is wrong in the setup and I cannot find out what.
So I pressed an old Lenovo miniPC into action and rebuilt the server onto it. One thing I learned ages ago was to document everything that goes onto the server (and indeed all my other systems) so I can simply run through the list, add everything back in and copy /home across. Relatively QED.
But this time round something caught me out. After everything was moved across to the Lenovo and IP addresses swapped so it became the server web access was still going to the Pi. It transpired that the PlusNet broadband router associates the port forwarding with a physical device, not a IP address. Easy enough to sort out via web access to the broadband hub, but one more thing to remember (and duly documented!)