Update: it's the network!
I'm running a cronjob that logs the default route and then does a traceroute to an external host.
2022-07-01 04:08:42 UTC: external ping monitor detected the VPS went offline
2022-07-01 11:03:13 UTC: VPS rebooted from control panel
Once it was back up and running, I examined the log.
Here's the last entry before it went offline:
2022-07-01, 04:05:01 UTC:
2a02:180:6:1::659 dev eth0 proto kernel metric 256 pref medium
fd78:232c:fe45:934b::1 dev eth0 metric 1024 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fd78:232c:fe45:934b::1 dev eth0 metric 1024 pref medium
Start: 2022-07-01T04:05:01+0000
HOST: eu2 Loss% Snt Last A
vg Best Wrst StDev
1. AS??? fd78:232c:fe45:934b::1 0.0% 10 0.2 0.2 0.2 0.2 0.0
2. AS35366 2a02-0180-0006-0001-0000-0000-0000-0001.blue.kundencontroller.de (2a02:180:6:1:: 0.0% 10 3.3 24.6 0.7 178.3 54.6
3. AS35366 po161.ipv6.bbsw-h2-j1a.as35366.net (2a02:180:6:9::6) 0.0% 10 3.4 6.3 0.8 38.0 11.4
4. AS35366 po205.ipv6.bbsw-h4a-fra.as35366.net (2a02:180:6:7::16) 0.0% 10 10.6 18.3 7.4 51.8 15.6
5. AS174 2001:978:2:42::e8:1 0.0% 10 19.0 14.4 8.3 27.3 6.5
6. AS174 te0-6-0-1.agr21.fra06.atlas.cogentco.com (2001:550:0:1000::9a19:175) 0.0% 10 22.9 36.1 7.6 146.2 41.5
7. AS174 multi-use.cogentco.com (2001:550:0:1000::8275:d) 80.0% 10 9.6 29.2 9.6 48.9 27.8
8. AS174 be2846.ccr42.fra03.atlas.cogentco.com (2001:550:0:1000::9a36:251d) 70.0% 10 19.0 12.6 8.6 19.0 5.6
9. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
10. AS174 be2154.rcr22.ams06.atlas.cogentco.com (2001:550:0:1000::8275:32ce) 10.0% 10 22.7 24.9 15.6 48.0 10.4
11. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
12. AS12876 2001:bc8:400:100::b6 70.0% 10 15.1 17.1 15.1 21.3 3.6
13. AS12876 2001:bc8:1404:104::1 0.0% 10 16.0 33.5 16.0 87.7 21.8
14. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
15. AS12876 2001:bc8:1404:104::4 0.0% 10 23.1 27.6 15.2 71.0 18.1
16. AS12876 2001:bc8:1830:201::1 0.0% 10 33.1 29.9 15.2 61.8 13.8
From then on, the log shows that the traceroute failed. This proves the VPS was powered on and running, but it had no network access. The network gateway was replying to pings, but not routing.
2022-07-01, 04:10:01 UTC:
2a02:180:6:1::659 dev eth0 proto kernel metric 256 pref medium
fd78:232c:fe45:934b::1 dev eth0 metric 1024 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fd78:232c:fe45:934b::1 dev eth0 metric 1024 pref medium
Start: 2022-07-01T04:10:02+0000
HOST: eu2 Loss% Snt Last Avg Best Wrst StDev
1. AS??? fd78:232c:fe45:934b::1 0.0% 10 0.2 0.4 0.1 1.5 0.5
2. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
Pretty conclusive that the problem is with the gateway, right?