Network Presence Operations

An unscheduled outage has occurred on our Adelaide POP’s Telstra Fibre link to the Internet from ~2:30pm Adelaide time on the afternoon of Tuesday March 26th 2024.

I’ll update this post with more information as it comes to hand and as received from both Telstra and the Internet Provider.

Update is that the fault is logged with the upstream Internet Provider and is not a physical fibre fault. It has been acknowledged by them to be a major outage affecting many thousands of customers in Adelaide, including our Adelaide POP’s Internet link and location.

Service has been back and stable for 10 minutes now at 3:40pm. We’ll wait for a good hour of normal service before calling this event closed, as we did have short returns of service (a couple of 5 minute periods) during the last 1/2 hour before the connectivity failed again.

Over the last few months we’ve upgraded and increased in capacity the UPS capabilities of our Adelaide POP.

We’ve now got some 10 hours of Adelaide POP operations available at any time on batteries should the mains/grid supply fail for such a time period (that’s never happened at our Adelaide POP, but we’re ready for it if it may) and that’s before our diesel generator in Adelaide starts up to power the POP & charge the batteries.

There’s been no power outages for our Adelaide POP servers during these upgrades and we look forward to more long term sustained power for our Adelaide POP.

We’ve had a large UPS fail at the Adelaide POP and on Wednesday February 7th 2024 electricians are attending to replace the failed component and restore our Adelaide POP to its long term large off-grid battery capacity.

The Adelaide POP will switch to generator power for some minutes on Wednesday February 7th from ~11am, with these transitions to generator expected to be short while the POP’s power is reconnected to this battery array.

I’ll update this post with comments as work proceeds on the day.

We’ve found an issue with VPS Backups being able to be performed on VPS Hosting Node 70 in Sydney.

This is now fixed and backups can be run for VPS on Node 70 again.

FYI and regards,
Richard.

An unscheduled outage has occurred on our Adelaide POP’s Telstra Fibre link to the Internet from ~1:30am Adelaide time on the morning of Tuesday December 12th 2023.

The fault is logged with Telstra and has been acknowledged by them to be a major outage affecting many thousands of customers, including our Adelaide POP’s fibre link and location.

I’ll update this post with more information as it comes to hand and as received from Telstra.

Node 83 running some VPS from our Shopfront site experienced a fault & was quickly restarted this afternoon (Thursday Dec 7th 2023).

Our monitoring system alerted us within moments of the event & we had it restarted within minutes from there & we’ve got some logs to analyse as to what happened wrt the fault that took the node offline.

We’ve got the first signs of a storage devices failing sometime soon in Node 8 in our Sydney POP.

Node 8 was already on our list for aging out of our fleet as part of our regular upgrade program, but it’s now come to the top of the list for migration away from and decommissioning.

Accordingly we’re starting the migration of customer VPS from Node 8 and this is underway on Wednesday evenings and weekends.

Ideally Node 8 will be decommissioned due to migration completion in early December 2023.

A recent power system transition to our UPS batteries revealed a problem in the dual feed/redundant power supply of the VPS Hosting Server known as Node 100 in our Adelaide POP.

We’ve got a replacement for this PSU (power supply unit) of Node 100 in stock in our Adelaide office, so an outage notice will soon be posted for Node 100 to do this PSU replacement work on it.

Until then its operating in a non-redundant manner and we’ll schedule these hardware works to occur in our next slot after the coming works in our Sydney POP next week now.

Our Adelaide POP has experienced a grid power failure and is currently running on battery based UPS and is expected to be returning to the mains based power in 3 hours this evening, with the grid power lost at midnight on the evening of Monday November 13th.

Unfortunately VPS Hosting Node 100 appears to have suffered a power event in the transition to UPS battery power and is currently offline while investigations continue as to its issues.

All other Nodes and services at the Adelaide POP are still operational and online.

Updates will be posted as comments to this post as the situation evolves.

Our Internet connectivity suppliers in our main Sydney Data Centre POP have been doing updates (mostly software based) to their Sydney based routers and networks, and though we pay for a supposedly redundant service there with multiple physically separate fibre links, we saw a short issue across both links at the same time late one night last week, at the time window when the supplier had advised us of the works, but said we’d have no downtime.

They have more network reconfiguration work further in their core networks in Australia tonight and again they advise no downtime, but I’m not convinced.

There could be short Internet connectivity breaks to some Australian/domestic networks this evening as they update their main core routers software configuration, from 11pm through to 5am, with any outage expected to be short (minutes), on the night of Tuesday November 7th 2023 and the morning of Wednesday November 8th.