Network Presence Operations

This may not justify a post, but after many years of constant operation and availability, one of our fibre uplinks into the Internet from our Sydney racks in Equinix SY1 Data Centre went down (full loss of light, not just data) at around 11am Sydney time on Wednesday October 9th 2024.

We have a 2nd and redundant fibre link to this Internet connectivity supplier in SY1 and our network seemlessly, quickly and automatically failed-over to the remaining running link, with no customer impact seen.

This multiple physically separate fibre connectivity model we run in Sydney has proved itself today, as it took nearly 2 hours for the upstream provider to rectify the problem with a line card in one of their routers.

Operating these separate fibre paths to the Internet is incredibly expensive compared to what’s on offer in Sydney nowadays, but we gladly pay those $thousands so that those 2 hours today were un-interrupted for our customers.

FYI,
Richard.

There was a scheduled maintenance outage of our Adelaide connectivity while local Adelaide based works were completed by our upstream provider in Adelaide.

This resulted in a 15 minute (as scheduled) outage of our Adelaide POP from around 1:30am on the morning of Thursday August 15th 2024.

Apologies for not posting this notice of these scheduled works before they occurred.

Xen is still a popular and well used virtualisation technology and we’re upgrading and migrating many of our Xen VPS customers to newer hardware and hypervisor environments, on faster disk sub-systems, to continue their service on our newer server platforms.

So we’ll be migrating these Xen VPS to their new hosting environments most weekends through to September 2024.

We’ll be in contact with some customers who may benefit from changing their VPS during these migrations, and if any customers that know they’re on Xen based VPS want to change their hosting, then please contact us by email to admin AT networkpresence.com.au

Thanks,
Richard.

An unscheduled outage has occurred on our Adelaide POP’s Telstra Fibre link to the Internet from ~2:30pm Adelaide time on the afternoon of Tuesday March 26th 2024.

I’ll update this post with more information as it comes to hand and as received from both Telstra and the Internet Provider.

Update is that the fault is logged with the upstream Internet Provider and is not a physical fibre fault. It has been acknowledged by them to be a major outage affecting many thousands of customers in Adelaide, including our Adelaide POP’s Internet link and location.

Service has been back and stable for 10 minutes now at 3:40pm. We’ll wait for a good hour of normal service before calling this event closed, as we did have short returns of service (a couple of 5 minute periods) during the last 1/2 hour before the connectivity failed again.

Over the last few months we’ve upgraded and increased in capacity the UPS capabilities of our Adelaide POP.

We’ve now got some 10 hours of Adelaide POP operations available at any time on batteries should the mains/grid supply fail for such a time period (that’s never happened at our Adelaide POP, but we’re ready for it if it may) and that’s before our diesel generator in Adelaide starts up to power the POP & charge the batteries.

There’s been no power outages for our Adelaide POP servers during these upgrades and we look forward to more long term sustained power for our Adelaide POP.

We’ve had a large UPS fail at the Adelaide POP and on Wednesday February 7th 2024 electricians are attending to replace the failed component and restore our Adelaide POP to its long term large off-grid battery capacity.

The Adelaide POP will switch to generator power for some minutes on Wednesday February 7th from ~11am, with these transitions to generator expected to be short while the POP’s power is reconnected to this battery array.

I’ll update this post with comments as work proceeds on the day.

We’ve found an issue with VPS Backups being able to be performed on VPS Hosting Node 70 in Sydney.

This is now fixed and backups can be run for VPS on Node 70 again.

FYI and regards,
Richard.

An unscheduled outage has occurred on our Adelaide POP’s Telstra Fibre link to the Internet from ~1:30am Adelaide time on the morning of Tuesday December 12th 2023.

The fault is logged with Telstra and has been acknowledged by them to be a major outage affecting many thousands of customers, including our Adelaide POP’s fibre link and location.

I’ll update this post with more information as it comes to hand and as received from Telstra.

Node 83 running some VPS from our Shopfront site experienced a fault & was quickly restarted this afternoon (Thursday Dec 7th 2023).

Our monitoring system alerted us within moments of the event & we had it restarted within minutes from there & we’ve got some logs to analyse as to what happened wrt the fault that took the node offline.

We’ve got the first signs of a storage devices failing sometime soon in Node 8 in our Sydney POP.

Node 8 was already on our list for aging out of our fleet as part of our regular upgrade program, but it’s now come to the top of the list for migration away from and decommissioning.

Accordingly we’re starting the migration of customer VPS from Node 8 and this is underway on Wednesday evenings and weekends.

Ideally Node 8 will be decommissioned due to migration completion in early December 2023.