Network Presence Operations

A recent power system transition to our UPS batteries revealed a problem in the dual feed/redundant power supply of the VPS Hosting Server known as Node 100 in our Adelaide POP.

We’ve got a replacement for this PSU (power supply unit) of Node 100 in stock in our Adelaide office, so an outage notice will soon be posted for Node 100 to do this PSU replacement work on it.

Until then its operating in a non-redundant manner and we’ll schedule these hardware works to occur in our next slot after the coming works in our Sydney POP next week now.

Our Adelaide POP has experienced a grid power failure and is currently running on battery based UPS and is expected to be returning to the mains based power in 3 hours this evening, with the grid power lost at midnight on the evening of Monday November 13th.

Unfortunately VPS Hosting Node 100 appears to have suffered a power event in the transition to UPS battery power and is currently offline while investigations continue as to its issues.

All other Nodes and services at the Adelaide POP are still operational and online.

Updates will be posted as comments to this post as the situation evolves.

Our Internet connectivity suppliers in our main Sydney Data Centre POP have been doing updates (mostly software based) to their Sydney based routers and networks, and though we pay for a supposedly redundant service there with multiple physically separate fibre links, we saw a short issue across both links at the same time late one night last week, at the time window when the supplier had advised us of the works, but said we’d have no downtime.

They have more network reconfiguration work further in their core networks in Australia tonight and again they advise no downtime, but I’m not convinced.

There could be short Internet connectivity breaks to some Australian/domestic networks this evening as they update their main core routers software configuration, from 11pm through to 5am, with any outage expected to be short (minutes), on the night of Tuesday November 7th 2023 and the morning of Wednesday November 8th.

Over the late afternoon & evening of Tuesday November 21st 2023 and into the early morning of Wednesday the 22nd we will be performing upgrades to selected hosting server platforms & one of our network uplinks of Network Presence in Sydney, Australia.

Customers with services (VPS) on the hosting platforms to be upgraded may be emailed advisory notices, and this is a general notice related to other ancillary upgrades and Infrastructure Maintenance being performed during this scheduled window of time.

This Maintenance Window starts from 5pm on the Tuesday afternoon, through to 6am Sydney time on the morning of Wednesday November 22nd 2023, and I’ll be personally contactable by phone & online during and after these works in-case there’s any issues that arise.

During this Maintenance Window, selected network or server equipment will be worked on, upgraded and restarted. Works will be completed as quickly as possible (and where possible, timing information will be in emails to affected customers) and restarts are short service interruptions only, while upgrades may incur longer service interruptions.

This will incur outages to selected hosting services of Network Presence in Sydney, Australia during this off-peak scheduled works period. In addition, there may be some impact (short 5-10 minute periods) to the general Internet Connectivity of portions of Network Presence in Sydney during these works, though redundant routing will be utilised during these works.

Our Adelaide POP is unaffected by this Maintenance Notice and works.

Our Operations Site at ops.networkpresence.co is also unaffected by these works and updates will be posted there as required.

Progress updates leading up to and during the works will be posted to our Operations Blog site at https://ops.networkpresence.co (public feed, no login required to view it).

FYI and regards,
Richard.

Our Shopfront / Shop site went down today and the root cause was that it needed more RAM/memory and a couple more CPUs to keep up with its usage demands.

It’s running much better now after getting double its previous RAM and 50% more CPU and we’re adding deeper probes into its website to our internal monitoring, so we’ll see this quicker next time.

Thanks,
Richard.

Node 110 in Adelaide is getting an upgrade, which will ~double its hosting capacity and increase the speed of Node 110 in general.

These physical upgrades to Node 110 are scheduled to begin from Friday night October 27th 2023 through to late on Sunday night of October 29th.

During this weekend Node 110 will be restarted some # of times and will be offline for short periods during physical works and restarts.

We’ll update this post with comments as this work gets underway.

Node 190 in Adelaide has suffered a PSU failure and is being bought back online with replacements this evening.

I’ll update this post in comments as Node 190 and its VPS load come back online during this evening as repairs continue.

FYI,
Richard.

We’ve got the first signs of a storage devices failing sometime soon in Node 10 in our Sydney POP.

Accordingly we’re starting the migration of customer VPS on that alerting storage and this is underway on Wednesday evenings and weekends.

Ideally Node 10 will be decommissioned due to migration completion in November 2023, but current migrations are only occurring from its affected storage arrays.

An unscheduled outage has occurred on our Adelaide POP’s Telstra Fibre link to the Internet from ~5:30am Adelaide time on the morning of Thursday September 21st 2023.

The fault is logged with Telstra and has been acknowledged by them to be a major outage affecting many thousands of customers, including our Adelaide POP’s fibre link and location.

I’ll update this post with more information as it comes to hand and as received from Telstra.

As mentioned here and in our Blog site, we’ve got more of our new latest generation of VPS Hosting Nodes online in our Sydney POP.

This means we’ll be migrating more customer VPS over to these new hosting nodes in Sydney.

These migrations can be done “live” for some customers, but otherwise there is a short stop/start cycle required for the migration procedure and these migrations will be enacted weekly during our “Operations Windows” ie: Wednesday nights and weekends.