Network Status

0 Open      0 Scheduled      31 Resolved

Emergency Maintenance Cloud VM's (Resolved)

Affecting System - Cloud VM's | Priority - Critical


pdate 31/01/2015 17:10am
-----------------------

All VM's should now be up, we will continue monitoring from our side.

Once again we apologize for this inconvenience.
We greatly appreciate your patience and support regarding this matter.

-----------------------

Update 31/01/2015 15:51am
-----------------------

The consistency check has completed.

We are now bringing up the remaining VM's. 


Once again we apologize for this inconvenience.
We greatly appreciate your patience and support regarding this matter.

-----------------------


Update 31/01/2015 14:51am
-----------------------

The consistency check is 75% complete.

We are still bringing up VM's 
Please do NOT attempt to start your VM yourself, please wait until we have brought your VM up.

Once again we apologize for this inconvenience.
We greatly appreciate your patience and support regarding this matter.

-----------------------

Update 31/01/2015 14:02am
-----------------------

The consistency is 50% complete.

We are slowly bringing up VM's.
Please do NOT attempt to start your VM yourself, please wait until we have brought your VM up.

Once again we apologize for this inconvenience.
We greatly appreciate your patience and support regarding this matter.

-----------------------

Update 31/01/2015 13:05am
-----------------------

The consistency check has picked up 143 vdisks which need to be resynced. It is currently on 30/143 vdisks complete.

We are going to slowly start bringing up VM's which don't fall into these 143, please do NOT attempt to start your VM yourself, please wait until we have brought your VM up.

Once again we apologize for this inconvenience.
We greatly appreciate your patience and support regarding this matter.

-----------------------

Update 31/01/2015 13:05am
-----------------------

Diagnostics are looking good, we now running consistency checks on VM data, please do NOT attempt to start your VM at this time as this may cause data loss.

-----------------------

Update 31/01/2015 12:53am
-----------------------

All Cloud storage nodes have been rebooted, we are now running some diagnostics, please do NOT attempt to start your VM at this time as this may cause data loss.

-----------------------

Update 31/01/2015 12:28am
-----------------------

The Cloud storage is currently being rebooted.

-----------------------

Update 31/01/2015 11:50am
-----------------------

All VM's have been stopped, we are now proceeding with shutting down the storage controllers and preparing for rebooting.

-----------------------

Update: 31/01/2015 10.50am
-----------------------

The VM's are currently being shutdown 3 at a time, 40% through this process.

-----------------------


Start: 31/01/2015 10.20am
-----------------------
Good Day,

Earlier this week we picked up some anomolies on our Cloud storage. We were advised by OnApp (The Vendor) that an upgraded was needed to be performed in order to resolve the issues.

This upgrade has failed and our Cloud storage needs to be completely rebooted.

 - We are going to proceed with shutting down all cloud VM's to protect their data.
 - Once all VM's have been stopped, the cloud storage will be rebooted and consistency checks will be done.
 - Once the consistency checks have completed we will proceed with booting up the VM's again.

We sincerly apologize for this and are currently looking at alternative ways of avoiding situations like this in the future.
-----------------------



Date - 31/01/2016 10:17 - 31/01/2016 18:30
Last Updated - 02/02/2016 20:54

International Access (Resolved)

Affecting Other - Data Center and ADSL | Priority - Critical


There are widespread outages on International access and some local ADSL access.
Initial reports are showing outages at NAP Africa, Mweb, Vodacom, CrystalWeb, Vox Teleom.

More information to follow.

We have received the following From Vox Telecom:
Vox Telecom services are currently impacted as a result of a failure with the upstream network providers, SEACOM and WACS. Upstream provider engineers are currently attending.

update: 
Break in SEACOM identified in Egypt. NO ETTR at present

WACS - 
The patch tails at Global Switch have been tested & levels are within limit , however there is still a LOS at the first ILA from London on the main route (Gerards Cross).
The engineer is still busy on site. No ETTR as yet.

To note:
Traffic cannot be rerouted for Vox Telecom - there is no alternative
Date - 21/01/2016 16:25
Last Updated - 22/01/2016 09:57

Scheduled Maintenance - Cloud Platform (Resolved)

Affecting Server - manage.cloud.net.za | Priority - Low


Tonight (02 April 2015) from 21:00 there will be scheduled maintenance to our cloud platform.
Our cloud software will be upgraded to the latest version. This however does involve rolling restarts across all Hypervisors.
During these rolling restarts your Virtual Machine(s) will be auto-migrated to other Hypervisors to ensure your Virtual Machine remains available.


Date: 02 April 2015
Time: 21:00 - 07:00 SAST (GMT+2)
Duration: 10 Hours
Purpose : Routine Software Updates.
Expected Impact : Minimum, there may be very brief intermittent loss of connectivity to Virtual Machines while they are being switched between hypervisors during the auto-migrations.

 

Date - 02/04/2015 21:00 - 10/05/2015 14:50
Last Updated - 02/04/2015 08:21

Vox Outage (Resolved)

Affecting Other - Vox Telecom | Priority - Critical


Vox is experiencing an Outage on their DC3 Facility, Engineers are attending.


Update: 
We have switched over to our redundant links and all traffic is now being routed through our DC4 Facility.
Date - 17/02/2015 10:45 - 17/02/2015 12:30
Last Updated - 18/02/2015 12:38

VOX BGP issues (Resolved)

Affecting Other - VOX BGP issues | Priority - Critical


VOX are having BGP issues on their network, we are addressing this with them now and will have the issue resolved soonest


Date - 17/02/2015 10:07 - 17/02/2015 11:47
Last Updated - 17/02/2015 11:47

Intermittent Fibre connectivity / routing bet (Resolved)

Affecting System - Fibre connectivity between DC3 & DC4 | Priority - Medium


Intermittent Fibre connectivity / routing between our DC3 & DC4 affecting Cloud, VPS servers and Reseller Hosting, infrastructure department working on it

UPDATE:

10h30
Infrastructure department found a break in the fibre cable between DC3 and DC4 hence loss of connectivity, a temporary solution has been implemented that has resolved the issue. Infrastructure will now look at a permanent resolution.

UPDATE:

14h00
New Fibre cable installed, routing reset back to normal

Issue Resolved

Date - 10/02/2015 09:19
Last Updated - 10/02/2015 16:45

Websites Not Displaying (Resolved)

Affecting Server - | Priority - Critical


Websites are not displaying on the cp02.cphost.co.za webserver and server load is high.
Engineers are busy investigating.

Date - 16/01/2015 10:43 - 16/01/2015 11:36
Last Updated - 16/01/2015 10:44

ZACR EPP System DB Issues (Resolved)

Affecting Other - ZACR EPP Database | Priority - Critical


ZACR is currently having load issues on their EPP Database which is causing "Failed to Connect" error messages when trying to perform any domain registrations, transfers or renewals.

Engineers are attending to the issue on their side, no ETA has been provided.

Date - 13/12/2014 14:02 - 16/01/2015 10:43
Last Updated - 13/12/2014 17:04

High Network Latency (Resolved)

Affecting Other - Network | Priority - High


Both Primary and Secondary Dark Fibre Africa links between Teraco and Rutherford Estate are down.
Traffic is being routed through a failover 1Gbit link, due to this there is a high latency on the network.

DFA is currently attending to the problem.

Date - 15/11/2014 07:43 - 15/11/2014 09:30
Last Updated - 19/11/2014 11:06

Phone Lines Down due to Power Failure (Resolved)

Affecting Other - Domains Offices | Priority - Critical


The phone lines at our offices are currently down until after due to an Eskom power failure in the Highlands North Johannesburg Area.


Date - 04/11/2014 08:39 - 04/11/2014 14:09
Last Updated - 15/11/2014 07:52

Server Status

Below is a real-time overview of our servers where you can check if there's any known issues.


Server Status
Server Name HTTP FTP POP3 Server Load Uptime
cp1.domains.co.za Loading... Loading... Loading... Loading... Loading...
cp2.domains.co.za Loading... Loading... Loading... Loading... Loading...
cp3.domains.co.za Loading... Loading... Loading... Loading... Loading...
cp4.domains.co.za Loading... Loading... Loading... Loading... Loading...
cp5.domains.co.za Loading... Loading... Loading... Loading... Loading...
cp6.domains.co.za Loading... Loading... Loading... Loading... Loading...
cp7.domains.co.za Loading... Loading... Loading... Loading... Loading...
cp8.domains.co.za Loading... Loading... Loading... Loading... Loading...
cp9.domains.co.za Loading... Loading... Loading... Loading... Loading...
r01.cphost.co.za Loading... Loading... Loading... Loading... Loading...
r03.cphost.co.za Loading... Loading... Loading... Loading... Loading...
r04.cphost.co.za Loading... Loading... Loading... Loading... Loading...
r05.cphost.co.za Loading... Loading... Loading... Loading... Loading...
r06.cphost.co.za Loading... Loading... Loading... Loading... Loading...
rs1.cphost.co.za Loading... Loading... Loading... Loading... Loading...
rs2.cphost.co.za Loading... Loading... Loading... Loading... Loading...
rs3.cphost.co.za Loading... Loading... Loading... Loading... Loading...
rs4.cphost.co.za Loading... Loading... Loading... Loading... Loading...
rs5.cphost.co.za Loading... Loading... Loading... Loading... Loading...