Network Status

0 Open      0 Scheduled      62 Resolved

cp16.domains.co.za Maintenance (Resolved)

Affecting Server - cp16.domains.co.za | Priority - High


We are scheduling quick maintenance on cp16.domains.co.za
The server will be unresponsive during the maintenance run.

Apologies for any inconvenience caused.



Update: Maintenance has been done and the server is up and running.

Date - 29/08/2020 20:00 - 29/08/2020 20:15
Last Updated - 29/08/2020 20:09

Critical Maintenance (Resolved)

Affecting Server - wp1.hosted.site | Priority - Critical


Maintenance has been scheduled for wp1.hosted.site
Please expect downtime of about 30 - 45 minutes during maintenance.

Apologies for any inconvenience caused.
Date - 17/08/2020 20:00 - 17/08/2020 20:45
Last Updated - 17/08/2020 20:15

Email issues with Gmail (Resolved)

Affecting Other - Email Traffic | Priority - Critical


We are aware of an issue affecting slow delivery of emails to gmail.com

Currently it looks like Google is randomly rejecting connections from our Spamfilter cluster.
We are engaging with them to try and determine why this is happening and why all of a sudden.
Currently we do not have an ETA from Google's side.

UPDATE: 
we have switched our spamfilters over to use IPv6 delivery to gmail, this seems to be helping the situation and the backed up email queues on our side are dropping slowly.

Date - 03/06/2020 09:02 - 18/06/2020 17:34
Last Updated - 03/06/2020 11:05

29 May 2020 - Scheduled Downtime - Epyc VPS (Resolved)

Affecting System - Epyc VPS ndoes | Priority - High



On Friday the 29th May 2020 between 23:00 - 01:00 we will be bringing down the following VPS nodes for a BIOS upgrade:

node01.vps.net.za
node02.vps.net.za
node03.vps.net.za
node04.vps.net.za

This BIOS upgrade is to resolve a NUMA malfunction currently experienced in Windows VPS's that use more than 4 vCPU's.

All other Epyc VPS nodes above node04 are currently running on this newer BIOS and will not be shutdown for any upgrades.
Date - 29/05/2020 23:00 - 30/05/2020 01:00
Last Updated - 30/05/2020 09:41

RS4 Down (Resolved)

Affecting Server - rs4.cphost.co.za | Priority - Critical


Currently experiencing connectivity issues on rs4.cphost.co.za

Infrastructure is busy investigating the issue.

We will provide feedback as soon as the investigation is done.

**Update**

Services has been restored successfully.

Date - 18/03/2020 20:41
Last Updated - 18/03/2020 21:44

mail1.hosted.site Upgrade (Resolved)

Affecting Server - mail1.hosted.site | Priority - High


A memory upgrade has been scheduled for mail1.hosted.site for tonight.
Downtime is for about 1 hour.

Apologies for the inconvenience caused.

Update :

Shutting server down to start the upgrade.

Update :

Upgrade complete, booting up server.

Update :

Server up and running again.

Resolved.

Date - 18/03/2020 20:00 - 18/03/2020 21:00
Last Updated - 18/03/2020 20:07

Critical Maintenance (Resolved)

Affecting Server - rs2.cphost.co.za | Priority - Critical


We have detected that a drive has failed on rs2.cphost.co.za

The failed drive will be replaced as per scheduled time.
Server might be unresponsive during that time.

 

Apologies for any inconvenience caused.

All resolved, drive has been replaced.

Date - 02/03/2020 20:00 - 02/03/2020 23:00
Last Updated - 03/03/2020 07:48

ZACR CO.ZA Database issue (Resolved)

Affecting System - ZACR | Priority - Critical


ZACR is currently expericing Database issues on their CO.ZA EPP Platform.

We have informed that about this and they are currently investigating.

No ETR provided yet.
Date - 28/01/2020 11:40 - 29/02/2020 09:16
Last Updated - 28/01/2020 11:41

Critical Maintenance (Resolved)

Affecting Server - rs18.cphost.co.za | Priority - Critical



We detected that one of the drives has failed on our reseller server via our server logs.
The server is rs18.cphost.co.za (169.239.217.28)

The drive will be replaced this evening (27/01/2020) at 20:00 to minimize downtime, during this time the server will be non-responsive.

We will keep the status page updated during the period of the drive replacement.
Please see herewith a link to our Status page
https://www.domains.co.za/serverstatus.php
 
Our apologies for any inconvenience caused during this period.

************

 

The failed drive has been replaced and all is operational again.

Date - 27/01/2020 20:00 - 28/01/2020 00:00
Last Updated - 27/01/2020 20:58

DDoS Attack (Resolved)

Affecting System - Network Infrastructure | Priority - Critical


We are currently being targeted in a DDoS attack again. Our upstream providers and engineers are working on mitigating the attack.

These are the same DDoS attacks which have made the news recently and affected other providers like RSAWEB, Cool Ideas, Bitco, Liquid Telecom, Web Africa, Afrihost, etc.

The attack is causing intermittent and slow connectivity to Cloud and VPS servers.

Extra Information:
--------------------

These attacks are relentless, very complex, sometimes last for days at a time and are very difficult to completely mitigate currently. 
The majority of the attack traffic is coming from unsecured Microsoft Azure instances which are being used to amplify the attack traffic to levels previous unseen in local South African traffic.

We have reached out to Microsoft to ask for assistance, but as yet they are not willing to help and simply put, just don't care that their infrastructure is being used as a weapon to attack local network providers.

In the past, attacks like this have been fairly easy to mitigate, as 90% of the traffic came over international links, which made it easy to block outside the country and to take advantage of international DDoS scrubbing services.
Since Microsoft, Amazon, China Telecom etc have started plugging into the local South African exchanges, the attack traffic is now coming over all the local exchanges and peering links.

Internationally we have Terabits of DDoS mitigation through multiple DDoS scrubbing clouds, however scrubbing clouds and services like this do not exist in South Africa as they were never really needed.

Among the local providers, the general consensus is that these attacks are targeted at specific providers and are being funded by someone or some company who stands to gain from these attacks.

We however, are not sitting around and doing nothing about these attacks, we have a number of solutions that we are implementing and going to be implementing to better mitigating these attacks, however the time of year makes implementation and ordering tricky.
We cannot disclose what we are implementing yet, but we can assure you that we are bringing in the big guns to fight these attacks, and to restore your confidence in us as your chosen service provider.

Regards,
Dave - CTO @ Domains.co.za
----------------------
Date - 20/12/2019 14:55 - 26/12/2019 09:12
Last Updated - 20/12/2019 14:56

DDoS attack (Resolved)

Affecting System - Network Infrastructure | Priority - Critical


We are currently being targeted in a DDoS attack again. Our upstream providers and engineers are working on mitigating the attack.

These are the same DDoS attacks which have made the news recently and affected other providers like RSAWEB, Cool Ideas, Bitco, Liquid Telecom, Web Africa, Afrihost, etc.

The attack is causing intermittent and slow connectivity to Cloud and VPS servers.

Extra Information:
--------------------

These attacks are relentless, very complex, sometimes last for days at a time and are very difficult to completely mitigate currently. 
The majority of the attack traffic is coming from unsecured Microsoft Azure instances which are being used to amplify the attack traffic to levels previous unseen in local South African traffic.

We have reached out to Microsoft to ask for assistance, but as yet they are not willing to help and simply put, just don't care that their infrastructure is being used as a weapon to attack local network providers.

In the past, attacks like this have been fairly easy to mitigate, as 90% of the traffic came over international links, which made it easy to block outside the country and to take advantage of international DDoS scrubbing services.
Since Microsoft, Amazon, China Telecom etc have started plugging into the local South African exchanges, the attack traffic is now coming over all the local exchanges and peering links.

Internationally we have Terabits of DDoS mitigation through multiple DDoS scrubbing clouds, however scrubbing clouds and services like this do not exist in South Africa as they were never really needed.

Among the local providers, the general consensus is that these attacks are targeted at specific providers and are being funded by someone or some company who stands to gain from these attacks.

We however, are not sitting around and doing nothing about these attacks, we have a number of solutions that we are implementing and going to be implementing to better mitigating these attacks, however the time of year makes implementation and ordering tricky.
We cannot disclose what we are implementing yet, but we can assure you that we are bringing in the big guns to fight these attacks, and to restore your confidence in us as your chosen service provider.

Regards,
Dave - CTO @ Domains.co.za
----------------------

Date - 06/12/2019 15:56 - 09/12/2019 09:48
Last Updated - 07/12/2019 06:13

ZACR - co.za problems (Resolved)

Affecting System - Domains API | Priority - Critical



ZACR are currently having problems on their CO.ZA EPP system, engineers at ZACR are aware - no ETA provided yet.


Date - 28/11/2019 10:38 - 05/12/2019 15:41
Last Updated - 28/11/2019 10:39

DDoS Attack (Resolved)

Affecting System - Network | Priority - Critical


We are currently being targetted in a DDoS attack, engineers and upstream providers are busy working on mitigating the attack.

22/11/2019 12:42:
---------------------

The DDoS attack has resumed, engineers are monitoring it.
This will cause some intermittent packet loss, the attack is currently being mitigated.




Date - 21/11/2019 03:23 - 22/11/2019 21:49
Last Updated - 22/11/2019 12:54

ZACR (.co.za) are down (Resolved)

Affecting System - .co.za domains | Priority - Critical


ZACR is experiencing Database problems on their primary systems, not ETA as yet

Notice : https://status.dns.business/incidents/5679
Date - 16/11/2019 18:46 - 21/11/2019 06:34
Last Updated - 16/11/2019 18:48

Incorrect Pages being serviced (Resolved)

Affecting System - Domains.co.za website | Priority - High


During some testing of the Litespeed webserver on our domains.co.za website at 1:00am Saturday morning 9th November, Litespeed started caching our sites pages to increase the site speed.
A side effect of this is that, Litespeed started caching logged in page which were then served to customers incorrectly when logging into their account.

This resulted in a person logging into their account and being present with the logged in dashboard of another user. 
We had 3 very awesome customers who notified us of this and we have disabled the Litespeed web server and reverted back to Apache to resolve the issue.

No login credentials were comprosed and no actions could be performed on these cached pages, they were served as static html compiled pages.
Further investigation has shown that only these 3 individual customers were affected by this unfortunate caching issue.

We would like to thank these customers for notifying us of this and apologize for any inconvenience or worry caused by this.

Dave Strydom
CTO @ Domains.co.za
Date - 09/11/2019 08:02 - 16/11/2019 18:46
Last Updated - 09/11/2019 08:47

MySQL not running (Resolved)

Affecting System - rs3, rs4, rs12, rs19 | Priority - Critical


Due to a automatically cPanel upgrade which occured this morning, it appears to have broken the mysql files on these servers which is causing start failures.

We are currently investigation a solution. not ETA yet.



*UPDATE*

*OFFICIAL NOTICE FROM CPANEL* 

We are currently receiving an influx of tickets regarding MariaDB failing to start with InnoDB Failed Assertion errors. We’ve identified this as an upstream issue with MariaDB (Case ID MDEV-20987) affecting the following versions: 10.1.42 10.2.28 10.3.19 10.4.9 The only known workaround is to downgrade the MariaDB package. Please reference https://go.cpanel.net/mariadb_1152019 for more information. 
 
10.1.42,  10.2.28,  10.3.19,  10.4.9

*UPDATE*

Fix has been applied to hosting/reseller servers and services should be restored





Date - 06/11/2019 05:21 - 09/11/2019 08:18
Last Updated - 06/11/2019 10:59

Hardware Failure (Resolved)

Affecting Server - cp19.domains.co.za | Priority - Critical


We have experienced a hardware failure on our cp19.domains.co.za server.

Engineers are currently attending to the issue, no ETA as yet, we will continue to update this notice when we have more information.

UPDATE 16:00:
There have been multiple drive failures in this server, the raid is not recoverable.
We will now start the process of restoring the cpanel account backups to other servers.

UPDATE 17:39
Account restores are at 8% complete and started at 17:15

UPDATE 18:30
Account restores are at 16%

UPDATE 19:30
Account restores are at 33%

UPDATE 21:20
Account restores are at 60%

UPDATE 22:30
Account restores are at 85%

UPDATE 23:15
All accounts have been successfully restored from backup.
We are now updating DNS records from 169.239.218.29 to 169.239.218.31 - If your domain is hosted at another provider, please update your DNS records at the provider.



Date - 31/10/2019 14:58 - 06/11/2019 05:20
Last Updated - 31/10/2019 23:17

Massive DDOS Attack (Resolved)

Affecting System - Domains Network | Priority - Critical


We are currently being targetted in a massive DDoS attack that is targeted at South African ISP's and Banks.
This is affecting the performance of our network on some of our uplinks which is causing intermittent connectivity issues.

There have been a number of articles in the press regarding this here:

https://mybroadband.co.za/news/internet/324480-south-african-isps-are-under-attack.html

https://mybroadband.co.za/news/banking/324881-south-african-banks-hit-by-massive-ddos-attack.html


We apologize for this, this is a coordinated, orchestrated and sophisticated attack on our network and we are working with upstream providers to try and mitigate this as best we can.

Date - 24/10/2019 20:29
Last Updated - 28/10/2019 09:30

ZACR EPP system issues (Resolved)

Affecting System - EPP | Priority - Critical


ZACR is experiencing technical issues with the EPP system. This can influence any domain related actions (registrations, renewals, lookups, etc)

They are aware of the issue and investigating the cause.

We will update as soon as feedback has been provided.
Date - 04/09/2019 15:34 - 16/09/2019 08:26
Last Updated - 04/09/2019 15:39

VPS node Issues (Resolved)

Affecting Server - manage.vps.net.za | Priority - Critical


We are experiencing technical issues on one of our VPS hypervisors.

We are busy investigating the issue.

Services will be restored shortly.

Apologies for the inconvenience.


***UPDATE***
VPS's have been booted and services have been restored.
Date - 15/08/2019 07:50 - 04/09/2019 15:31
Last Updated - 15/08/2019 08:01

Disk Failure (Resolved)

Affecting Server - rs2.cphost.co.za | Priority - Critical


There has been a SSD disk failure on the rs2.cphost.co.za, we are current busy with a replacement server.

All services are up and running on the server and we will continue to monitor it until the failed drive has been replaced and the raid has rebuilt.


Date - 11/02/2019 10:00 - 11/02/2019 15:35
Last Updated - 11/02/2019 10:25

Domains API Migration (Resolved)

Affecting System - https://api-v3.domains.co.za | Priority - Critical


During this maintenance window we will be migrating our domains API from the Vox Rutherford Estate Data Center to the Teraco Isando Data Center.

During this migration, API calls to our platform will fail until the migration is complete.



20:30 - We have started with migrating the data across to Teraco.

20:59 - Data has been transferred, it is now being verified.

21:25 - The API should be up along with cp.resellerpanel.co.za
Date - 19/11/2018 20:00 - 19/11/2018 21:53
Last Updated - 19/11/2018 21:29

19th October 2018 - Scheduled Maintenance (Resolved)

Affecting System - Teraco Network infrastructure | Priority - Critical


Good Day,

This notification is to inform all our domains.co.za clients that we are performing scheduled maintenance on our Teraco, Isando infrastructure on the 19th October 2018 which will include intermittent downtime.

Date : Friday - 19 October 2018
Time : 21:00pm - 05:00am

Work being performed:
- Upgrade of OS on border routers and switches.
- Removal of the existing 10gbit core switching stack.
- Installation of a new 80gbit core switching stack to increase redundancy and capacity.
- Reconfiguration of core network devices and switches.
- Installation of additional firewalls for Shared and Reseller Hosting to better balance load.

Impact:
Intermittent downtime will be experienced during the maintenance window on Shared Hosting services, Reseller Hosting Services, Cloud services and our Domains.co.za website.
No downtime will be experienced for Domain Resellers and Hosting Resellers (cp03.cphost.co.za to cp06.cphost.co.za) or Dedicated Server customers who are still in the Vox Rutherford Estate Data Center.

We do not expect the downtime to exceed 45 minutes while the core switching stack is removed and replaced. 
However there will be additional brief intermittent interruptions in traffic as devices are reconfigured or upgraded.


Regards,
The Domains.co.za Team

Date - 19/10/2018 21:00 - 30/10/2018 04:29
Last Updated - 15/10/2018 07:32

Network Connectivity - Rutherford Estate (Resolved)

Affecting Other - Connectivity | Priority - Critical


Vox Telecom is currently experiecing network connectivity issues at their Rutherford Estate infrastructure, we are awaiting an ETA from their side.

Connectivity has been restored, we will continue monitoring it from our side.
Date - 18/07/2018 11:57 - 23/07/2018 10:22
Last Updated - 18/07/2018 12:07

Domains API issues (Resolved)

Affecting System - API | Priority - Critical


We are aware of an issue on our domains API, engineers are currently working on the issue, no ETA as of yet.
Date - 11/07/2018 16:20
Last Updated - 12/07/2018 09:52

Disk Issues - RS3, RS4, RS5 (Resolved)

Affecting System - rs3.cphost.co.za, rs4.cphost.co.za, rs5.cphost.co.za | Priority - High


In light of the disk issues which have occured on rs4.cphost.co.za yesterday we have started investigating all SSD drives which were ordered from the same batch.

We have identified problems on rs3.cphost.co.za, rs4.cphost.co.za and rs5.cphost.co.za.
We are working with the manufacture and the local suppliers regarding these.
So far rs3 does not have any smart errors on the drives, but we are currently monitoring it.
rs5 is displaying smart errors, but is not displaying any symptoms we experienced on rs4 yet.

As a pro-active effort, we are busy provisioning rs11 and rs12 with brand new hardware and SSD drives from another manufacture.
Once ready, all accounts from rs4 will be migrated to rs11 and all accounts on rs5 will be migrated to rs12.

We do understand that this will be an inconvience to resellers on these affected servers and we apologize for that.
At this point our concern is your businesses and your customers.
The migration to brand new servers is the best route to ensure continuous stability for your customers going forward.

We can also confirm that all backups for the affected servers have / are running successfully and are stored on an entirely different infrastructure to these servers.
In the event of an issue occuring before the migration, all your customers data is backed up by ourselves.

We will keep this notice updated as the events unfold.

Date - 16/05/2018 12:25 - 11/07/2018 16:54
Last Updated - 16/05/2018 12:43

rs4.cphost.co.za (Resolved)

Affecting Server - rs4.cphost.co.za | Priority - Critical


UPDATE 15:10pm
--------------------

The server is currently operational, we are continuing to investigate the issue.

We will be switching out the hard drives systematically and allowing raid to rebuilt on the new drives until all existing drives are replaced.


--------------------


UPDATE 14:15
----------------

We have detected issues relating to storage on this server, we are currently doing disk checks.
Brand new drives are on route.

We are already putting in place a plan B of migrating accounts to other servers should we not be able to resolve this issue.

----------------


Good Day,

We would like to provide you with an update on rs4.cphost.co.za

We are currently busy with updates on the server, such as a kernel updates as well as inspecting the hardware for any issues.

Our expected ETA is 30 - 45 minutes, apologies for any inconvenience caused. 

 

Date - 15/05/2018 13:53 - 16/05/2018 12:25
Last Updated - 15/05/2018 15:04

Intermittent Connectivity (Resolved)

Affecting System - Core Network | Priority - Critical


We are currently experiencing intermittent connectivity issues on our Teraco Infrastructure.

Tech's are already investigating.


Date - 06/05/2018 18:35 - 06/05/2018 19:30
Last Updated - 06/05/2018 18:35

Server maintenance (Resolved)

Affecting Server - rs1.cphost.co.za | Priority - Critical


rs1.cphost.co.za will be rebooted 23:10 mail and website services will be affected.
Date - 07/04/2018 23:01 - 07/04/2018 23:34
Last Updated - 07/04/2018 23:05

rs1.cphost.co.za Server Maintenance (Resolved)

Affecting Server - rs1.cphost.co.za | Priority - Critical


Scheduled Maintenance will occur on the reseller hosting server RS1.CPHOST.CO.ZA (169.239.217.11) from 23:00 PM 30/03/2018 to 05:00 AM 31/03/2018. Websites and mail services will be down for the maintenance period window.

Date - 30/03/2018 19:18 - 31/03/2018 05:00
Last Updated - 31/03/2018 01:09

Website issues (Resolved)

Affecting Other - www.domains.co.za | Priority - High


Good Day,

We have pushed a new version of our site live and have detected a few minor bugs.
Fixes for these bugs will be deployed throughout the day.


Date - 07/03/2018 08:15 - 06/05/2018 19:30
Last Updated - 07/03/2018 08:17

Server Maintenance (Resolved)

Affecting Server - cp8.domains.co.za | Priority - High


Website access to this server will be interrupted during this maintenance window as we are upgrading Apache & PHP versions.


Date - 24/09/2017 07:00 - 24/09/2017 08:00
Last Updated - 11/01/2018 10:45

Site Builder Upgrade (Resolved)

Affecting Other - Site Builder Upgrade | Priority - Medium


Site Builder Upgrade

Domains.co.za is in the process of upgrading our website builder to the latest version with new features and functionality, your websites will remain active on the internet, however, unfortuately you will not be able to "Test Drive" the system or edit your website during this upgrade

Thank you for understanding


Date - 15/02/2017 09:04 - 15/02/2017 09:09
Last Updated - 15/02/2017 13:54

BGP Router Upgrade (Resolved)

Affecting Other - Core Network | Priority - Medium


At 21:00 12 January 2017 we will be replacing our primary BGP border router with an upgraded model.

No downtime is expected as traffic will be pushed across to our secondary BGP router.

Regards,
Domains.co.za
Date - 12/01/2017 21:00 - 30/01/2017 17:22
Last Updated - 12/01/2017 16:55

Cloud Software Upgrade (Resolved)

Affecting Server - manage.cloud.net.za | Priority - Medium




We will be performing a scheduled upgrade of our cloud infrastructure from 15 December 2016 from 19:00 until 7:00am 19 December 2016

This upgrade is update our cloud software to the latest LTS version.

No extended downtime is expected, there may be brief moments where your VM is inaccessible for a few minutes while it is hot-migrated between hypervisor nodes.
If there is a problem Hot-Migrating the VM between hypervisor nodes, your VM will be rebooted.



Date - 15/12/2016 19:00 - 26/12/2016 05:39
Last Updated - 01/12/2016 15:26

ZACR EPP system down (Resolved)

Affecting System - | Priority - Critical


ZACR is experiecing problems with connectivity to their EPP servers. Their engineers are currently investigating.


Date - 20/11/2016 07:00 - 08/12/2016 05:34
Last Updated - 20/11/2016 09:41

Emergency Power Maintenance DC3 Rutherford Es (Resolved)

Affecting Other - Emergency Power Maintenance DC3 Rutherford Estate | Priority - Critical


Emergency Power Maintenance DC3 Rutherford Estate 

VOX has just advised us that there is a faulty power circuit breaker in their Rutherford Estate DC3 facility. The circuit has to be replaced urgently and maintenance has been scheduled tomorrow evening (Saturday) 29 October between the hours of 23h00 – 01h00.

This will effect some of our Reseller hosting services and VPS servers.

NOTE:

Reseller and Cloud services in Teraco will not be effected

We apologize for any inconvenience caused

 

Date - 29/10/2016 23:00 - 08/12/2016 05:35
Last Updated - 28/10/2016 17:34

PHP websites not loading (Resolved)

Affecting Server - rs1.cphost.co.za | Priority - Critical


Hi,

We are aware of an issue on rs1.cphost.co.za regarding PHP websites not loading.

This appears to be caused by an automated cpanel update which has broken.
cPanel + Cloudlinux are currently working on the problem.

This is only affecting websites running "Native PHP" under the PHP Selector.
If you switch your PHP version to 5.6 and click "Save" your site will display online.


Date - 26/10/2016 03:00 - 20/11/2016 09:41
Last Updated - 26/10/2016 08:07

Cloud Migration - 5th August 2016 (Resolved)

Affecting System - Cloud Infrastructure | Priority - Medium




22:26 - All VM's are shutdown, The management engine is now being migrated to Teraco.


21:30 - VM's are currently being shutdown

_____________________________________________________

On the 5th August 2016 we will be migrating our cloud hosting infrastructure to our new infrastructure in Teraco, Isando.

We are scheduling a maintenance window between 21:30 pm Friday 5th August 2016 and 06:00 am Saturday 6th August 2016.
During this time all VM's will be shutdown and the infrastrucute will be migrated to Teraco Isando.

Downtime will be experienced during the physical move, however we do not expect downtime for the entire duration of this maintenance window.
We do not expect any problems during this migration, but as a pre-caution, we have arranged dedicated staff from all our vendors to be on standby during this maintenance window.

Notes:
 - No IP Addresses will be changing.

Should you have any questions, please contact support@domains.co.za 

Regards,
Domains.co.za

Date - 05/08/2016 21:30 - 06/08/2016 06:43
Last Updated - 05/08/2016 22:22

ADSL Maintenance (Resolved)

Affecting System - ADSL Infrastructure | Priority - Low


Please take note that there will be planned maintenance performed in the Johannesburg region on July 08th.

The maintenance window is scheduled from 00:01AM and will be concluded at 09:00AM.

Customers using ADSL services may experience increased latency and packet loss while maintenance is carried out.


Date - 08/07/2016 00:00 - 05/08/2016 11:28
Last Updated - 05/07/2016 10:48

ZACR Connection Issues (Resolved)

Affecting System - EPP Reseller / Register Platform | Priority - Critical


We have picked up connection issues with the ZACR epp platform.

ZACR is investing the network issue.

We will keep you updated.


Date - 27/05/2016 09:48
Last Updated - 27/05/2016 10:11

Emergency Maintenance Cloud VM's (Resolved)

Affecting System - Cloud VM's | Priority - Critical


pdate 31/01/2015 17:10am
-----------------------

All VM's should now be up, we will continue monitoring from our side.

Once again we apologize for this inconvenience.
We greatly appreciate your patience and support regarding this matter.

-----------------------

Update 31/01/2015 15:51am
-----------------------

The consistency check has completed.

We are now bringing up the remaining VM's. 


Once again we apologize for this inconvenience.
We greatly appreciate your patience and support regarding this matter.

-----------------------


Update 31/01/2015 14:51am
-----------------------

The consistency check is 75% complete.

We are still bringing up VM's 
Please do NOT attempt to start your VM yourself, please wait until we have brought your VM up.

Once again we apologize for this inconvenience.
We greatly appreciate your patience and support regarding this matter.

-----------------------

Update 31/01/2015 14:02am
-----------------------

The consistency is 50% complete.

We are slowly bringing up VM's.
Please do NOT attempt to start your VM yourself, please wait until we have brought your VM up.

Once again we apologize for this inconvenience.
We greatly appreciate your patience and support regarding this matter.

-----------------------

Update 31/01/2015 13:05am
-----------------------

The consistency check has picked up 143 vdisks which need to be resynced. It is currently on 30/143 vdisks complete.

We are going to slowly start bringing up VM's which don't fall into these 143, please do NOT attempt to start your VM yourself, please wait until we have brought your VM up.

Once again we apologize for this inconvenience.
We greatly appreciate your patience and support regarding this matter.

-----------------------

Update 31/01/2015 13:05am
-----------------------

Diagnostics are looking good, we now running consistency checks on VM data, please do NOT attempt to start your VM at this time as this may cause data loss.

-----------------------

Update 31/01/2015 12:53am
-----------------------

All Cloud storage nodes have been rebooted, we are now running some diagnostics, please do NOT attempt to start your VM at this time as this may cause data loss.

-----------------------

Update 31/01/2015 12:28am
-----------------------

The Cloud storage is currently being rebooted.

-----------------------

Update 31/01/2015 11:50am
-----------------------

All VM's have been stopped, we are now proceeding with shutting down the storage controllers and preparing for rebooting.

-----------------------

Update: 31/01/2015 10.50am
-----------------------

The VM's are currently being shutdown 3 at a time, 40% through this process.

-----------------------


Start: 31/01/2015 10.20am
-----------------------
Good Day,

Earlier this week we picked up some anomolies on our Cloud storage. We were advised by OnApp (The Vendor) that an upgraded was needed to be performed in order to resolve the issues.

This upgrade has failed and our Cloud storage needs to be completely rebooted.

 - We are going to proceed with shutting down all cloud VM's to protect their data.
 - Once all VM's have been stopped, the cloud storage will be rebooted and consistency checks will be done.
 - Once the consistency checks have completed we will proceed with booting up the VM's again.

We sincerly apologize for this and are currently looking at alternative ways of avoiding situations like this in the future.
-----------------------



Date - 31/01/2016 10:17 - 31/01/2016 18:30
Last Updated - 02/02/2016 20:54

International Access (Resolved)

Affecting Other - Data Center and ADSL | Priority - Critical


There are widespread outages on International access and some local ADSL access.
Initial reports are showing outages at NAP Africa, Mweb, Vodacom, CrystalWeb, Vox Teleom.

More information to follow.

We have received the following From Vox Telecom:
Vox Telecom services are currently impacted as a result of a failure with the upstream network providers, SEACOM and WACS. Upstream provider engineers are currently attending.

update: 
Break in SEACOM identified in Egypt. NO ETTR at present

WACS - 
The patch tails at Global Switch have been tested & levels are within limit , however there is still a LOS at the first ILA from London on the main route (Gerards Cross).
The engineer is still busy on site. No ETTR as yet.

To note:
Traffic cannot be rerouted for Vox Telecom - there is no alternative
Date - 21/01/2016 16:25
Last Updated - 22/01/2016 09:57

Scheduled Maintenance - Cloud Platform (Resolved)

Affecting Server - manage.cloud.net.za | Priority - Low


Tonight (02 April 2015) from 21:00 there will be scheduled maintenance to our cloud platform.
Our cloud software will be upgraded to the latest version. This however does involve rolling restarts across all Hypervisors.
During these rolling restarts your Virtual Machine(s) will be auto-migrated to other Hypervisors to ensure your Virtual Machine remains available.


Date: 02 April 2015
Time: 21:00 - 07:00 SAST (GMT+2)
Duration: 10 Hours
Purpose : Routine Software Updates.
Expected Impact : Minimum, there may be very brief intermittent loss of connectivity to Virtual Machines while they are being switched between hypervisors during the auto-migrations.

 

Date - 02/04/2015 21:00 - 10/05/2015 14:50
Last Updated - 02/04/2015 08:21

Vox Outage (Resolved)

Affecting Other - Vox Telecom | Priority - Critical


Vox is experiencing an Outage on their DC3 Facility, Engineers are attending.


Update: 
We have switched over to our redundant links and all traffic is now being routed through our DC4 Facility.
Date - 17/02/2015 10:45 - 17/02/2015 12:30
Last Updated - 18/02/2015 12:38

VOX BGP issues (Resolved)

Affecting Other - VOX BGP issues | Priority - Critical


VOX are having BGP issues on their network, we are addressing this with them now and will have the issue resolved soonest


Date - 17/02/2015 10:07 - 17/02/2015 11:47
Last Updated - 17/02/2015 11:47

Intermittent Fibre connectivity / routing bet (Resolved)

Affecting System - Fibre connectivity between DC3 & DC4 | Priority - Medium


Intermittent Fibre connectivity / routing between our DC3 & DC4 affecting Cloud, VPS servers and Reseller Hosting, infrastructure department working on it

UPDATE:

10h30
Infrastructure department found a break in the fibre cable between DC3 and DC4 hence loss of connectivity, a temporary solution has been implemented that has resolved the issue. Infrastructure will now look at a permanent resolution.

UPDATE:

14h00
New Fibre cable installed, routing reset back to normal

Issue Resolved

Date - 10/02/2015 09:19
Last Updated - 10/02/2015 16:45

Websites Not Displaying (Resolved)

Affecting Server - | Priority - Critical


Websites are not displaying on the cp02.cphost.co.za webserver and server load is high.
Engineers are busy investigating.

Date - 16/01/2015 10:43 - 16/01/2015 11:36
Last Updated - 16/01/2015 10:44

ZACR EPP System DB Issues (Resolved)

Affecting Other - ZACR EPP Database | Priority - Critical


ZACR is currently having load issues on their EPP Database which is causing "Failed to Connect" error messages when trying to perform any domain registrations, transfers or renewals.

Engineers are attending to the issue on their side, no ETA has been provided.

Date - 13/12/2014 14:02 - 16/01/2015 10:43
Last Updated - 13/12/2014 17:04

High Network Latency (Resolved)

Affecting Other - Network | Priority - High


Both Primary and Secondary Dark Fibre Africa links between Teraco and Rutherford Estate are down.
Traffic is being routed through a failover 1Gbit link, due to this there is a high latency on the network.

DFA is currently attending to the problem.

Date - 15/11/2014 07:43 - 15/11/2014 09:30
Last Updated - 19/11/2014 11:06

Phone Lines Down due to Power Failure (Resolved)

Affecting Other - Domains Offices | Priority - Critical


The phone lines at our offices are currently down until after due to an Eskom power failure in the Highlands North Johannesburg Area.


Date - 04/11/2014 08:39 - 04/11/2014 14:09
Last Updated - 15/11/2014 07:52

VCS Credit Card System (Resolved)

Affecting Other - VCS Credit Card System | Priority - Medium



Please note that VCS will be conducting a routine credit card system maintenance on Sunday 16 November 2014, starting 12am CAT and lasting a maximum of 2 hours.

During this time, access to our system will be temporarily unavailable.
Date - 16/11/2014 00:00 - 13/12/2014 17:04
Last Updated - 12/11/2014 07:56

Connectivity Issues to CP03 (Resolved)

Affecting Server - | Priority - High


We are currently investigating slow response times to cp03.domains.co.za
Date - 27/10/2014 10:32 - 27/10/2014 10:45
Last Updated - 27/10/2014 11:20

Limited Connectivity to Shared Hosting Server (Resolved)

Affecting Other - Shared Hosting | Priority - High


We are experiencing limited connectivity to all our shared hosting servers. Engineers are on site and the problem should be resolved shortly.

Update: At 12:05 the routing issue has been resolved.
Date - 21/08/2014 11:15 - 21/08/2014 12:05
Last Updated - 21/08/2014 12:19

ZACR Hardware Failure (Resolved)

Affecting Other - ZACR Registry | Priority - Critical


17/07/2014 21:10
ZACR has experienced a Hardware Failure on their Main EPP system which is affecting all .CO.ZA Domains.
No ETA yet.
Date - 17/07/2014 21:10
Last Updated - 27/07/2014 12:10

Hard Drive corruption (Resolved)

Affecting Server - | Priority - High


We experienced a hard drive failure and are busy performing consistency checks on the data.
No ETA yet.


UPDATE 14:31
We are investigating slow response times on this server
Date - 07/03/2014 13:38 - 07/03/2014 15:30
Last Updated - 17/03/2014 11:48

ZACR Outage (Resolved)

Affecting Other - ZACR Registry | Priority - Critical


The ZACR registry is experiecing epp database issues on their side and are currently working on resolving the matter.
No ETA has been provided.
Date - 04/03/2014 08:34 - 07/03/2014 13:39
Last Updated - 04/03/2014 08:35

Peering issue between Cape Town and Jhb (Resolved)

Affecting Other - Peering Issue | Priority - High


A peering issue has been reported between the Cape Town Internet Exchange (CINX) and our datacentre located in Johannesburg. This will affect all connectivity via major network providers (ADSL and Cellular) until the issue is resolved. We do not have an ETA for the resolution of the issue as yet but our upstream provider is working hard at rectifying the issue.


Date - 30/01/2014 07:30 - 30/01/2014 10:30
Last Updated - 30/01/2014 18:22

Down for emergency maintainence (Resolved)

Affecting Server - | Priority - Critical


We are busy with emergency maintenance on this server.
Date - 15/06/2013 07:03 - 15/06/2013 08:00
Last Updated - 15/06/2013 07:16

Rolling Reboots for Kernel Upgrades (Resolved)

Affecting Server - | Priority - Medium


There will be rolling reboots tonight for kernel upgrades to fix a new root exploit making rounds that affects CloudLinux 6/CentOS 6/RHEL 6 systems.
Date - 15/05/2013 19:45 - 15/05/2013 19:52
Last Updated - 15/05/2013 19:48

Rolling Reboots for Kernel Upgrades (Resolved)

Affecting Server - | Priority - Medium


There will be rolling reboots tonight for kernel upgrades to fix a new root exploit making rounds that affects CloudLinux 6/CentOS 6/RHEL 6 systems.
Date - 15/05/2013 19:45 - 15/05/2013 19:52
Last Updated - 15/05/2013 19:47

ZACR network outage (Resolved)

Affecting Other - Registrar Platform | Priority - Medium


Thr ZACR is experiencing a network outage which is affecting the performance of our registrar platform.

Technicians are investigating on their side.


10/03/2013 07:18 Update
-------------------------------------

ipv4 traffic has returned to normal, ipv6 is still down.
We are routing registrars on ipv4 for the time being until the ipv6 transit comes up.

10/03/2013 08:09 Update
-------------------------------------
Issue has been resolved
Date - 10/03/2013 06:02
Last Updated - 11/03/2013 12:27