From today morning 10:30 AM IST hours onwards, laptops across all the regions are getting crashed and restarting for several users. For many of them, it's successfully getting restarted and able to continue operations and for some of them, it's failing to restart.

We're currently on the bridge call involving all the IT Teams across regions and cases have been raised with Microsoft Vendor Support Team to identify the root-cause and permanent fix.

As of now, Security Team confirmed that this is not any kind of malicious activity or a Cyber Attack.

We'll provide updates periodically. Thanks for your patience. 

We've identified the root-cause of the problem.

Recent version update of the CrowdStrike sensors running on the Windows laptops caused this bluescreen error and loop restarts of the laptops. We're working on pushing a global updates on all the machines to uninstall the sensor. 

Thanks for your patience. 

We're still working on a fix for this issue. In the meanwhile, we strongly advise users to not turn on their laptops unless its production critical and really necessary for delivery standpoint.

This is a global issue and several organizations' across the globe are impacted. We'll send out communications when the issue is fixed so that you can use your laptops without any issues.

Incase, you're using your laptops, please contact your local IT Team to uninstall the CrowdStrike software in your machine immediately. Once uninstalled, kindly refrain from accessing any unwanted sites and avoid downloading any content from the internet. 

Thanks for your patience. We will continue to send out periodic updates.

Security Team is still following up with CrowdStrike Vendor for permanent fix. 

If you're using your laptops, please contact your local IT Team to uninstall the CrowdStrike software in your machine immediately. Once uninstalled, kindly refrain from accessing any unwanted sites and avoid downloading any content from the internet. 

Thanks for your patience. We will continue to send out periodic updates.

IT team has been recovering laptops and PCs of affected users across the regions. As of 9:00 AM AZ time, many business critical user computers and leadership PCs are restored and the effort is ongoing in Mexico region. 

If you're using laptop, please contact your local IT Team to restore your machine. Once recovered, kindly refrain from accessing any unwanted sites and avoid downloading any content from the internet. 

This is a global issue impacting multiple organizations. A permanent fix is still being worked by Microsoft and Crowd strike.

Thanks for your patience and co-operation. Periodic updates will follow.

IT Team has restored majority of the impacted laptops and desktop computers in all regions. This enabled to continue regular operations and resume business activities.

Team has also reached out many users proactively and continued the restoring process. If you are one of the users (on-site or remote) who is still impacted, please reach out to your local IT support personal as mentioned in below list. Remote users will be supported over the phone and doesn't require to ship the laptop. 

As of 2:00 PM AZ Time, all critical applications, including Oracle ERP, tpiOne, QMS are working normal. 

Next update will follow on 07/20 9:00 AM AZ time. 

IT Teams from each region restored laptops that are identified and reported. If you are one of the users (on-site or remote) who is still impacted, please reach out to your local IT support by calling their contact Phone number or messaging in teams, as mentioned in below list. 

Regional IT teams are available to support over the weekend. Remote users will be supported over the phone and doesn't require to ship the laptop. 

Purpose of Maintenance: As part of a global improvement project, Megaport is in the process of installing new devices across our network, in a continued effort to ensure the efficiency and stability of the Megaport network. As part of this project, your services at Switch SUPERNAP 7, 7135 S Decatur Blvd, Las Vegas, NV 89118, United States, will be migrated to a new device. There is no action required at your end and there will be no changes to your service id or any other information related to your services. We thank you for your patience throughout the device migration.

Impacted Service(s): 30 minutes

Start Date and Time: Fri, 08 Oct 2021 07:00:00 UTC
End Date and Time: Fri, 08 Oct 2021 14:00:00 UTC

Outage window complete

Hi All,

Issue: The IT department is performing a switch replacement in the AZ COLO which might impact COLO services like Oracle, Fortinet VPN and QMS between 6:00 PM and 6:30 PM AZ and once this activity is completed we will be applying 27732516 and 26115786 patches on Oracle Production servers from 6.30 PM to 10:00 PM AZ.

Business Impact: All Oracle EBS, QMS users will be impacted.

Current Status: As part of the fix recommended by Oracle we need to apply two patches during outage window.

Outage Window: Saturday 05/15/2020 06:00 PM AZ to Saturday 05/15/2020 10:00 PM AZ.

Thanks,
AZ IT.

Hi All,

Nexus switch in the COLO has been successfully replaced. All applications are up and running fine.

Oracle patches are not being applied today due to limited availability of our DBA.

Thanks,

AZ IT.

This will not cause downtime to AWS network connection

Please be advised that a Planned maintenance event (CSS31136) will impact the services listed below.

Purpose of Maintenance: To ensure uptime and stability of the Megaport network, we have scheduled a software update for a device at Switch SUPERNAP 7, 7135 S Decatur Blvd, Las Vegas, NV 89118, United States. The purpose of this software upgrade is to resolve critical software bugs that have impacted some services. Downtime is not expected to be any longer than 30 minutes, with works starting as close to the beginning of the maintenance window as possible. Megaport always indicates a longer maintenance window than required, so that in the rare event of any issues, Megaport engineers can resolve them within the prescribed maintenance window. Megaport maintenance schedule start and finish time window includes the implementation and possible rollback time. Megaport encourages our customers to ensure they have diversity available for all services. If you require assistance in arranging diversity to mitigate the impact of this planned downtime, please contact your Megaport Account Manager. We thank you for your patience during this maintenance event.

Impacted Service(s): 1x60 minute

Start Date and Time: Sun, 25 Apr 2021 07:00:00 UTC
End Date and Time: Sun, 25 Apr 2021 09:00:00 UTC

Upgrade completed successfully for Switch SUPERNAP 7 

The Internet circuit to AWS Transit is currently down. This will prevent connections to AWS from the Colo The Circuit vendor has been notified and is working to resolve the issue.

Service s now online

The switch has a fan issue and reboots itself. A replacement switch is being shipped by Cisco. Once the switch is received, we will need to arrange a service outage during which the failing switch will be replaced.

Due to electrical instructure failure Servers and Network has shutdown involuntarily. 

Services were unavailable for 13hrs.

Eletrical Maintanence team fixed the issue in 13hrs.

Systems are back up and running.

Maintenance on Backup Power Generators caused electrical short curciut, and caused factory wide general electrical failure.

Tk1 MPLS connection to USA went down at 02:10 AM MST (12:10PM GMT+3) for 19 minutes

It was back up at 02:29 AM MST (12:29PM GMT+3)

Ticket opened to Lumen. There was a maintenace and we did not know about it.

Response from Lumen:

Initial service diagnostic tests have identified that your service was affected by maintenance 20537997 from Mon, 2021/02/08 23:00:00 GMT to Tue, 2021/02/09 03:00:00 GMT

Maintenance on circuit.

TK1 Site will be upgrading Backbone switch. 6hr downtime is expected.

Local time in Turkey for the downtime is from 09:00 to 17:00 (GMT+3)

All IP related services, applications, and servers will be offine during these hours.

Also WAN IP will be changed for TK1 Site. New IP is: 88.255.188.20

Due to unforseen issues, Backbone repalement took longer than expected. Completed with 1hr 20min delay.

Completed

Hi All,

Please find the SEV1 Process:

Definition:

•Emergency issues are raised as incidents by local IT only when the production environment is down (partially or completely) impacting any function from performing their work.

Process:

•On creation of the ticket the IT production support team are immediately notified.
•Production support team opens a bridge on MS teams.
•Ticket originator, relevant business SME, regional IT leads, technical and functional IT support contacted as a priority to join – by phone if necessary.
•Priority 1/Urgent Vendor support tickets opened providing business impact if required.
•24x7 support provided.
•Solution implemented based on optimal production stability and cycle time.
•Approvals required will be requested verbally/electronic with supporting documentation.
•RCCA details will be documented, stored and communicated within 24 hours of ticket closure, corrective actions implemented within 1 week.

Communication:

•Statuscast communication initiated detailing issue, impact and ETC.
•2 hourly updates communicated until issue resolved.
•Ticket originator will be contacted directly and via status cast. 

Thanks,
AZ IT.
 

Purpose of Maintenance: Please be advised that Megaport will be performing a maintenance to upgrade the MCR on 615 N 48th St, Phoenix, AZ 85008

Impacted Service(s): Customers may experience downtime to their service(s) during the maintenance windows for up to 1 x 30 minutes.

Start Date and Time: Tue, 17 Nov 2020 08:00:00 UTC
End Date and Time: Tue, 17 Nov 2020 14:00:00 UTC

MCR upgrade completed. VXCs are back online.