From today morning 10:30 AM IST hours onwards, laptops across all the regions are getting crashed and restarting for several users. For many of them, it's successfully getting restarted and able to continue operations and for some of them, it's failing to restart.

We're currently on the bridge call involving all the IT Teams across regions and cases have been raised with Microsoft Vendor Support Team to identify the root-cause and permanent fix.

As of now, Security Team confirmed that this is not any kind of malicious activity or a Cyber Attack.

We'll provide updates periodically. Thanks for your patience. 

We've identified the root-cause of the problem.

Recent version update of the CrowdStrike sensors running on the Windows laptops caused this bluescreen error and loop restarts of the laptops. We're working on pushing a global updates on all the machines to uninstall the sensor. 

Thanks for your patience. 

We're still working on a fix for this issue. In the meanwhile, we strongly advise users to not turn on their laptops unless its production critical and really necessary for delivery standpoint.

This is a global issue and several organizations' across the globe are impacted. We'll send out communications when the issue is fixed so that you can use your laptops without any issues.

Incase, you're using your laptops, please contact your local IT Team to uninstall the CrowdStrike software in your machine immediately. Once uninstalled, kindly refrain from accessing any unwanted sites and avoid downloading any content from the internet. 

Thanks for your patience. We will continue to send out periodic updates.

Security Team is still following up with CrowdStrike Vendor for permanent fix. 

If you're using your laptops, please contact your local IT Team to uninstall the CrowdStrike software in your machine immediately. Once uninstalled, kindly refrain from accessing any unwanted sites and avoid downloading any content from the internet. 

Thanks for your patience. We will continue to send out periodic updates.

IT team has been recovering laptops and PCs of affected users across the regions. As of 9:00 AM AZ time, many business critical user computers and leadership PCs are restored and the effort is ongoing in Mexico region. 

If you're using laptop, please contact your local IT Team to restore your machine. Once recovered, kindly refrain from accessing any unwanted sites and avoid downloading any content from the internet. 

This is a global issue impacting multiple organizations. A permanent fix is still being worked by Microsoft and Crowd strike.

Thanks for your patience and co-operation. Periodic updates will follow.

IT Team has restored majority of the impacted laptops and desktop computers in all regions. This enabled to continue regular operations and resume business activities.

Team has also reached out many users proactively and continued the restoring process. If you are one of the users (on-site or remote) who is still impacted, please reach out to your local IT support personal as mentioned in below list. Remote users will be supported over the phone and doesn't require to ship the laptop. 

As of 2:00 PM AZ Time, all critical applications, including Oracle ERP, tpiOne, QMS are working normal. 

Next update will follow on 07/20 9:00 AM AZ time. 

IT Teams from each region restored laptops that are identified and reported. If you are one of the users (on-site or remote) who is still impacted, please reach out to your local IT support by calling their contact Phone number or messaging in teams, as mentioned in below list. 

Regional IT teams are available to support over the weekend. Remote users will be supported over the phone and doesn't require to ship the laptop. 

We are experiencing  connectivity issue with Fortinet Client VPN. IT users at various TPI locations are impacted.

The Lumen team has confirmed that there is connectivity issue to the Lumen firewall, they are currently collaborating with their internal team to restore the firewall connection, but no estimated time of resolution has been provided. We are closely following up with the vendor, we will provide update when issue is fixed.

Note: Global protect VPN is working fine, kindly reach to local IT to enable global protect VPN for the time being.

We have escalated the VPN connectivity issue to Lumen's top level management and the service ticket is still being worked on.

We will keep everyone informed as soon as we receive further updates.

Sorry for the inconvenience caused.

Hi IT All,

Please be informed that Fortinet VPN connectivity issue has been resolved and it is working.

Thank you for your understanding and patience.

Hi All,

Issue: The IT department is performing a switch replacement in the AZ COLO which might impact COLO services like Oracle, Fortinet VPN and QMS between 6:00 PM and 6:30 PM AZ and once this activity is completed we will be applying 27732516 and 26115786 patches on Oracle Production servers from 6.30 PM to 10:00 PM AZ.

Business Impact: All Oracle EBS, QMS users will be impacted.

Current Status: As part of the fix recommended by Oracle we need to apply two patches during outage window.

Outage Window: Saturday 05/15/2020 06:00 PM AZ to Saturday 05/15/2020 10:00 PM AZ.

Thanks,
AZ IT.

Hi All,

Nexus switch in the COLO has been successfully replaced. All applications are up and running fine.

Oracle patches are not being applied today due to limited availability of our DBA.

Thanks,

AZ IT.

Hi All,

Please find the SEV1 Process:

Definition:

•Emergency issues are raised as incidents by local IT only when the production environment is down (partially or completely) impacting any function from performing their work.

Process:

•On creation of the ticket the IT production support team are immediately notified.
•Production support team opens a bridge on MS teams.
•Ticket originator, relevant business SME, regional IT leads, technical and functional IT support contacted as a priority to join – by phone if necessary.
•Priority 1/Urgent Vendor support tickets opened providing business impact if required.
•24x7 support provided.
•Solution implemented based on optimal production stability and cycle time.
•Approvals required will be requested verbally/electronic with supporting documentation.
•RCCA details will be documented, stored and communicated within 24 hours of ticket closure, corrective actions implemented within 1 week.

Communication:

•Statuscast communication initiated detailing issue, impact and ETC.
•2 hourly updates communicated until issue resolved.
•Ticket originator will be contacted directly and via status cast. 

Thanks,
AZ IT.
 

We will be adding network equipment to the AZ COLO. This will cause an outage of all network systems at the AZ COLO. The following services will be unavailable during this outage:

1) Oracle

2) HFM

3) VPN

4) Citrix

5) SolidWorks licensing

6) AutoCAD licensing

All work is complete. All applications are available.