From today morning 10:30 AM IST hours onwards, laptops across all the regions are getting crashed and restarting for several users. For many of them, it's successfully getting restarted and able to continue operations and for some of them, it's failing to restart.

We're currently on the bridge call involving all the IT Teams across regions and cases have been raised with Microsoft Vendor Support Team to identify the root-cause and permanent fix.

As of now, Security Team confirmed that this is not any kind of malicious activity or a Cyber Attack.

We'll provide updates periodically. Thanks for your patience. 

We've identified the root-cause of the problem.

Recent version update of the CrowdStrike sensors running on the Windows laptops caused this bluescreen error and loop restarts of the laptops. We're working on pushing a global updates on all the machines to uninstall the sensor. 

Thanks for your patience. 

We're still working on a fix for this issue. In the meanwhile, we strongly advise users to not turn on their laptops unless its production critical and really necessary for delivery standpoint.

This is a global issue and several organizations' across the globe are impacted. We'll send out communications when the issue is fixed so that you can use your laptops without any issues.

Incase, you're using your laptops, please contact your local IT Team to uninstall the CrowdStrike software in your machine immediately. Once uninstalled, kindly refrain from accessing any unwanted sites and avoid downloading any content from the internet. 

Thanks for your patience. We will continue to send out periodic updates.

Security Team is still following up with CrowdStrike Vendor for permanent fix. 

If you're using your laptops, please contact your local IT Team to uninstall the CrowdStrike software in your machine immediately. Once uninstalled, kindly refrain from accessing any unwanted sites and avoid downloading any content from the internet. 

Thanks for your patience. We will continue to send out periodic updates.

IT team has been recovering laptops and PCs of affected users across the regions. As of 9:00 AM AZ time, many business critical user computers and leadership PCs are restored and the effort is ongoing in Mexico region. 

If you're using laptop, please contact your local IT Team to restore your machine. Once recovered, kindly refrain from accessing any unwanted sites and avoid downloading any content from the internet. 

This is a global issue impacting multiple organizations. A permanent fix is still being worked by Microsoft and Crowd strike.

Thanks for your patience and co-operation. Periodic updates will follow.

IT Team has restored majority of the impacted laptops and desktop computers in all regions. This enabled to continue regular operations and resume business activities.

Team has also reached out many users proactively and continued the restoring process. If you are one of the users (on-site or remote) who is still impacted, please reach out to your local IT support personal as mentioned in below list. Remote users will be supported over the phone and doesn't require to ship the laptop. 

As of 2:00 PM AZ Time, all critical applications, including Oracle ERP, tpiOne, QMS are working normal. 

Next update will follow on 07/20 9:00 AM AZ time. 

IT Teams from each region restored laptops that are identified and reported. If you are one of the users (on-site or remote) who is still impacted, please reach out to your local IT support by calling their contact Phone number or messaging in teams, as mentioned in below list. 

Regional IT teams are available to support over the weekend. Remote users will be supported over the phone and doesn't require to ship the laptop. 

Planned tpiOne Outage this Weekend

There is a planned tpiOne outage for all users starting this Friday/Saturday based on your country due to a system upgrade. During this planned outage you will not have any access to tpiOne. Please communicate to your HR team, managers and associates that they will not be able to access tpiOne during the below time windows (impact should be minimal as these are afterhours).  Additionally, all transactions submitted and in the queue must be completely approved before the upgrade begins. Any transactions remaining may need to be canceled and resubmitted afterward. If you need support with pushing transactions through before the outage, please reach out to Ahmed Mohammed or Mary Sesma for support.

China:

• Begins:  Saturday, June 18, 2022 at 10:00AM
• Ends: Saturday, June 18, 2022 at 10:00PM

Denmark/Germany/Spain:

• Begins: Saturday, June 18, 2022 at 4:00AM
• Ends: Saturday, June 18, 2022 at 4:00PM

India:

• Begins: Saturday, June 18, 2022 at 7:30AM
• Ends: Saturday, June 18, 2022 at 7:30PM

Mexico:

• Begins: Friday, June 18, 2022 at 9:00PM
• Ends: Saturday, June 18, 2022 at 9:00AM

Turkey:

• Begins: Saturday, June 18, 2022 at 5:00AM
• Ends: Saturday, June 18, 2022 at 5:00PM

United States:

• Begins: Friday, June 17, 2022 at 7:00PM MST/9:00PM CST/10:00PM EST
• Ends: Saturday, June 18, 2022 at 7:00AM MST/Saturday March 26, at 9:00AM CST/10:00AM EST

tpiOne maintenance is completed as of now.

Planned tpiOne Outage this Weekend

There is a planned tpiOne outage for all users starting this Friday/Saturday based on your country due to a system upgrade. During this planned outage you will not have any access to tpiOne. Please communicate to your HR team, managers and associates that they will not be able to access tpiOne during the below time windows (impact should be minimal as these are afterhours).  Additionally, all transactions submitted and in the queue must be completely approved before the upgrade begins. Any transactions remaining may need to be canceled and resubmitted afterward. If you need support with pushing transactions through before the outage, please reach out to Ahmed Mohammed or Mary Sesma for support.

China:

• Begins:  Saturday, March 26, 2022 at 9:00AM
• Ends: Saturday, March 26, 2022 at 1:00PM

Denmark/Germany/Spain:

• Begins: Saturday, March 26, 2022 at 2:00AM
• Ends: Saturday March 26, 2022 at 6:00AM

India:

• Begins: Saturday, March 26, 2022 at 6:30AM
• Ends: Saturday, March 26, 2022 at 10:30AM

Mexico:

• Begins: Friday, March 25, 2022 at 7:00PM
• Ends: Friday March 25, 2022 at 11:00PM

Turkey:

• Begins: Saturday, March 26, 2022 at 4:00AM
• Ends: Saturday March 26, 2022 at 8:00AM

United States:

• Begins: Friday, March 25, 2022 at 7:00PM PST/9:00PM CST/10:00PM EST
• Ends: Friday March 25, 2022 at 11:00PM PST/Saturday March 26, at 1:00AM CST/2:00AM EST

This incident is resolved.

Overview: tpiOne will be down to apply the Oracle 21D quarterly upgrade. 

Business Impact: All users will be unable to log into tpiOne during this window.

The 21D upgrade is completed and the instance is up and running.

Category: Planned outage

Date: 8/20, 9pm pst to 8/22 6pm pst

Overview: tpiOne will be down to apply the Oracle 21c quarterly upgrade.

Business Impact: All users will be unable to log into tpiOne during this window.

TpiOne is now online

Due to electrical instructure failure Servers and Network has shutdown involuntarily. 

Services were unavailable for 13hrs.

Eletrical Maintanence team fixed the issue in 13hrs.

Systems are back up and running.

Maintenance on Backup Power Generators caused electrical short curciut, and caused factory wide general electrical failure.

Hi All,

Please find the SEV1 Process:

Definition:

•Emergency issues are raised as incidents by local IT only when the production environment is down (partially or completely) impacting any function from performing their work.

Process:

•On creation of the ticket the IT production support team are immediately notified.
•Production support team opens a bridge on MS teams.
•Ticket originator, relevant business SME, regional IT leads, technical and functional IT support contacted as a priority to join – by phone if necessary.
•Priority 1/Urgent Vendor support tickets opened providing business impact if required.
•24x7 support provided.
•Solution implemented based on optimal production stability and cycle time.
•Approvals required will be requested verbally/electronic with supporting documentation.
•RCCA details will be documented, stored and communicated within 24 hours of ticket closure, corrective actions implemented within 1 week.

Communication:

•Statuscast communication initiated detailing issue, impact and ETC.
•2 hourly updates communicated until issue resolved.
•Ticket originator will be contacted directly and via status cast. 

Thanks,
AZ IT.
 

Hi All,

Issue: Currently we are seeing slowness in the performance of the QMS and service unavailability (503 error) as CPU utilization is max with 8 GB and 2 VCPU's.
Business Impact: All QMS users are being impacted.
Current Status: We will be up scaling the CPU and bouncing the AWS Database.
Outage Window: Tuesday 10/27/2020 11:30 PM MST to Wednesday 10/28/2020 12:00 AM MST.

Thanks,
AZ IT.

Note: Please ignore the Impact field in the notification. "Only QMS is being Impacted" and we have updated the Outage Window.

Hi All,

Issue: Currently we are seeing slowness in the performance of the QMS and service unavailability (503 error) as CPU utilization is max with 8 GB and 2 VCPU's.
Business Impact: All QMS users are being impacted.
Current Status: We will be up scaling the CPU and bouncing the AWS Database.
Outage Window: Tuesday 10/27/2020 11:30 PM MST to Wednesday 10/28/2020 12:00 AM MST.

Thanks,
AZ IT.

Note: Please ignore the Impact field in the notification. "Only QMS is being Impacted"

Hi All,

Issue: Currently we are seeing slowness in the performance of the QMS and service unavailability (503 error) as CPU utilization is max with 8 GB and 2 VCPU's.
Business Impact: All QMS users are being impacted.
Current Status: Resolved: QMS application is back up and running. Restarted the AWS MySQL database.

Thanks,
AZ IT.

Note: Please ignore the Impact field in the notification. "Only QMS is being Impacted" and we have updated the Outage Window.