Hi All,

Please be informed that users across sites were experiencing issue with QMS application.

Technical Team found the problem that was due to recent deployment and fixed it , QMS application is working fine now and available for business operations.

Sorry for the inconvenience caused.

Thanks for your understanding and co-operation.

Hi All,

Please be informed that users across sites were experiencing issue with QMS application.

Technical Team found the problem that was due to recent deployment and fixed it , QMS application is working fine now and available for business operations.

Sorry for the inconvenience caused.

Thanks for your understanding and co-operation.

Dear Senvias Team,

The migration to the Senvias PLM tenant will take place on Friday, August 2, 2024, beginning at 9:00 AM EDT and is expected to last approximately 7-8 hours. As a result, the TPI PLM tenant will be unavailable throughout the entire day on Friday.

Starting Monday, August 5, 2024, you will seamlessly resume your work from where you left off using the new Senvias PLM tenant.

Please plan your tasks accordingly. The current TPI PLM tenant will remain accessible only until the end of Thursday, August 1.

TPI PLM Wind users will not be affected by this migration and can continue their daily tasks as usual.

The cloning process has been successfully completed, and Wind data has been removed from the Senvias PLM tenant.

From today morning 10:30 AM IST hours onwards, laptops across all the regions are getting crashed and restarting for several users. For many of them, it's successfully getting restarted and able to continue operations and for some of them, it's failing to restart.

We're currently on the bridge call involving all the IT Teams across regions and cases have been raised with Microsoft Vendor Support Team to identify the root-cause and permanent fix.

As of now, Security Team confirmed that this is not any kind of malicious activity or a Cyber Attack.

We'll provide updates periodically. Thanks for your patience. 

We've identified the root-cause of the problem.

Recent version update of the CrowdStrike sensors running on the Windows laptops caused this bluescreen error and loop restarts of the laptops. We're working on pushing a global updates on all the machines to uninstall the sensor. 

Thanks for your patience. 

We're still working on a fix for this issue. In the meanwhile, we strongly advise users to not turn on their laptops unless its production critical and really necessary for delivery standpoint.

This is a global issue and several organizations' across the globe are impacted. We'll send out communications when the issue is fixed so that you can use your laptops without any issues.

Incase, you're using your laptops, please contact your local IT Team to uninstall the CrowdStrike software in your machine immediately. Once uninstalled, kindly refrain from accessing any unwanted sites and avoid downloading any content from the internet. 

Thanks for your patience. We will continue to send out periodic updates.

Security Team is still following up with CrowdStrike Vendor for permanent fix. 

If you're using your laptops, please contact your local IT Team to uninstall the CrowdStrike software in your machine immediately. Once uninstalled, kindly refrain from accessing any unwanted sites and avoid downloading any content from the internet. 

Thanks for your patience. We will continue to send out periodic updates.

IT team has been recovering laptops and PCs of affected users across the regions. As of 9:00 AM AZ time, many business critical user computers and leadership PCs are restored and the effort is ongoing in Mexico region. 

If you're using laptop, please contact your local IT Team to restore your machine. Once recovered, kindly refrain from accessing any unwanted sites and avoid downloading any content from the internet. 

This is a global issue impacting multiple organizations. A permanent fix is still being worked by Microsoft and Crowd strike.

Thanks for your patience and co-operation. Periodic updates will follow.

IT Team has restored majority of the impacted laptops and desktop computers in all regions. This enabled to continue regular operations and resume business activities.

Team has also reached out many users proactively and continued the restoring process. If you are one of the users (on-site or remote) who is still impacted, please reach out to your local IT support personal as mentioned in below list. Remote users will be supported over the phone and doesn't require to ship the laptop. 

As of 2:00 PM AZ Time, all critical applications, including Oracle ERP, tpiOne, QMS are working normal. 

Next update will follow on 07/20 9:00 AM AZ time. 

IT Teams from each region restored laptops that are identified and reported. If you are one of the users (on-site or remote) who is still impacted, please reach out to your local IT support by calling their contact Phone number or messaging in teams, as mentioned in below list. 

Regional IT teams are available to support over the weekend. Remote users will be supported over the phone and doesn't require to ship the laptop. 

Update from Technical Team -  To improve on the performance related issues  - our support team will have to apply some updates to the QMS Database, for which we will be planning an OUTAGE of 2 hours today.  We are awaiting necessary approvals from TPI leadership since this is Global incident. Please co-operate and be patient until we resolve this issue !!! ***  We will keep the entire team informed about the timings of OUTAGE

Completed Schedule outage and applied relevant fixes and the issue has been resolved.

We are experiencing  connectivity issue with Fortinet Client VPN. IT users at various TPI locations are impacted.

The Lumen team has confirmed that there is connectivity issue to the Lumen firewall, they are currently collaborating with their internal team to restore the firewall connection, but no estimated time of resolution has been provided. We are closely following up with the vendor, we will provide update when issue is fixed.

Note: Global protect VPN is working fine, kindly reach to local IT to enable global protect VPN for the time being.

We have escalated the VPN connectivity issue to Lumen's top level management and the service ticket is still being worked on.

We will keep everyone informed as soon as we receive further updates.

Sorry for the inconvenience caused.

Hi IT All,

Please be informed that Fortinet VPN connectivity issue has been resolved and it is working.

Thank you for your understanding and patience.

There was an internet outage on 09.07, at Menemen location. The MetroEthernet and R/L switches were down. 

Now, back to normal, up and running. 

Contacted ISP provider. 

They came to factory, checked the fiber circuits and switches, performed tests and changed the sfp tranceivers, came back to normal.

For thr R/L devices, the power supply was died and once we changed it, the R/L device and circuit came back to normal as well.

Working on the backing up the devices of MetroEthernet and R/L.

Sfp tranceivers stopped working. 

Power supply of the R/L device stopped working. 

Users across the TPI locations are unable to connect to the Oracle applications over Fortinet Client, Global Protect VPN and from Office premise network , due to network outage with Megaport cloud service provider, we are closely working with the vendor.

And also the integration to oracle will not work until network connectivity issue is fixed.

Case has been raised with Megaport service provider to understand the issue. We will keep you all posted for further updates. Sorry for the inconvenience and thank you for your understanding.
 

Network connectivity issue has been resolved. 

Oracle users should be able to connect oracle now.

Thanks for your understanding.

Users across the TPI location are unable to connect to the Fortinet Client VPN. An E-Case has been raised with Lumen to understand the issue. We will keep you all posted for further updates. Sorry for the inconvenience and thank you for your understanding.

We have successfully resolved the issue, and you should now be able to connect without any problems. Our logs also confirm the successful connections.

An issue has been identified with EPBCS causing incorrect information of our balance sheet. Application support team found the issue related to a recent April patch that was deployed and actively working with Oracle roll back the changes. In parallel, Oracle is debugging the change to find the root cause and fix. If Oracle finds the bug in production and the required fix, we may avoid having to rebuild production. 

Technical Support teams are regrouping at 6:30 PM AZ Time for progress. 

Issue Resolved in Production after Oracle technical team fixed the bug.  Root Cause Analysis will be performed by working with Oracle team and implement plans to prevent future occurrence. 

Fiber Optic cable has been broken by excavator during the excavation work managing by investment department. 

Radiolink circuit's configuration has already been changed by ISP TTNet, thus the backup line did not work. 

The cut-over duration was 2.30 hours. 

Fiber optic cable has been repaired by 3rd party company. 

ISP repaired the R/L circuit, then everything started to work. 

Excavator operator was not informed and aware of Fiber optic cable place. That should be marked and informed by investment department. 

Actions taken: 

Monthly R/L test will be handled. Root cause analysis is started with ISP, trying to get the reason of the configuration change.

The map having highlighted fiber optic cable roads, will be shared with IT by investment department and the roads will be painted with red or yellow to mark if there is fiber optic cable under the road.