Hi All 

   Currently IND site users not able to launch Oracle forms while connecting outside VPN.

   Application team working with relevant infra and IT Support teams to check the issue.

  We'll keep you posted with updates.

Thanks & Regards

Oracle Support

Hi All

   MX users are able launch application without any issues in support or PROD instance.

   IND users are alone facing the issue while connecting outside VPN.

   Networks have done checks at their side and could see application connections are going through without any issues.

  Currently bringing Oracle vendor for this issue (Sev-1) being raised for the same.

Thanks & Regards,

Oracle Support

Hi All

   IND site users are able to launch Oracle Forms from factory network itself without any issues.

 Infra teams checked and confirmed no changes/update done at their side.

Application team checking with Oracle vendor for RCA as why this intermediate disconnect has happened and how to prevent in future.

Issue stands as closed and no more further updates would be rolled out.

Thanks for your understanding and support.

Thanks & Regards,

Oracle Support

Oracle vendor has reported threat for customers in E-business suite environment and extort funds. This specific event threat is only related to EBS environments deployed both on prem and cloud.

So, Oracle E-Business product team has issued a critical Severity Alert, advising customers to patch their system immediately and be diligent of abnormal and malicious activity in their E-Business ecosystem.

Currently DBA team has done patching in non-prod environment and testing is going on - also discussions are being done with security team to confirm on appropriate timing in production as it involves downtime.

We'll keep you posted further with updates.

Thanks & Regards.

Oracle IT Support

Testing has been completed and no issues has been observed in non-prod.

It has been decided to proceed with this critical security threat patching in PROD around 7 PM Arizona time (19 MST).

We'll update once patching has been started and subsequent progress have been made.

Hi All

Due to recent Windows OS update,  Java has been uninstalled and because of it Oracle forms not able to launch. 

IT Helpdesk has provided below steps to resolve the issue - please make a note of it and quickly fix from your end.

For users with Java issues, please restart their PC then right click to open the Endpoint central icon (the TPI icon) by the clock on their desktop and select "Self Service Portal". There is an installer for Java that should put the app and any missing files back. 

If issue still didn't get fixed, please work with your corresponding local IT team to get Java installed and have Oracle forms launched properly.

Thanks

Oracle IT Support

IT team identified 416 Move Order lines stuck in EBS from WMS and impacting job orders, CoC generation, and customer invoicing.  IT team will start reprocessing the stuck move orders to be consumed by Manufacturing traceability application.  Frequent updates will be published on the progress. 

SEV-1 Ticket: INC0099928 

Regards,
IT Team

IT Team checking for possibility of reprocessing of affected items - IT SME's will be assigned responsibilities of same as user to split and reprocess those items. DBA team is working to have the roles assigned to provided users.

SEV-1 Ticket Reference: INC0099928 

Regards,
IT Team

DBA team has provided required roles for SME's to reprocess affected items.

IT team is currently working with high priority to get it reprocessed at the earliest.

SEV-1 Ticket Reference: INC0099928 

Regards,
IT Team

IT team analyzed the issue and found manual PO Updates is causing Receipt transactions to fail, which creates On-hand unavailability.

Currently they have enabled the customization of restriction to prevent PO updates in EBS.

Following is the latest update of reprocessing:

Total: 307 Lines specific for the 3 pending sets

Pending: 24 Lines

Processed Lines: 283 Lines

Regards,

IT Support

For the issue mentioned about PO Updates in previous communication, which was causing around 65 receipts have failed due to the POs not being in an approved status, and because the existing shipment lines were closed and replaced with new ones, restriction has been re-applied in Oracle EBS at the PO level to prevent buyers from updating POs once receiving has started in WMS. This is a temporary action till it a permanent solution is identified and implemented.

Currently IT team working to clear remaining 24 pending lines to complete the processing.

SEV-1 Ticket: INC0099928 

Regards

IT Team

Pending items reprocessing status:

Total - 24

Processed - 16

Yet to be processed - 8

SEV-1 Ticket: INC0099928 

Latest update of reprocessing: All pending items have got reprocessed except below two move orders - it will be discussed further with users once they are available by IT.

18475645 - Line 2 was cancelled in WMS; Line 4 was over consumed in EBS.

18479245 - Line was cancelled in WMS.

Please validate remaining move orders and let us know if all processed successfully.

SEV-1 Ticket: INC0099928 

Regards,
IT Team

Users are still reviewing reprocessed move orders done from morning - will update us the status in sometime.

SEV-1 Ticket: INC0099928 

Regards,
IT Team

A decision has been made to roll back the WMS application and resume warehouse operations directly in Oracle ERP (EBS). The IT team is prioritizing the following actions to complete before 3:30 PM AZ time, with further updates and next steps to follow:

• Deactivate business user access in the WMS application 
• Disable the integration between EBS and WMS
• Disable WMS-related screen customizations in EBS
• Extract and share all putaways, stuck receipts, and move orders with the business team

Once these steps are completed, the business team will perform validation before proceeding with subsequent activities.

IT completed all WBS roll back tasks and handed over pending transactions extract to business teams for manually processing in Oracle EBS.

Below activities are planned to be completed on priority by business team tonight and tomorrow. 

•             Manually Process in EBS

•             Pending Putaway and INV ADJ

•             Pending stuck Receipts  

•             Pending Move Orders  

•             Finance team to verify after all the pending transactions processed in EBS

•             Business users to start using the EBS process as normal

•             Perform Physical Inventory Count in EBS (Time to be decided)

IT teams in standby for assisting business on need basis.  

Warehouse team are currently working on following activities for manual movements inside EBS

• Pending stuck Receipts
• Pending Putaway and INV ADJ
• Pending Move Orders 

Warehouse team has completed all manual entry operations for required movements inside EBS and same has been validated.

This marks the completion of all material changes for the pending COC blades.

Henceforth, marking this incident as resolved and no more further updates will be rolled out.

Thanks for your understanding, patience and co-operation.

SEV-1 Ticket: INC0099928 

It was identified an issue with the WMS Outbound Order Interface not sending to EBS full orders.

Teams Bridge call opened, analysis in progress and waiting Redwood Connect Team to look at the issue on the interface.

Redwood has applied the fix in test instance and IT is testing the same to confirm if fix is working fine

IT testing with Redwood for the fix is working fine in test instance and further validation is in progress.

Also existing affected orders are validated and investigated by Redwood and will share the list to TPI IT for further reprocessing

Vendor Redwood has applied the fix in Production.

Currently they are working on getting list of failed items which wasn't processed to give to TPI IT for further review before initiating reprocessing.

Root cause of the issue seems to come out of a fix applied by vendor on 29th August to suppress the duplicates causing missing lines processing issue.

Vendor Redwood has applied fix for 1 move order which was affected and IT teams validated and could see it got processed in WMS and Oracle EBS.

Vendor needs some few hours to process remaining missing ones (400+ orders) and will update IT once successfully processed.

Vendor Redwood still working on reprocessing of missing items.

They will inform us once it is completed for further validations and confirmation.

Vendor Redwood has provided another 1 hour to complete script for reprocessing of missing items

They will update us once completed for further validations

Reprocessed Move orders are validated from Oracle to MFG application too - hence complete flow is getting recovered.

We'll come back with next update once vendor Redwood has completed all reprocessing of missing orders.

Vendor Redwood has processed all missing items (700+ items) - complete list will be shared once they are back in office on Tuesday. So flow is reinstated (Oracle EBS --> WMS --> Oracle EBS).

TPI IT team has checked and validated all missing orders are processed and inserted properly. Also MFG flow is working properly and all items are present.

Root cause is followed up with vendor for preventive fix and other discussions.

Issue is resolved and no more further updates.

Deliver transactions not created in EBS - Totally 352 lines not created deliver records in EBS which are completed in WMS.

Currently, TPI IT team are in call with vendor for further investigation.

Currently TPI IT is working in Support Instance to do bulk update and verify if all process are properly created and receipts are getting generated.

Once updated and verified with end-to-end process flows, same will be initiated in Production after sign-off.

Currently Bulk Script is running in PROD - we are running now in batches of each day and seeing if receipts are getting created automatically.

We'll provide next update once all script processing is completed and verified by users and confirm the same.

All pending records are processed via script - business has been informed further to check on this.

IT team investigating on error records during this bulk update and looking for next steps.

Script processing status:

Total affected - 352 

Total Processed successful in script - 350

Total Errored out  - 2

Currently IT team is checking with business to process new receipt in EBS and ensure no issues further.

All Pending/errored receipts are processed - nothing outstanding from affected ones. All 352 receipts got processed into EBS.

User has also created new receipt flow and confirmed all good and BAU.

IT will be on heightened awareness/monitoring of the system and ensure no further error happens or get reported.

Deliver transactions not created in EBS - Totally 294 lines not created deliver records in EBS which are completed in WMS

Currently, TPI IT team are in call with vendor for further investigation.

We fixed initially 1 record, tested and it worked fine. Later fixed and tested for 4 records and confirmed all good.
Currently we are planning to fix remaining 289 records in bulk to have it processed - ETA ~ 1 hour. 

For new receipt creation, we are not seeing this issue. However we are actively monitoring to see same issue won't occur.
We'll keep you posted once bulk update processing is complete.

Vendor has suggested to reprocess affected record from Oracle - TPI IT has initiated the same for 1 record and found its processing as BAU.

Currently TPI IT preparing bulk script for processing remaining affected records and will check with users if all have processed and no more issues observed.

P.S :-   For new receipt creation, we are not seeing this issue. However we are actively monitoring to see same issue won't occur.

Script for updating affected items in bulk has been completed and all has been corrected properly.

Currently no more items are in affected list. 

Fix Done Yesterday Evening : We have updated EBS Receipt number in WMS which will help finance team for invoice processing. Also we have streamlined the receipt to putaway process.

P.S :-   For new receipt creation, we are not seeing this issue. However we are actively monitoring to see same issue won't occur.

From today morning 10:30 AM IST hours onwards, laptops across all the regions are getting crashed and restarting for several users. For many of them, it's successfully getting restarted and able to continue operations and for some of them, it's failing to restart.

We're currently on the bridge call involving all the IT Teams across regions and cases have been raised with Microsoft Vendor Support Team to identify the root-cause and permanent fix.

As of now, Security Team confirmed that this is not any kind of malicious activity or a Cyber Attack.

We'll provide updates periodically. Thanks for your patience. 

We've identified the root-cause of the problem.

Recent version update of the CrowdStrike sensors running on the Windows laptops caused this bluescreen error and loop restarts of the laptops. We're working on pushing a global updates on all the machines to uninstall the sensor. 

Thanks for your patience. 

We're still working on a fix for this issue. In the meanwhile, we strongly advise users to not turn on their laptops unless its production critical and really necessary for delivery standpoint.

This is a global issue and several organizations' across the globe are impacted. We'll send out communications when the issue is fixed so that you can use your laptops without any issues.

Incase, you're using your laptops, please contact your local IT Team to uninstall the CrowdStrike software in your machine immediately. Once uninstalled, kindly refrain from accessing any unwanted sites and avoid downloading any content from the internet. 

Thanks for your patience. We will continue to send out periodic updates.

Security Team is still following up with CrowdStrike Vendor for permanent fix. 

If you're using your laptops, please contact your local IT Team to uninstall the CrowdStrike software in your machine immediately. Once uninstalled, kindly refrain from accessing any unwanted sites and avoid downloading any content from the internet. 

Thanks for your patience. We will continue to send out periodic updates.

IT team has been recovering laptops and PCs of affected users across the regions. As of 9:00 AM AZ time, many business critical user computers and leadership PCs are restored and the effort is ongoing in Mexico region. 

If you're using laptop, please contact your local IT Team to restore your machine. Once recovered, kindly refrain from accessing any unwanted sites and avoid downloading any content from the internet. 

This is a global issue impacting multiple organizations. A permanent fix is still being worked by Microsoft and Crowd strike.

Thanks for your patience and co-operation. Periodic updates will follow.

IT Team has restored majority of the impacted laptops and desktop computers in all regions. This enabled to continue regular operations and resume business activities.

Team has also reached out many users proactively and continued the restoring process. If you are one of the users (on-site or remote) who is still impacted, please reach out to your local IT support personal as mentioned in below list. Remote users will be supported over the phone and doesn't require to ship the laptop. 

As of 2:00 PM AZ Time, all critical applications, including Oracle ERP, tpiOne, QMS are working normal. 

Next update will follow on 07/20 9:00 AM AZ time. 

IT Teams from each region restored laptops that are identified and reported. If you are one of the users (on-site or remote) who is still impacted, please reach out to your local IT support by calling their contact Phone number or messaging in teams, as mentioned in below list. 

Regional IT teams are available to support over the weekend. Remote users will be supported over the phone and doesn't require to ship the laptop. 

User experiencing issues while login into Oracle EBS using Okta SSO.  Users already logged into EBS - do not have any issues - they can continue as normal.

We have opened a bridge call and working with the teams - will keep posted on the updates soon. 

The issue has been resolved.  The SSO - Okta login is working fine now. 

Thanks,

TPI IT