|
|
Enron Mail |
I may have zapped some of these updates from John since he has started this. May be good in our budget review meeting on Monday with Louise to have all of these updates in hand - and any documentation that you all may have kept on performance issues prior to John starting this update process - as back up for maintenance and capital dollars to be spent on Unify in 2002 to improve performance and reliability. Can one of you get all of these updates? Thanks. --Sally
-----Original Message----- From: Warner , John Sent: Wednesday, September 26, 2001 6:13 PM To: Rao, Ramesh; Beck, Sally; Piper, Greg; Perlman, Beth; Pena, Matt; Hall, Bob M; Superty, Robert; Pickering, Mark Subject: Unify Operational Status Summary of 9/25: No major issues were encountered with Unify gas, other than periods of poor performance in the afternoon. On Unify power, we are experiencing performance degradation with our deal load process. The process is now taking 6 hours, whereas in the past it would complete in 2 hours. We restarted SQL Server on the evening of the 24th, but deal load performance did not improve on the 25th. We experienced a major issue with Unify power whereby our replication from Enpower was down for nearly an entire day. We were made aware of the problem at roughly 5:00 p.m. We notified all users to exit the system, restarted SQL Server and our database server (as previously planned), and were caught up at 10:00 pm. We then started our planned nightly processing, but deal load performance did not improve. We are working with the DBA team to improve notification in the event our replication processes are down. Summary of 9/26: Unify gas performance was spotty during the morning under heavy usage. We discovered a report being run in our production environment was contributing to our performance problems. We will (again) reinforce the need to use the reporting database for such reporting in the future. Our Sitara bridge experienced significant backlogs at roughly 2:00 pm. Upon investigation, we were receiving another SQL Server error (not the buffer latch error) which seems to indicate that processes were "sleeping" while waiting for system resources to free. At that time, the Sitara bridge process was consuming a tremendous amount of system processes so we stopped that process (after consulting with Bob Superty). Other processes in waiting immediately began executing normally, and we started the bridge roughly 30 minutes later. Summary of Critical Unify Operational Issues: Issue Status Next Steps Buffer latch errors with SQL Server have caused us to bring down Unify during business hours. Open Microsoft will not deliver a fix to us by October as planned. They have discovered that the problem is an architectural issue with SQL Server, and will require additional time to fix. In the interim, they will provide us with diagnostic tools which may help us predict when the buffer latch error will surface. Database locks are causing performance problems. Open As previously stated, we have changed our pre-emptive maintenance thresholds, and are working on other refinements of our ongoing maintenance efforts. We are also closely monitoring database activity to locate programs which are contributing to this problem. Power Deal Load processing is degrading each month. As a result, our end of month processes are bleeding over into normal business hours and causing additional performance problems. Open We have analyzed our deal load process in detail, and are performing a significant re-work. Our plan is to have the process streamlined by October's end of month processing.
|