|
|
Enron Mail |
The August end of month close is behind us. Once again, we learned some
things. We made progress.....but we also have issues to work. Summary On Improvements For This End Of Month Close I would like to congratulate the users, the application team and the DBA team for a much improved end of month close for August gas. This would not have been possible without the significant commitment, focus and teamwork exhibited by all the folks involved in this effort. The most tangible measure of progress over the July close : We were able to keep all users in the system throughout the final week. No one was kicked off the system. The gas accounting close process finished on time. Key users and IT folks didn't have to work significant overtime to finish the process. Improvements arose from : Upgrading the Unify system to Sybase 12 with extra server memory. This reduced system 'locking and blocking' and made it possible to keep all users in the system throughout the week. Nightly settlements for gas invoicing also ran much faster. Many thanks to the DBAs - Jim Ogg and Kishroe Bharathi-- for many late nights and weekends. Many thanks also to the application team and the users for lots of testing before the production rollout. Conducting a pre-planning session that identified new end of month techniques to improve efficiency. The users generated several new ideas including the idea to pre-process PMAs the night before running the current mass draft. This meeting provided several new approaches, a clear plan and cohesive communication throughout the end of month process. Many thanks to the all the users as well as Bob Superty, Bryce Baxter, Melissa Ratnala, Dave Nommensen and Chris Schomer for making this happen. Many of these ideas will now be part of a standard process. The performance monitoring and tuning team reacted quickly to address changes in system performance. Many thanks to Matt Pena and the rest of this team for keeping the system up and running as fast as possible during the day. Release of a new tuned version of the Accurals process that runs much faster. Issues, Challenges, Action Items While we made progress this month, there are still several significant issues and opportunities for improvement. The following is a list of the things that 'went bump in the night' and the action items we've identified. 1. We experienced several Sybase 12.0 errors early in the week that impacted gas business. All of these problems are now fixed. This had the following business impacts: The bridgeback of deals to Sitara was delayed. These deals were bridged manually before the end of month valuation. We also fixed the code problem that generated th original problem. Global Systems replication to Unify was down on Monday. This problem was fixed with a software patch from Sybase Monday evening. All affected deals were then uploaded. Nominations were completed for these deals on Monday through manual estimation by the schedulers. Communication between the DBA group and the application group was delayed on the Global Systems replication problem. This delayed our ability to notify users of the problem. Communication on problems will be be improved in the future. The default assumption will be to notify key application and infrastructure personnel as soon as any serious problem is identified. 2. We experienced a hardware issue with a bad server CPU at 3:00AM on Thursday. As a result, database backups ran into the morning when the server came back up. This created slow system response and delayed logins for users until 10:00 AM. This problem is now fixed. Action Item: Kishore will coordinate with Randy Matson from the infrastructure group to coordinate a stress test of all production server components. This will allow us to see if we can find any other marginal components. This will be completed by September 18. 3. All service PMAs for ENA and HPLC were not picked up by the Settlements process. HPLC PMAs were run manually by the users. For ENA, missing PMAs were immaterial and the users decided to close with these results. We have now found and fixed the error that caused this problem. Action Item: Chris Schomer is having his team investigate this problem to see if this is just a Sybase 12.0 issue or whether this error could have impacted previous months. A determination should be available by September 15. 4. Enpower deal load did not load deals with referenced prices correctly. These deals have now been successfully loaded and are currently completed through the mass draft process for power. We have identified and fixed the software error that caused this problem. 5. Power mass draft is now running very slow as compared to prior months. We are currently working to complete the Power mass draft process. It will be completed by the user deadline for close of business Wednesday Sept. 6. Action Item: Chris Schomer's team is investigating why the Power mass draft is running so slow. At present we do not have a definitive answer. We hope to develop a definitive answer and action plan by September 15. Our current hypothesis for why we have this problem includes : Data changes - we now have a much higher volume of hourly priced deals Potential database index issues. We will re-build all power database indexes and re-compile the stored procedures Sybase optimizer flakiness. We often see Sybase suddenly start to have processing problems for no apparent reason. Update On Future Tactical And Strategic Improvements We have several pending tactical Unify performance improvements that are in process. These include: 1) Isolating Unify Gas on a server of it's own 2) Delivery of a middle tier mass draft process to reduce locking and blocking 3) Upgrade of the middle tier mass draft process to allow for drafting at a deal level (rather than just a counter-party level) 4) Delivery of the nominations and balancing process on a middle tier architecture. Ramesh Rao and his team have the strategic projects progressing on schedule. We are seeing significant performance gains in our testing of SQL Server over Sybase. We also have identified a couple of hardware vendors that could provide very large, scaleable computing environments for this type of Unify architecture. We plan to have all critical evaluation testing completed by the end of September as planned. We will present our recommendations along with a high-level strategic delivery plan by early October. Let me know if you have any questions or concerns -- x39694 or (713) 503-4873. Thanks.
|