San Diego Water has taken a giant technological leap forward. It has gone from a 15-year-old monitoring system operating with tone telemetry on leased lines to a state-of-the-art supervisory control and data acquisition (SCADA) system that integrates numerous technology systems throughout the enterprise.
Less than two years ago, an alarm sounding at the San Diego Water Departments control center meant that a crew would be dispatched to the trouble site (as far as 30 miles away) to assess the problem. Based on their assessment, other crews would be called in, and those crews often would discover that yet additional work crews were needed.
Insufficient data with no control meant that all alarms were treated as urgent, since there was no way of knowing the importance of an alarm or the attention it required. A simple problem routinely meant numerous crews driving hundreds of miles, often resulting in many hours of lost productivity and overtime. This was no way to run a business.
Today, when an alarm goes off at San Diego Waters new SCADA control center, an operator calls up a screen that shows the exact problem. The operator quickly determines the importance of the alarm and simply makes an adjustment or, if necessary, dispatches the appropriate crew directly to fix the problem. SCADA now provides monitoring of the entire distribution system, including the citys three treatment facilities. Full control functions now are being implemented in the second phase of the SCADA project.
"The difference is like night and day," said Charlie Sims, San Diego Waters instrumentation and control supervisor, and the projects technical leader. "Now were in control, were able to see whats going on and make changes without sending someone out to assess every alarm we have a tool to make a decision about a problem before it becomes a major expense or catastrophe."
There have been added benefits as well. Using technology as a strategy, San Diego Water is leveraging its new SCADA systems value by integrating it with TIDES (Totally Integrated Data Enterprise System), linking 30 other departmental information systems.
"This robust network allows the utility to use its most valued asset (i.e.; data) to maximize its business potential," said Linda Schmidt, information systems integrator and enterprise data manager.
Perhaps the most significant legacy of this project will be its 10-year span from planning to implementation. San Diego Water is a classic example of how leadership, determination, champions, persistence and planning will result in change. Perseverance and vision finally paid off with project approval, although it required an 18 percent rate hike (over three years) and approval of a revenue bond. The project includes design and construction of the $4.3 million SCADA system, building or rebuilding pump stations, replacing pipelines, improving reservoirs, rehabbing/replacing equipment and other necessary projects.
Persistence Pays Off
Anyone who has managed a project has faced some of the obstacles that come up in the public sector (e.g., funding loss, changing requirements and priorities, changing management, new regulations leading to competing projects and budget shortfalls). San Diego Water faced all of these, in addition to the separation of its water and wastewater group into two departments. Ironically, successful water conservation efforts resulted in a reduction of revenues, further contributing to the delay of the project.
Complicating the issues even more, San Diego Water is one of the largest and most complex water departments in the United States. The department serves more than 1.2 million people populating more than 400 square miles of developed land. In addition to three water treatment plants, the century-old utility operates and maintains more than 2,700 miles of water lines, 45 water pump stations, 100 pressure zones, with more than 200 million gallons of potable water storage capacity in 27 standpipes, elevated tanks and concrete and steel reservoirs.
Since rainfall in the semi-arid desert region of the southwestern United States can vary greatly, local water availability has always been an issue. On average, the nations sixth largest city imports nearly 90 percent of its water from northern California and the Colorado River, provided by the San Diego County Water Authority (CWA) and the Metropolitan Water District of Southern California (MWD). This means San Diego Water must be very aware of its water supply and distribution system at all times, since buying water at a high price in greater-than-needed quantities is a costly mistake, ultimately affecting rates.
"Larry [Gardner, San Diego Water director] saw the importance of the project and gave it a top priority," Sims said. "Our needs were great, and he persisted."
"A common sense of urgency was critical to getting the SCADA system on board," Gardner said. "Our sense of urgency was the madness of having to operate a system with 2,700 miles of pipeline and hundreds of pump stations and pressure zones, being 99 percent blind at any moment. To be able to control it, you have to be able to see it."
Making The Technology Fit
San Diego Water and EMA Services, St. Paul, Minn., planned the project in two phases in order to maximize benefits as quickly as possible. Phase I included the design of the SCADA system and implementation of the data acquisition system. Having the SCADA system online means the utility will be able to control distribution of water during Phase II of the project, when equipment and service disruptions will be inevitable.
"This will be extremely valuable as we work on those other projects; well be able to see whats going on systemwide," Sims explained. "We can monitor it while were improving it."
The SCADA system monitors three plants and 46 other remote sites, utilizing Modicon programmable logic controllers and the Modbus Protocol, with 900 MHz radios sending signals to five mountaintop receivers where they join the Citys microwave backbone.
The control room at the Alvarado Filtration Plant houses redundant SCADA servers for the operation of the system. A third relational database server provides data to the enterprise network so as not to slow down the SCADA system server.
This unique setup, Sims explained, came about "because it became clear that we just couldnt satisfy all the applications needing data. We couldnt get specific needs identified, and those we did were constantly changing. So to keep from having to compromise our standards and to avoid producing different reports or data for many needs, we decided to give everyone direct access to the data so they can do with it whatever they want. Since its available on a separate server, it doesnt slow down any of our work on the SCADA servers."
Real-time data is stored on the server for one year before being backed up to tape or optical drives for historic reference.
Phase II is underway and is scheduled to be completed by 2001. Included are the automation of the treatment plants, upgrade of other facilities and installation of controls throughout the rest of the system, adding the treated water storage facilities, standpipes, pressure points, valve points and flowpoints.
"Well add them one-by-one and bring them online as theyre ready," Sims said. "Our database already is built to accept them as we bring them online. EMA provided technical leadership and industry ideas to ensure that the system exceeded the Departments mission."
"We continue to reengineer the human issues in order to use smart technology," Gardner said. "We are embarking on a process of creating a more flexible workforce, well-trained and certified employees, with the knowledge needed for hands-on operation of the system."
Technology As Strategy
San Diego Water is using "technology as a strategy" to maximize customer service and utility effectiveness. (See Figure 1.)
"The SCADA system is the backbone for water distribution and delivery, providing San Diego Water with an improved operator interface, increased reliability, increased data accuracy, effective data management, timely and complete report generation, enhanced communications, fully integrated information systems, and flexibility for growth," Schmidt said. (See Figure 2.)
By integrating with TIDES, the SCADA system will share information with such systems as Water Modeling, Geographic Information Systems (GIS), a Computerized Maintenance Management System (CMMS), Laboratory Information Management System (LIMS), Plant Control System (PCS) and a daily operations status system called Morning Reports.
"Our job is not just to provide efficient service, but also to be an effective business," asserted Walter Cooke, water production superintendent. "The SCADA system not only will make us more reliable and cost effective, but it will enable us to make better business decisions."
"One of the most successful aspects of this project was that we relied on the employees themselves to envision how we were going to move forward into the future," Gardner said. "We interviewed or talked to virtually every staff member in the department. That generated more than 400 ideas on how we can reengineer or reorganize this department to make it more efficient and more service-oriented. We had lots of team meetings that generated ideasa number of internal groups were working on different parts, all with a common purpose. It was a very successful process, with extensive involvement."
"It took a very collaborative effort from many people to make this project happen," Cooke added. "A lot of motivated people were involved. Leadership provided guidance but allowed everyone the freedom to be creative. Ideas were contributed by many dedicated people at all levels."
"We made it an issue that we would be involved in the project from start to finish," Sims explained. "It was important to bring everyone on the planning team together from the beginning, to share information, to understand each other and where were all coming from, to get buyin."
"We had a number of problems along the way, some expected and some not expected," Gardner readily admitted. "Some decisions wed made had to be changed because of unforeseen problems; others were changed because of the timespan for this project. We had to maintain a sense that when something went wrong, we didnt failwe were doing the best we knew, and we learned from that. Hindsight gives you 20/20 vision."
Suggestions from project team members
Instead of limiting factory tests to pre-scripted results, San Diego Water staff used real data, and then went through a rigorous 72-hour "unstructured test" to simulate various scenarios. This identified a number of problems early, enabling them to be resolved prior to installation.
Be actively involved in planning and design, as well as implementation. Seek and provide input at all stages and from all levels.
Start planning an IT integration project with a business model so you can see what you want to accomplish, suggests Schmidt. Assemble all your requirements upfront. Then look at the business processes you want to implement and how they will impact the operation overall. The next steps include developing a data model and an implementation plan.
Make sure that the specifications are tight, precise and incontrovertible, said Carl Spier, project management coordinator. "Make sure theres no area open to interpretation." He also suggests spelling out the criteria that will be used in making your selection. This was a proposal evaluation process, not the low-bid process typical of a government agency. Pre-proposal meetings were held that vendors were required to attend, so that everyone would have the same information and understand the selection process. Spier suggests rigorous reference checks, site visits and separate price proposals to be opened after the technical criteria rating process has been completed.
Although too early to quantify the ultimate results, Gardner estimates an overall annual saving of $11.5 million once the project is completed.
Gardner noted that the overtime rate in the Operations Division averaged about 1213 percent in past years, but it has already dropped to 57 percent, "since we dont have to send someone out to respond to every alarm." Overtime cuts alone are projected to save $400,000500,000 per year. (See Figure 3.)
"Were also going to realize savings in our water purchases," Deputy Director Mark Stone said. "Well be a lot more efficient at providing water appropriately throughout the system. Well deliver it more efficiently and be able to purchase it more economically."