Report 2011-103 Data Reliability Assessments

Emergency Management Agency
Description of Data Agency Purpose of Data
Lotus Notes Invoicing System (invoicing system)

Emergency Services uses the invoicing system to process local agencies' reimbursement requests for resources provided during an emergency response.

Magnitude of Data

Between 2006 and 2010, Emergency Services processed invoices totaling $387.9 million using its invoicing system.

Purpose of Testing Data Reliability Determination

To determine pay rates for personnel and hours billed to paying entities.

Sufficiently Reliable—We performed data-set verification procedures, electronic testing of key data elements, accuracy, and completeness testing. We identified no issues when performing data-set verification procedures or electronic testing of key data elements. To test the accuracy of the data, we randomly selected a sample of 29 records from the invoicing system and traced key data elements to source documents. We found an error in each of the six key data fields. In response to these errors, we increased our accuracy sample from 29 to 46 records for these six key data fields, and found no additional errors. Finally, we tested the completeness of the invoicing system by comparing a haphazard sample of 29 records collected from Emergency Services' office against Emergency Services' invoicing system and found no exceptions.

To compare the dollar amount of aid provided and received between 2006 and 2010 and to review various components in the amount of aid, such as the average hourly rates.

Sufficiently Reliable—We performed data-set verification procedures, electronic testing of key data elements, accuracy, and completeness testing. We identified no issues when performing data-set verification procedures or electronic testing of key data elements. To test the accuracy of the data, we randomly selected a sample of 29 records from the invoicing system and traced key data elements to source documents. We found an error in each of the three key data fields. In response to these errors, we increased our accuracy sample from 29 to 46 records for these three key data fields and found no additional errors. Finally, we tested the completeness of the invoicing system by comparing a haphazard sample of 29 records collected from Emergency Services' office against Emergency Services' invoicing system and found no exceptions.

To identify the number of firefighting resources agencies have available.

Not Sufficiently Reliable—We performed data-set verification procedures, electronic testing of key data elements, accuracy, and completeness testing. We identified no issues when performing data-set verification procedures or electronic testing of key data elements. To test the accuracy of the data, we randomly selected a sample of 29 records from the invoicing system and traced key data elements to source documents. We found that seven key data fields had two or more errors in the first three records we tested. Due to the number of errors identified, we did not continue testing the remaining sample items. Finally, we tested the completeness of the invoicing system by electronically comparing the emergency aid records data to inventory records to ensure that the data contained inventory records for all agencies responding to emergencies. We found the data to be materially complete.

Corrective Action Recommended Status of Corrective Action
We did not recommend corrective action. N/A
Description of Data Agency Purpose of Data
Response Information Management System (RIMS)

Emergency Services uses the mission request tasking form in RIMS to order and coordinate state agency resources to respond to and assist local governments and other state agencies in emergencies and disasters.

Magnitude of Data

Roughly 4,500 users—including cities, counties, special districts, and state agencies—enter data directly into RIMS.

Purpose of Testing Data Reliability Determination

To select a sample of law enforcement missions.

Undetermined Reliability—A November 2008 feasibility study report prepared by the Governor's Office of Emergency Services identified challenges reported by RIMS users, such as the lack of a consistent method for updating RIMS reports and the loss of large amounts of data due to system "timeouts" when users were completing forms. Further, a 2009 report prepared by Emergency Services' consultant found that data and information reported within RIMS were frequently inconsistent, not credible, and unusable for analyses because Emergency Services lacked standardized processes and detailed desk instructions and because it did not have a comprehensive approach to train all staff in the use of RIMS. The consultant also found that Emergency Services lacked adequate internal controls over RIMS, such as the ability to overwrite data fields when staff do not use the "update" feature. Finally, we interviewed an Emergency Services official who stated that the information in RIMS may not always be accurate. As a result, we did not conduct data-set verification procedures or completeness testing.

Corrective Action Recommended Status of Corrective Action
We did not recommend corrective action. N/A
Description of Data Agency Purpose of Data
U.S. Department of Agriculture Forest Service's National Interagency Resource Ordering and Status System (ROSS)

ROSS was designed to provide the current status of resources available to support mobilization activities, to enable dispatch offices to exchange and track resource order information electronically, and to reliably exchange emergency electronic messages that are critical to response efforts.

Magnitude of Data

ROSS contained data for nearly 31,000 incidents for the period 2006 through 2010.

Purpose of Testing Data Reliability Determination

To select fire emergencies.

Undetermined Reliability—We did not perform data-set verification procedures or completeness testing because we were unable to obtain an extract from the ROSS database.

Corrective Action Recommended Status of Corrective Action
We did not recommend corrective action. N/A

Report type

Report type
















© 2013, California State Auditor | Privacy Policy | Conditions of Use | Download Adobe PDF Reader