Report 2011-116.2 Data Reliability Assessments

For the purpose of identifying the following for the period between July 1, 2008, and June 30, 2011:
- The percentage of projects closed that had a plan change document submitted after construction ended.
- The number of plan change documents that were submitted after construction ended, and the percentage of these documents that were change orders.
- The number of projects voided or canceled.

Undetermined Reliability—We performed data-set verification procedures and electronic testing of key data elements. We identified no issues when performing data-set verification procedures, but we found errors in our electronic testing, some of which we were able to correct.

We performed completeness testing by selecting 29 projects from regional files and verifying that these projects existed in Tracker; we found no errors in this testing.

We also tested the accuracy of Tracker by testing key data elements for a random sample of 29 projects and tracing the selected elements to the project files. In this sample, we found one error, so we continued testing until we had tested a total of 47 randomly selected projects and found no additional errors. However, because the division did not have a consistent method for identifying the date construction ended, we were unable to test the accuracy of this field.

Our review of existing information identified two data limitations: Tracker does not track information on any projects submitted to the division before November 1997. Further, Tracker does not identify which projects are reopened regardless of whether the project was initially recorded in the database. Because some applicants are required to pay a fee when they reopen projects, we were able to identify a portion of the reopened projects using the fee information. Although we were not able to identify all of the reopened projects, we included those we did identify in our analysis.

For the purpose of calculating the following for the period between July 1, 2008, and September 30, 2011:
- The number and total estimated dollar value of projects the division received by region and quarter.
- The average time between application receipt date and plan approval date (total plan approval time) for approved projects.
- The number of projects and average amount of total plan approval time for over-the-counter and access-only projects.
- The average bin time, plan review time, and client time for projects completing these phases of the plan review process.
- The number and total estimated dollar value of over-the-counter and access-only projects.

Undetermined Reliability—We performed data-set verification procedures and electronic testing of key data elements. The results of electronic testing identified minor errors in a few key data elements.

We performed completeness testing by tracing a haphazardly selected sample of projects from regional files to Tracker and testing the sequential numbering of the projects in Tracker, and found no errors in this testing.

We performed accuracy testing by testing key data elements for a random sample of 29 projects with plan review activity and tracing the selected elements to the project files. In addition, we selected a supplemental sample of 10 randomly selected projects with no plan review activity. We found no errors in the fields we were able to test. However, we were not able to verify the accuracy of the project receipt and plan review dates because the documentation needed to verify this information is not consistently documented or retained by the division.