Our audit concerning the development and implementation of the California Department of Veterans Affairs’ (CalVet) Enterprise‑Wide Veterans Home Information System (system) revealed the following:
- The system has not improved the efficiency of the homes’ process for documenting medical care nor has it reduced reliance on paper because of system flaws.
- System instability and concerns about functionality resulted in CalVet implementing fewer system functions at some homes, thereby limiting CalVet’s ability to provide more efficient care for veterans.
- CalVet’s project management failed to promptly recognize the severity of the system’s problems and address them.
- The California Department of Technology (Technology Department) lacked documentation to demonstrate that its efforts to ensure that the settlement with the system contractor was in the State’s best interest.
- CalVet did not exercise adequate oversight of its system project.
- It did not complete or partially completed six of the 12 required management oversight plans to ensure effective project management
- It hired one contractor to provide both independent project oversight and independent verification and validation services, and those services were inadequate.
- The Technology Department did not adequately fulfill its oversight responsibilities for CalVet’s system project.
- It did not identify significant concerns until August 2012, after CalVet had already spent $15 million on the system.
- CalVet’s limited documentation for the selection of the system contractor prevents it from demonstrating it complied with state contracting requirements.
- It could not provide proposals for six of the seven vendors or the evaluations for three proposals, including the winning proposal.
- CalVet approved payments totaling $733,000 for some of its key system implementation deliverables but could not provide adequate documentation of receiving these final deliverables.
Results in Brief
The mission of the California Department of Veterans Affairs (CalVet) is to serve California’s veterans and their families. Further, the mission of CalVet’s veterans homes is to provide the State’s aged and disabled veterans with rehabilitative, residential, and medical care and services in a homelike environment at the State’s eight veterans homes.
In 2006, to support the healthcare information needs of its five planned new veterans homes and its three existing homes, CalVet decided to implement a computerized information system, called the Enterprise‑Wide Veterans Home Information System (system), to ensure that veterans receive consistent and integrated care in any veterans home. In its feasibility study report (FSR)—the planning document used to assess the practicality of a proposed project—CalVet described the need for a system that would make its processes more efficient and reduce its reliance on paper records. The FSR projected the cost of the system to be $34 million and estimated that it would be complete in 2010. Actual implementation of the system began in mid‑2012. However, the system has not achieved the expected or planned efficiencies that CalVet anticipated. Specifically, the system has not improved the homes’ process for documenting medical care nor has it reduced their reliance on paper records because of flaws that staff encountered with the system. Additionally, system instability and concerns about functionality resulted in CalVet implementing fewer system functions at some homes than originally planned, thereby limiting CalVet’s ability to provide more consistent, efficient care for veterans. Further, although some of CalVet’s claims of noncompliance with state and federal regulations are overstated, its need to take additional steps outside of the system to remain compliant demonstrates that it has not achieved its goal of implementing a system that improved that compliance with regulations.
CalVet’s project management failed to promptly identify and address the system’s functionality issues. Although it was aware of such issues as early as mid‑2012, CalVet’s project management did not begin to take steps to address those issues until late 2013. By that time, it had spent nearly $6 million since staff began notifying them of the functionality problems. The steps CalVet’s project management did take—pausing the implementation of the system, sending the system contractor a cure letter to identify specific documents required in its contract that the system contractor had not provided, conducting an assessment of the system, and ultimately ending the contract with the system contractor in December 2014—did not fix the problems. Although the California Department of Technology (Technology Department) facilitated the contract dispute discussions and the negotiations between CalVet and the contractor, the Technology Department could not provide sufficient documentation to demonstrate its efforts to ensure that the agreements reached were in the best interests of the State. When CalVet signed the settlement agreement in December 2014, it had spent $26.2 million on its implementation of the system. By the time it completed its post‑implementation evaluation report in June 2015, its project costs totaled $27.9 million. Although CalVet is now in the process of identifying a replacement system, it plans to continue to use the current system as best it can until it replaces the system.
We noted several key deficiencies that contributed to CalVet’s failure to implement a system that meets its needs. First, CalVet did not exercise adequate oversight of its system project. Specifically, it did not complete or partially completed six of the 12 management oversight plans state policy requires for ensuring effective project management. For the six completed plans, CalVet fully followed only four. For instance, in its configuration change control management plan, which describes the process the project team will follow to document, control, and manage changes to key project components and deliverables throughout the project, CalVet stated that the change control manager would assign an analyst to conduct impact analyses on each change request to properly understand how each one would affect project costs, scope, and timeline. However, CalVet did not consistently conduct impact analyses on the project’s change requests. Because it did not always follow this plan, it was unable to demonstrate that it properly understood the impact of various change requests on the project’s costs, scope, and timeline. Additionally, the project executive, who is charged with the highest level of project review within the CalVet organization, was not involved with oversight of the management plans. CalVet’s final project manager stated that if project management plans were not being followed during the project, the individual assigned ownership of the plan would have been responsible to escalate the problem to the project manager. Then if the project manager could not resolve the problem, it should have been elevated to the project executive. CalVet did not show that problems regarding the plans were ever escalated to the project executive.
Moreover, although CalVet hired one contractor (oversight contractor) to provide both independent project oversight (IPO) and independent verification and validation (IV&V) services for its system project, those services were inadequate. Therefore, CalVet did not identify deficiencies with the system as early as it should have. IPO provides an independent review and analysis of project management practices to determine if the project is being well managed. IV&V provides a client with technically proficient “eyes and ears” to oversee a system vendor while an information technology (IT) system is being developed and implemented. CalVet’s oversight contractor provided both IPO and IV&V services. However, according to our IT expert, separation of IV&V and IPO duties is important and provides a number of advantages to the project and to the State. An important function of IPO is to determine whether IV&V functions are being performed appropriately, and by using the same contractor to perform both functions, CalVet did not ensure that it had effective oversight for this project. For example, our IT expert stated that effective IPO should have raised concerns that IV&V was not managing requirements traceability—the tracing of project requirements throughout the project life cycle to ensure that the system meets specified contract requirements; however, the oversight contractor did not provide this. Although the oversight contractor’s IPO reports should have identified these types of deficiencies in its own IV&V work, it did not. Because the IPO reports did not identify critical work IV&V was not performing, neither CalVet nor the Technology Department received an accurate assessment of whether the system contractor’s processes were effective and whether the system reflected the agreed‑upon quality and solution.
The Technology Department also did not adequately fulfill its oversight responsibilities for CalVet’s system project. Since 2008 the Technology Department and its predecessor agencies have been responsible for providing oversight to IT projects, such as CalVet’s system, by reviewing their IPO and other oversight reports. Specifically, the Technology Department did not identify significant concerns with the system until August 2012, after CalVet had already spent $15 million on the system. The deputy director of the Technology Department’s IT Oversight Division indicated that because of the nature of its oversight—that is, reviewing reports that the oversight contractor created—and because the reports did not indicate any critical errors with the project, the Technology Department did not raise any concerns about the project. However, our IT expert reviewed the IPO reports and indicated that they were consistently lacking in critical analysis and that this omission should have raised concerns at the Technology Department.
CalVet maintained limited documentation for its process both for selecting a contractor to implement its system and for approving payment of some invoices. This limited documentation prevents it from demonstrating that it made a prudent decision in awarding the contract and in approving payment for deliverables. Specifically, it could not provide the proposals it received from six of the seven vendors that responded to the system’s request for proposals and it could not provide the evaluations for three of the proposals, including the winning proposal. Because it did not maintain proper documentation, CalVet cannot demonstrate that it complied with state contracting requirements. Additionally, CalVet accepted and approved for payment claims totaling $733,000 for some of its key system implementation deliverables even though it could not provide adequate documentation of receiving these final deliverables.
Finally, although CalVet conducted lessons‑learned sessions at various points throughout the project, it cannot demonstrate it used those lessons learned to make improvements in its subsequent implementation. CalVet did, however, capture final lessons learned and has an opportunity to incorporate these lessons in the future.
To ensure that its project management of IT projects promptly identifies potential problems and develops resolutions, by September 2016 CalVet should define the project executive and project manager responsibilities to ensure that the individuals who fill those positions take an active role in each project.
To ensure that it adequately identifies and monitors problems in its future IT projects, by September 2016 CalVet should establish a formal process for its project executive to verify that the project team prepares all of the required project management plans and other required plans. This formal process should also include a process to periodically verify that the project team is adhering to all these plans.
To ensure accountability and independence between the provision of IPO and IV&V services on future IT projects, by September 2016 CalVet should establish a policy requiring it to use separate contractors for IPO and IV&V services when IPO services are not provided directly by the Technology Department.
To ensure that it complies with state contracting laws and can demonstrate the basis for its decisions when awarding contracts, by September 2016 CalVet should establish a process to periodically verify that its staff follow state contracting requirements and maintain all required contract documentation.
To ensure that it only accepts and pays for deliverables that are complete and that meet the contract requirements, by September 2016 CalVet should establish a process to verify and maintain documentation of all contract deliverables before approving payment.
To ensure that it maximizes its opportunity to successfully implement future IT projects, by September 2016 CalVet should establish a formal process to verify that its staff conduct lessons learned sessions for all key phases of future projects.
To ensure that it can demonstrate that it is acting in the best interest of the State, by December 2016 the Technology Department should create a formal process to summarize its involvement and document key actions taken and decisions reached during agencies’ contract disputes and negotiations for the termination of a contract and maintain those documents according to its records retention schedules.
Although the Technology Department indicated that its intent is not to outsource its statutory responsibility for IPO, in any instances when its staff conduct a portfolio review of a project’s IPO, the Technology Department should, by December 2016, establish a process for its review of documents the agency’s IPO contractor creates that verifies whether these reports include critical analysis of project progress and vendor performance so that it can intervene when necessary.
CalVet indicated it understands and agrees with each of our recommendations and plans to have several completed by September 2016. The Technology Department agreed with our recommendations and indicated it will take steps to implement them.