The mission of the California Department of Veterans Affairs (CalVet) is to serve California’s veterans and their families. Further, the mission of CalVet’s veterans homes is to provide the aged and disabled veterans residing in the State’s eight veterans homes (residents) with rehabilitative, residential, and medical care and services in a homelike environment. The eight veterans homes are located in Yountville, Barstow, Chula Vista, Ventura, Lancaster, West Los Angeles, Fresno, and Redding. As of the end of 2015, they had the budgeted capacity to care for more than 2,300 veterans. Figure 1 shows the timeline of the opening of each of the homes.
Location of California Veterans Homes and Timeline of Opening Dates
Sources: California Department of Veterans Affairs’ website.
CalVet’s veterans homes provide different levels of care to their residents, including domiciliary care, residential care for the elderly, intermediate nursing care, and skilled nursing care. In addition, CalVet provides on‑site outpatient clinics at all of its homes as a service to its residents. These clinics provide comprehensive, multidisciplinary assessments as well as ongoing primary care to address residents’ routine medical needs. Table 1 shows the levels of care available at each of the homes.
Levels of Care Available at the California Veterans Homes
|Minimum care and supervision for members who are able to perform all the activities of daily living.||✓||✓||✓|
|Residential care for the elderly:|
|Provides assistance and supervision to members in activities of daily living.||✓||✓||✓||✓||✓||✓||✓|
|Intermediate nursing care:|
|Provides skilled nursing supervision and supportive care to members on less than a continuous basis.||✓||✓|
|Skilled nursing care:|
|Provides skilled nursing and supportive care to members on an extended basis, including 24-hour inpatient care, with medical, nursing, dietary, pharmaceutical, and rehabilitation services, and an activity program.||✓||✓||✓||✓||✓||✓|
Sources: California Department of Veterans Affairs’ website, Titles 12 and 22 of the California Code of Regulations.
History and Evolution of the Enterprise-Wide Veterans Home Information System
In late 2006, to support the health care information needs of its planned five new veterans homes and its three existing homes, CalVet sought approval for a new information system that could be used across its eight homes as well as in its headquarters. Former state policy required agencies wishing to launch an information technology (IT) project to prepare a feasibility study report (FSR) that describes the relative need for, cost of, and benefits to be derived from the proposed IT investment. The FSR for the proposed Enterprise‑Wide Veterans Home Information System (system) indicated that the new system would replace an existing implementation of Meditech, a commercial software product that CalVet had used since 1996 to support acute care, ambulatory care, and long‑term care operations at its homes. The FSR defined acute care services as hospital‑like services oriented toward clinically complex patients who need a high level of care; ambulatory care as care provided on an outpatient basis, similar to that provided by a physician’s office; and long‑term care as care delivered over a longer period of time and that includes skilled nursing home, rehabilitation, assisted living, and other services. The FSR stated that the primary reason for the limited use of Meditech was that the software lacked specific functionality for supporting long‑term care operations.
In its FSR, CalVet further indicated that considerable technical progress had occurred since the Meditech system was originally installed and that systems were available that could provide coverage across all care settings that CalVet offered. CalVet indicated that newer systems provided improved efficiency, better clinical documentation, and automated medication orders with edits to prevent common medication errors.
Key Business Opportunities of the Enterprise‑Wide Veterans Home Information System
San Rafael and Novato
- Improve care management by providing information on care quality and efficiency.
- Improve clinical quality, regulatory compliance, and reimbursement by improving clinical documentation.
- Reduce vulnerability to local disasters by reducing reliance on paper records.
Source: Enterprise-Wide Veterans Home Information System Feasibility Study Report.
The proposed new system was intended to be a catalyst to transition CalVet from semi‑autonomous homes to an eight‑home integrated system of care. As shown in the text box, in its FSR CalVet described the key business opportunities to be realized from the new system.
According to the FSR, CalVet’s implementation of a new system presented an opportunity to take an affirmative step toward its vision of ensuring that veterans receive consistent and integrated care in any CalVet health care facility in California. It indicated that with the Meditech system, when a veteran transferred between facilities, staff at the new facility were unable to access the veteran’s medical history electronically. This lack of access to care information across facilities hampered clinicians’ ability to provide optimal care and led to increased costs, among other effects. For example, CalVet noted that patients’ tests might need to be repeated when information was not available. The new system was also intended to eliminate the need for cumbersome workarounds and unnecessary paperwork, to automate medication orders, to reduce the potential for medication errors, and to improve the quality of care veterans receive. The FSR estimated the cost of the system at nearly $34 million and gave a projected completion date of December 2010.
In January 2007, the California Department of Finance (Finance) approved CalVet’s FSR for the system. By the time CalVet issued its first special project report (SPR) for approval in August 2008, the project was already experiencing significant delays.1 The FSR stated that CalVet would release the vendor request for proposal (RFP) in September 2007, but delays occurred in the review, validation, and updates to the original system requirements. Further, CalVet added two additional homes (Fresno and Redding) to the scope of the project. These changes, in part, required the creation of the first SPR. Additionally, CalVet took more than six months longer than planned to contract for independent project oversight (IPO) and independent verification and validation (IV&V) services. The SPR indicated that the new release date for the RFP would be November 2008, a delay of 14 months; however, the RFP was not actually released until February 2009, a further delay of three months.
Although the RFP was issued, it was canceled in June 2009, as described in the second SPR that CalVet submitted, because only one vendor submitted a proposal and that proposal did not address all of the required system functionality. A request for information (RFI) was issued in July 2009 to a broader vendor pool to determine their ability to meet CalVet’s system requirements. The outcome of the RFI indicated that none of the vendors could meet all of CalVet’s mandatory requirements; as a result, CalVet revised the RFP. For instance, according to the second SPR, the revised RFP listed significantly fewer mandatory requirements and contained a new “highly desirable” category of requirements that were not mandatory. The second RFP was released in November 2009. In December 2010, CalVet awarded the project to Solutions West Consulting, LLC (Solutions West; later Brekken Technology, Inc.) as its system contractor. The project was contained in two contracts. The first contract and two subsequent contract amendments of work included the purchase of system hardware and software, the installation and configuration of the system, and the training of CalVet staff on how to use the new system. The second contract provided for the post‑implementation maintenance and support services for the system by the system contractor. CalVet’s third and final SPR, in January 2013, adjusted the projected cost of the system to $36.7 million.
CalVet’s headquarters and its homes in Barstow, West Los Angeles, Lancaster, and Ventura started using selected modules of the system during the system’s pilot implementation in May 2012. The home in Chula Vista implemented and began using the system in June 2013. The final three homes—in Yountville, Fresno, and Redding—implemented and began using modules of the system in November 2013. The eight homes’ use of the system modules currently varies. According to the manager of the project management office (PMO manager) of CalVet’s Information Services Division (ISD), the system was not fully implemented and used at all of the homes because of certain functionality problems staff experienced with the system.
Although both CalVet and the California Department of Technology (Technology Department)—the department that now has responsibility for oversight of state IT projects—had identified project delays during the development and early implementation of the system, according to CalVet’s June 2015 response to questions from the Legislature regarding the project, new leadership in CalVet’s ISD reported critical project problems to the Technology Department in December 2013.2 According to the final project manager for the system, CalVet identified these problems in late 2013 through a review of outstanding unpaid invoices, many of which lacked sufficient documentation to support the work performed. The June 2015 response document stated that the critical project problems CalVet identified related to project oversight, project management, contract management, contract deliverables, and the viability of the system contractor. The response document noted that to resolve these deficiencies, CalVet effectively suspended the project in December 2013 and, after a 12‑month period for reevaluation and remediation, terminated the implementation contract with the system contractor and a system support contract with the software vendor in December 2014. The response document also stated that CalVet and the Technology Department mutually agreed to close out the system project in January 2015. CalVet agreed to pay the system contractor $350,000 to settle the implementation contract. By the time CalVet reached its settlement agreement with the system contractor, it had spent $26.2 million, including the settlement costs. These costs had increased to $27.9 million when CalVet completed its post‑implementation evaluation report in June 2015. Figure 2 shows key milestones during the course of the project.
Enterprise-Wide Veterans Home Information System Project Timeline
Sources: CalVet’s FSR and SPRs for the Enterprise-Wide Veterans Home Information System (system), FSR and SPR approval letters, contracts for the system, project management documents, cure letter, the implementation contract settlement agreement, post‑implementation evaluation report—June 2015, and the Technology Department’s system briefing document.
* The cure letter identified the contract requirements that the system contractor had not met and required that the system contractor meet those requirements by February 13, 2014.
† According to steering committee meeting minutes, direction was given to close out the system project through the post‑implementation evaluation report (PIER) and initiate efforts to replace the system. In June 2015 CalVet submitted its PIER to the Technology Department to support its request for the Technology Department to consider the system project complete and to terminate project reporting. At that point, CalVet’s project costs totaled $27.9 million. According to CalVet’s deputy secretary of veterans homes division, CalVet continues to use the system as best it can until it completes the process of finding a replacement system.
Changing Roles and Responsibilities for Project Approval and Oversight
Both the Technology Department and the Department of General Services (General Services) have had certain responsibilities for overseeing the State’s IT project procurements, and as shown in Figure 3 on the following page, each entity’s roles have changed since January 2007 when Finance approved CalVet’s FSR for the system. Figure 3 also identifies the roles and responsibilities held by oversight agencies at key milestones during the development and implementation of CalVet’s system project. Effective January 2008, a change in state law transferred the authority to approve, suspend, or terminate IT projects from Finance to the Office of the Chief Information Officer—the predecessor entity to the Technology Department. In December 2010, when CalVet executed the contract to develop and implement the system, General Services had responsibility to review state agencies’ IT RFPs. Then state law effective in 2011 required General Services and the Technology Department to review all IT RFPs. Subsequent legislation operative in July 2013 transferred General Services’ share of this authority as well as General Services’ authority over IT project procurement to the Technology Department. This shift in responsibilities from General Services to the Technology Department, and the increase in the Technology Department’s responsibilities, reflects the Legislature’s conclusion that the unique aspects of IT projects and their importance to state programs warrant a separate acquisition authority.
Information Technology Roles and Responsibilities by Key Milestone for the Enterprise-Wide Veterans Home Information System Project
Sources: Government Code, sections 11545 and 11546 and Public Contract Code, sections 12100 and 12104.
* Until July 2013, the Technology Department was known as the California Technology Agency and before that the Office of the State Chief Information Officer.
† IT project approval and oversight include numerous activities, such as evaluating IT projects based on the business case justification, resource requirements, proposed technical solution, project management, oversight and risk mitigation approach, and compliance with statewide strategies, policies, and procedures.
State policy requires departments to implement independent oversight for all reportable projects. The Technology Department, which assumed IT project oversight responsibility under state law effective in 2008, generally provides the IPO services while state entities undertaking the IT projects are responsible to contract for the IV&V services. IPO services provide an independent review and analysis of project management practices to determine if the project is being well managed. IV&V services provide a client with technically proficient “eyes and ears” to oversee a system vendor while an IT system is being developed and implemented, and they also provide an early warning of process and technical discrepancies, issues, and problems that might not otherwise be detected until late in testing or implementation For its system, CalVet contracted with a single outside consultant (oversight contractor) to provide both IPO and IV&V services.
Scope and Methodology
The Joint Legislative Audit Committee (audit committee) directed the California State Auditor to conduct an audit of the development and implementation of CalVet’s Enterprise‑Wide Veterans Home Information System. Table 2 outlines the audit committee’s audit objectives and our methods for addressing them.
|1||Review and evaluate the laws, rules, and regulations significant to the audit objectives.||
|2||Evaluate the level of oversight the California Department of Veterans Affairs (CalVet) management exercised regarding development and implementation of the Enterprise‑Wide Veterans Home Information System (system), including whether the executive office was involved in making key decisions and ensuring that the system met the needs of the veterans homes.||
|3||Assess whether CalVet followed laws, rules, regulations, policies, and best practices when selecting vendors for the system, including, to the extent possible, those prohibiting a conflict of interest during the selection process.||
|4||Determine the estimated and actual implementation costs and timeline for the system as well as the number of and reasons for change orders and contract amendments.||
|5||Determine whether the original project requirements, as defined by the scope of work, were delivered during implementation of the system project.||
|6||Evaluate the steps CalVet took when project variances were identified. To the extent possible, determine whether CalVet could have identified problems with the system earlier.||
|7||Review the role of the California Department of Technology (Technology Department) in this project and evaluate whether it fulfilled its roles and responsibilities.||
|8||Determine the steps CalVet has taken to improve the functionality of the system and what affect any lack in functionality has had on quality of care and cost and efficiency of the homes’ operations. In addition, determine whether CalVet has documented lessons learned for use in future phases of system implementation.||
|9||Identify the level of system functionality and use within CalVet’s veterans homes and administrative offices and evaluate CalVet’s efforts to train staff and otherwise address resistance to using the system.||
|10||Review and assess any other issues that are significant to the audit.||
Sources: California State Auditor’s analysis of the Joint Legislative Audit Committee’s audit request number 2015-121, and information and documentation identified in the column titled Method.
Assessment of Data Reliability
In performing the audit, we obtained financial records from the California State Accounting and Reporting System (CalSTARS) for CalVet’s expenditures related to its system. The U.S. Government Accountability Office, whose standards we are statutorily required to follow, requires us to assess the sufficiency and appropriateness of the computer‑processed information that we use to support our findings, conclusions, and recommendations. Specifically, we used expenditure reports from CalSTARS from July 2007 to June 2015. For each fiscal year, we used these data to report CalVet’s total expenditures for its system implementation. However, we did not conduct accuracy or completeness testing on these data because, in accordance with its records retention policy, CalVet did not maintain source documents for many of its historical transactions that we would need to test. As a result, any testing we performed would be incomplete. Further, CalVet stores the source documents it does maintain at locations throughout the State, making this testing cost‑prohibitive. Thus, we determined that the State’s CalSTARS data were of undetermined reliability for the purposes of this audit. Although this determination may affect the precision of the numbers we present, there is sufficient evidence in total to support our audit findings, conclusions, and recommendations. To gain some assurance of the accuracy of CalVet’s system transactions recorded in CalSTARS, we tested a selection of expenditures and hours worked for fiscal years 2013–14 and 2014–15, and we determined that CalVet accurately recorded the transactions we reviewed.
1 An SPR provides a summary of proposed changes to the original project cost, schedule, or scope. It is generally required when the project cost or total financial program benefits deviate or are anticipated to deviate from the original by 10 percent or more, or when a major change occurs in project requirements or methodology. Go back to text
2 Until July 2013 the Department of Technology was known as the California Technology Agency, and before that, it was the Office of the Chief Information Officer. Go back to text