Report 2014-116 All Recommendation Responses

Report 2014-116: California Department of Consumer Affairs' BreEZe System: Inadequate Planning and Oversight Led to Implementation at Far Fewer Regulatory Entities at a Significantly Higher Cost (Release Date: February 2015)

Recommendation for Legislative Action

To ensure that it receives timely and meaningful information regarding the status of the BreEZe project, the Legislature should enact legislation that requires Consumer Affairs to submit a statutory report annually, beginning on October 1, 2015, that will include Consumer Affairs' plan for implementing BreEZe at those regulatory entities included in the project's third phase, including a timeline for the implementation.

Description of Legislative Action

Legislation has not been introduced to address this specific recommendation.

  • Legislative Action Current As-of: October 2016

California State Auditor's Assessment of Status: No Action Taken


Description of Legislative Action

AB 507 was introduced on February 23, 2015, and would require the Department of Consumer Affairs, on or before March 1, 2016 or thereafter when available, to submit an annual report to the Legislature and the Department of Finance that includes the department's plans for implementing the BreEZe system at specified regulatory entities included in the Department's 3rd phase of the implementation project, including a timeline for the implementation. The bill would also require the department to post on its Internet website the name of each regulatory entity that is utilizing the BreEZe system once the regulatory entity begins using the BreEZe system.

  • Legislative Action Current As-of: March 2016

California State Auditor's Assessment of Status: Legislation Introduced


Description of Legislative Action

Legislation has not been introduced to address this recommendation.

  • Legislative Action Current As-of: January 2016

California State Auditor's Assessment of Status: No Action Taken


Recommendation for Legislative Action

To ensure that it receives timely and meaningful information regarding the status of the BreEZe project, the Legislature should enact legislation that requires Consumer Affairs to submit a statutory report annually, beginning on October 1, 2015, that will include the total estimated costs through implementation of the system at the remaining 19 regulatory entities and the results of any cost-benefit analysis it conducted for phase 3.

Description of Legislative Action

Legislation has not been introduced to address this specific recommendation.

  • Legislative Action Current As-of: October 2016

California State Auditor's Assessment of Status: No Action Taken


Description of Legislative Action

AB 507 was introduced on February 23, 2015, and would require the Department of Consumer Affairs, on or before March 1, 2016 or thereafter when available, to submit an annual report to the Legislature and the Department of Finance that includes the department's plans for implementing the BreEZe system at specified regulatory entities included in the Department's 3rd phase of the implementation project, including a timeline for the implementation. The bill would also require the department to post on its Internet website the name of each regulatory entity that is utilizing the BreEZe system once the regulatory entity begins using the BreEZe system.

  • Legislative Action Current As-of: March 2016

California State Auditor's Assessment of Status: Legislation Introduced


Description of Legislative Action

Legislation has not been introduced to address this recommendation.

  • Legislative Action Current As-of: January 2016

California State Auditor's Assessment of Status: No Action Taken


Recommendation for Legislative Action

To ensure that it receives timely and meaningful information regarding the status of the BreEZe project, the Legislature should enact legislation that requires Consumer Affairs to submit a statutory report annually, beginning on October 1, 2015, that will include a description of whether and to what extent the system will achieve any operational efficiencies resulting from implementation by the regulatory entities.

Description of Legislative Action

Legislation has not been introduced to address this specific recommendation.

  • Legislative Action Current As-of: October 2016

California State Auditor's Assessment of Status: No Action Taken


Description of Legislative Action

AB 507 would have required the Department of Consumer Affairs, on or before March 1, 2016 or thereafter when available, to submit an annual report to the Legislature and the Department of Finance that includes the department's plans for implementing the BreEZe system at specified regulatory entities included in the Department's 3rd phase of the implementation project, including a timeline for the implementation. The bill also would have required the department to post on its Internet website the name of each regulatory entity that is utilizing the BreEZe system once the regulatory entity begins using the BreEZe system.

  • Legislative Action Current As-of: March 2016

California State Auditor's Assessment of Status: Legislation Proposed But Not Enacted


Description of Legislative Action

Legislation has not been introduced to address this recommendation

  • Legislative Action Current As-of: January 2016

California State Auditor's Assessment of Status: No Action Taken


Recommendation #4 To: Technology, California Department of

To help ensure the success of the BreEZe project going forward, CalTech should ensure that Consumer Affairs responds promptly to, and adequately addresses, concerns raised by its IPO specialist and its IV&V consultant.

Agency Response*

The department has implemented a process which involves summarizing concerns raised by its IPO Specialist and its IV&V consultant and the tracking Consumer Affairs' actions. This monitoring process is documented monthly in a Corrective Action Plan, which we have provided for the month of August 2015 (BreEZe CAP-1507) to CSA separately.

  • Response Type†: 6-Month
  • Completion Date: August 2015
  • Response Date: September 2015

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

60-Day Update:

Following approval of the Section 11 and subsequent execution of contract amendment #3 with Accenture, CalTech IT Project Oversight Division will follow-up with DCA and confirm DCA's completion of the conditions below which were placed on DCA for approving SPR 3.1. These are:

1. In order to ensure transparency and an appropriate level of decision making with a proper focus on scope and cost management, the DCA Project team must incorporate the BreEZe governance process and structures (including the Executive Steering Committee and Change Control Board), the roles and responsibilities of participants, their authority, and criteria for selection into a Governance Plan document within 30 business days following SPR 3.1 approval.

2. Within 30 business days following SPR 3.1 approval, the Project must provide the Department of Technology with an organization chart which shows the reporting relationships of the individuals who will be responsible for contract management, risk management, change management and schedule management. Beginning in April 2015, the Project must submit an updated project organizational chart identifying positions filled in the prior month and remaining open positions with the monthly Project Status Report (PSR) submission.

3. The Project must continue to maintain documentation of the current status and response to all Independent Verification and Validation (IV&V) and Independent Project Oversight Consultant (IPOC) findings which must be provided to the Department of Technology monthly along with the submission of monthly PSR.

CalTech will closely monitor Consumer Affairs' responsiveness and will report in its 6 month status update.

  • Response Type†: 60-Day
  • Estimated Completion Date: 12/31/2015
  • Response Date: April 2015

California State Auditor's Assessment of Status: Pending


Recommendation #5 To: Technology, California Department of

To help ensure the success of the BreEZe project going forward, CalTech should require Consumer Affairs to develop and follow all project management plans and ensure that the BreEZe project team leads receive all required training.

Agency Response*

Four BreEZe project team leads received California Project Management Methodology (CA-PMM) Boot Camp for Project Managers training since the last report. See the attached report.

In addition, all of the required plans have been completed, including those due in September 2015:

a) Quality Management Plan

b) Change Management Plan

c) Document Management Plan

The December 2015 Independent Project Oversight Report (IPOR) will be forwarded to the auditor for additional reference.

  • Response Type†: 1-Year
  • Completion Date: December 2015
  • Response Date: February 2016

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

There have been no new project team leads since the contract amendment was approved and therefore no training has been required.

Most required plans have been completed. However, the following three plans are still in process and will not be due until the middle of September 2015:

a) Quality Management Plan

b) Change Management Plan

c) Document Management Plan

We will report on the status of these plans in our next status update.

  • Response Type†: 6-Month
  • Estimated Completion Date: Sept 2015
  • Response Date: August 2015

California State Auditor's Assessment of Status: Partially Implemented


Agency Response*

60-Day Update:

Following approval of the Section 11 and subsequent execution of contract amendment #3 with Accenture, CalTech IT Project Oversight Division will follow-up with DCA and confirm DCA's completion of the conditions below which were placed on DCA for approving SPR 3.1. These are:

4. With the increase in staffing for the Project PMO and for organizational change management and because of the change in the system development methodology, the DCA project team must review and update the Project Management Plans listed below in the following priority order:

Within 30 business days of the implementation of contract amendment 3 for Design, Development, and Implementation (DDI):

a) Communications Management Plan

b) Transition Plan

Within 90 business days of the implementation of contract amendment 3 for DDI:

c) Maintenance Plan (if impacted by DDI contract amendment 3)

d) Organizational Change Management Plan

e) Quality management Plan

f) Change Management Plan

g) Document Management Plan

5. A baseline work plan for the remainder of the project must be completed and provided to the Department of Technology within 30 business days following the implementation of contract amendment 3 for DDI.

6. CalTech will also require that DCA provide updated information about the knowledge and experience of the individuals serving in the leadership and critical role positions for the project along with any plans for training for these staff. CalTech will determine the sufficiency of these plans and require that all necessary training take place within 120 days of the execution of contract amendment 3.

CalTech will closely monitor this situation and report in our 6 month status update.

  • Response Type†: 60-Day
  • Estimated Completion Date: 12/31/2015
  • Response Date: April 2015

California State Auditor's Assessment of Status: Pending


Recommendation #6 To: Technology, California Department of

To help ensure the success of the BreEZe project going forward, CalTech should document key discussions with Consumer Affairs in which significant concerns are raised about Accenture and assessments of the department's performance by the IV&V consultant and the IPO specialist.

Agency Response*

Attachment 2 will be sent separately via e-mail.

The Department's IT Project Oversight Division (ITPOD) established a procedure for documenting significant concerns and issues that are discussed in Portfolio meetings with Department executives. See Attachment 2.

  • Response Type†: 6-Month
  • Completion Date: June 2015
  • Response Date: August 2015

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

60-Day Update:

Over the next 30 days, ITPOC will establish a new procedure for documenting significant concerns and agreed upon actions discussed during meetings with Consumer Affairs.

The new procedures are expected to in place by June 2015.

  • Response Type†: 60-Day
  • Estimated Completion Date: 06/30/2015
  • Response Date: April 2015

California State Auditor's Assessment of Status: Pending


Recommendation #7 To: Technology, California Department of

If Consumer Affairs receives the necessary funding and resources to successfully implement BreEZe at the phase 2 regulatory entities and the project continues to face escalating costs, CalTech should require Consumer Affairs to conduct an analysis of the costs and benefits of moving forward with the project as planned or of suspending or terminating the project.

Agency Response*

This recommendation is no longer applicable as DCA has completed Phase 2 implementation. To date, the BreEZe project remains within the budget approved in SPR 3.1. Through December 2015, the Project continues to operate under budget. Staffing and other costs have been below anticipated levels.

As of the end of December 2015, $917,560 remains in the Unanticipated Tasks Fund for future Requests for Change (RFC).

  • Response Type†: 1-Year
  • Completion Date: December 2015
  • Response Date: February 2016

California State Auditor's Assessment of Status: Resolved

Consumer Affairs implemented phase 2 of BreEZe and, based on documentation provided by CalTech, the project did not face escalating costs. As a result, it is no longer necessary for CalTech to require Consumer Affairs to conduct an analysis of the costs and benefits of moving forward with the project as planned or of suspending or terminating the project. Thus, we have reported the status of this recommendation as resolved.


Agency Response*

Consumer Affairs has reported it will conduct an analysis of the costs and benefits of moving forward after the completion of Release #2.

  • Response Type†: 6-Month
  • Estimated Completion Date: February 18, 2018
  • Response Date: September 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

60-Day Update:

In the event that the BreEZE project continues to face escalating costs, CalTech will require Consumer Affairs to conduct an analysis of the costs and benefits of moving forward with the project as planned in SPR 3.1. Failure on the part of DCA to do so will result in a recommendation of ITPOC to the Director to suspend or terminate the project.

CalTech will closely monitor this situation and report in our 6 month status update.

  • Response Type†: 60-Day
  • Estimated Completion Date: 12/31/2015
  • Response Date: April 2015

California State Auditor's Assessment of Status: Pending


Recommendation #8 To: Technology, California Department of

Continue implementing the STAR project for state departments and ensure that its framework requires departments undertake detailed business process analyses before submitting FSRs to CalTech for approval.

Agency Response*

Final policy for the Project Approval Lifecycle, Stages 1 & 2 were issued on July 1, 2015. Stage 3 was implemented in January 2016. Stage 4 was released in August 2016.

Business process analysis is required in Stage 2 for project proposals that involve development of/or changes to business processes. In Stage 2 departments are required to document current business processes and the anticipated new business process, if possible. This is used to help evaluate viable alternative solutions and also contributes to the requirements that are in the solicitation document (RFP). The Collaborative Review Guidelines emphasize the importance to taking into account business processes. (SIMM Section 19B Stage 2 Alternatives Analysis B.5 Gate 2 Collaborative Review Guidelines)

  • Response Type†: Annual Follow Up
  • Completion Date: August 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

Final policy for the Project Approval Lifecycle, Stages 1 & 2 were issued on July 1, 2015. Stage 3 was implemented in January 2016. See attached Policy Letter (TL 16-02). Stage 4 is scheduled for final release in July 2016. It should be noted that the new Project Approval Lifecycle replaces the FSR and as such, FSRs will no longer be submitted.

Business process analysis is required in Stage 2 for project proposals that involve development of or changes to business processes. In Stage 2 Departments are required to document current business processes and the anticipated new business process if possible. This is used to help evaluate viable alternative solutions and also contributes to the requirements that are in the solicitation document (RFP).

  • Response Type†: 1-Year
  • Estimated Completion Date: July 2016
  • Response Date: February 2016

California State Auditor's Assessment of Status: Partially Implemented


Agency Response*

Final policy for the Project Approval Lifecycle, stages 1 & 2 were issued on July 1, 2015. Stages 3 & 4 are scheduled for final release in January 2016.

  • Response Type†: 6-Month
  • Estimated Completion Date: January 2016
  • Response Date: August 2015

California State Auditor's Assessment of Status: Partially Implemented


Agency Response*

60-Day Update:

CalTech is continuing to build out the Stage Gate framework. The final policy for implementation is expected to be issued in January 2016.

  • Response Type†: 60-Day
  • Estimated Completion Date: 01/31/2016
  • Response Date: April 2015

California State Auditor's Assessment of Status: Pending


Recommendation #9 To: Technology, California Department of

Continue implementing the STAR project for state departments and ensure that its framework requires departments use detailed business process analyses to justify the type of IT solution they are proposing, such as a COTS or a fully customized system.

Agency Response*

Final policy for the Project Approval Lifecycle, Stages 1 & 2 were issued on July 1, 2015. Stage 3 was implemented in January 2016. Stage 4 was released in August 2016.

Business process analysis is required in Stage 2 for project proposals that involve development of/or changes to business processes. In Stage 2 departments are required to document current business processes and the anticipated new business process, if possible. This is used to help evaluate viable alternative solutions and also contributes to the requirements that are in the solicitation document (RFP). The Collaborative Review Guidelines emphasize the importance to taking into account business processes. (SIMM Section 19B Stage 2 Alternatives Analysis B.5 Gate 2 Collaborative Review Guidelines)

  • Response Type†: Annual Follow Up
  • Completion Date: August 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

Final policy for the Project Approval Lifecycle, Stages 1 & 2 were issued on July 1, 2015. Stage 3 was implemented in January 2016. See attached Policy Letter (TL 16-02). Stage 4 is scheduled for final release in July 2016. It should be noted that the new Project Approval Lifecycle replaces the FSR and as such, FSRs will no longer be submitted.

Business process analysis is required in Stage 2 for project proposals that involve development of or changes to business processes. In Stage 2 Departments are required to document current business processes and the anticipated new business process if possible. This is used to help evaluate viable alternative solutions and also contributes to the requirements that are in the solicitation document (RFP).

  • Response Type†: 1-Year
  • Estimated Completion Date: July 2016
  • Response Date: February 2016

California State Auditor's Assessment of Status: Partially Implemented


Agency Response*

Final policy for the Project Approval Lifecycle, stages 1 & 2 were issued on July 1, 2015. Stages 3 & 4 are scheduled for final release in January 2016.

  • Response Type†: 6-Month
  • Estimated Completion Date: January 2016
  • Response Date: August 2015

California State Auditor's Assessment of Status: Partially Implemented


Agency Response*

60-Day Update:

CalTech is continuing to build out the Stage Gate framework. The final policy for implementation is expected to be issued in January 2016.

  • Response Type†: 60-Day
  • Estimated Completion Date: 01/31/2016
  • Response Date: April 2015

California State Auditor's Assessment of Status: Pending


Recommendation #10 To: Technology, California Department of

Continue implementing the STAR project for state departments and ensure that its framework requires departments use detailed business process analyses to inform the resulting RFPs and that they ensure that the RFPs contain adequate requirements.

Agency Response*

Final policy for the Project Approval Lifecycle, Stages 1 & 2 were issued on July 1, 2015. Stage 3 was implemented in January 2016. Stage 4 was released in August 2016.

Business process analysis is required in Stage 2 for project proposals that involve development of/or changes to business processes. In Stage 2 departments are required to document current business processes and the anticipated new business process, if possible. This is used to help evaluate viable alternative solutions and also contributes to the requirements that are in the solicitation document (RFP). The Collaborative Review Guidelines emphasize the importance to taking into account business processes. (SIMM Section 19B Stage 2 Alternatives Analysis B.5 Gate 2 Collaborative Review Guidelines)

  • Response Type†: Annual Follow Up
  • Completion Date: August 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

Final policy for the Project Approval Lifecycle, Stages 1 & 2 were issued on July 1, 2015. Stage 3 was implemented in January 2016. See attached Policy Letter (TL 16-02). Stage 4 is scheduled for final release in July 2016. It should be noted that the new Project Approval Lifecycle replaces the FSR and as such, FSRs will no longer be submitted.

Business process analysis is required in Stage 2 for project proposals that involve development of or changes to business processes. In Stage 2 Departments are required to document current business processes and the anticipated new business process if possible. This is used to help evaluate viable alternative solutions and also contributes to the requirements that are in the solicitation document (RFP).

  • Response Type†: 1-Year
  • Estimated Completion Date: July 2016
  • Response Date: February 2016

California State Auditor's Assessment of Status: Partially Implemented


Agency Response*

Final policy for the Project Approval Lifecycle, stages 1 & 2 were issued on July 1, 2015. Stages 3 & 4 are scheduled for final release in January 2016.

  • Response Type†: 6-Month
  • Estimated Completion Date: January 2016
  • Response Date: August 2015

California State Auditor's Assessment of Status: Partially Implemented


Agency Response*

60-Day Update:

CalTech is continuing to build out the Stage Gate framework. The final policy for implementation is expected to be issued in January 2016.

  • Response Type†: 60-Day
  • Estimated Completion Date: 01/31/2016
  • Response Date: April 2015

California State Auditor's Assessment of Status: Pending


Recommendation #11 To: Technology, California Department of

To ensure that IT projects have the oversight needed to better position them for success, CalTech should require state departments to follow its IT policies, including developing all necessary plans and receiving all required training.

Agency Response*

Statewide Information Management Manual (SIMM) 45 updated for departments to follow CDT policies.

  • Response Type†: Annual Follow Up
  • Completion Date: March 2017
  • Response Date: November 2017

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

California Project Management Framework (CA-PMF) was published in May 2016. See TL 16-04. The CA-PMF offers guidance and insight on project management methods and approach (through the use of resources, tools, and templates), as well as narratives detailing the justifications for why specific activities should be performed. The CA-PMF is designed to be scalable to accommodate the various needs of projects.

Prior to completion of the CA-PMF, California Department of Technology (CDT) decided to separate policy mandates from project management guidance. In particular, CDT determined that the priority for policy mandates should be on reportable IT projects. As a result, CDT modified the IT Project Oversight Framework - SIMM 45 in July 2016 to include the oversight related components previously published in the CA-PMM.

These are:

- Project Management Risk Assessment

- Project Complexity Assessment

- Project Status Reports

- Independent Project Oversight Reports

Information Technology Project Oversight Division (ITPOD) has drafted an escalation process to raise awareness and foster appropriate actions concerning significant risks and issues to increasingly higher levels of CDT leadership. ITPOD has also drafted a Corrective Action Plan framework to notify departments that remediation is required. ITPOD intends to incorporate both of these processes to the IT Project Oversight Framework after state entities are provided with the opportunity to ask questions or provide comments - anticipated to be completed by December 2016.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: December 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Not Fully Implemented


Agency Response*

Work on revisions to the (CA-PMM) is underway with implementation anticipated in July 2016. The new version will be re-named to: California Project Management Framework (CA-PMF). Release of the new CA-PMF will include the a policy clarification that departments must follow the CA-PMF or another documented methodology based on PMBOK (Project Management Institute Project Management Body of Knowledge). CA-PMM was written as a set of guidelines - not requirements.

ITPOD will document deficiencies concerning compliance with IT policies and when these are not remediated within a reasonable period of time, ITPOD will use an escalation process to bring greater attention to the resulting risks and issues to increasingly higher levels of leadership. Updates to the ITPOD Project Oversight Escalation process are under review, and implementation is expected March 2016.

  • Response Type†: 1-Year
  • Estimated Completion Date: July 2016
  • Response Date: February 2016

California State Auditor's Assessment of Status: Pending


Agency Response*

Work on CA-PMM is underway with implementation anticipated in July 2016. Updates to the Project Oversight Escalation process are in development and implementation is planned for September 2015.

  • Response Type†: 6-Month
  • Estimated Completion Date: July 2016
  • Response Date: August 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

60-Day Update:

CalTech is undertaking a project to update and enhance the CA-PMM. This will provide the opportunity to strengthen the direction given to state entities concerning the Department's IT policies. At the same time, ITPOC is enhancing and updating the Project Oversight Escalation process to ensure that ITPOC management is apprised of project compliance deficiencies and so that the recommendation for remediation action are developed.

  • Response Type†: 60-Day
  • Estimated Completion Date: 12/31/2015
  • Response Date: April 2015

California State Auditor's Assessment of Status: Pending


Recommendation #12 To: Technology, California Department of

To ensure that IT projects have the oversight needed to better position them for success, CalTech should ensure that departments obtain IPO and IV&V services as soon as an IT project is approved. Additionally, CalTech should document key discussions with any department in which the IV&V consultant and the IPO specialist raise significant concerns about a project, and ensure that the respective department responds promptly to, and adequately addresses, the concerns that the IPO specialist and the IV&V consultant raise.

Agency Response*

Written instructions have been completed and incorporated into SIMM 45 on the requirement for obtaining both IPO and IV&V services.

  • Response Type†: Annual Follow Up
  • Completion Date: September 2017
  • Response Date: November 2017

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

ITPOD established the new positions approved in the FY 2016-17 State Budget. Efforts to fill these positions are on-going.

Starting in the latter half of 2015, ITPO began assigning staff on a part-time basis to provide assistance and evaluate departments going through PAL. This facilitates the planning process for ITPOD concerning when a project will be approved in Stage 4 and then will require the assignment of a full time oversight resource. Projects approved at stage 4 may have conditions that will be required depending on any specific risks or issues identified during the PAL. The assigned ITPOD staff will monitor compliance as the assigned project progresses.

For more than two years, ITPOD has included significant or unanswered findings from IV&V in the Independent Project Oversight Report (IPOR) on all projects subject to CDT oversight. And, as mentioned above, ITPOD has drafted an escalation process to raise awareness and foster appropriate actions concerning significant risks and issues to increasingly higher levels of CDT leadership. ITPOD has also drafted a Corrective Action Plan framework to notify departments that remediation is required. ITPOD intends to incorporate both of these processes to the IT Project Oversight Framework after state entities are provided with the opportunity to ask questions or provide comments - anticipated to be completed by December 2016.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: December 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Not Fully Implemented


Agency Response*

The Department submitted a Budget Change Proposal for additional ITPOD staff, which has been included in the Governor's budget for FY 16/17. ITPO staff will be assigned to projects that are determined to be medium or high complexity in Stage 2 of the Project Approval Lifecycle. Projects approved at stage 4 may have conditions that will be required depending on identified risks or issues. The assigned ITPO staff will monitor compliance as the assigned project progresses.

As of June 2015 and reported in the 6th month update, the Department's IT Project Oversight Division (ITPOD) established a procedure for documenting significant concerns and issues that are discussed in Portfolio meetings with Department executives. Updates to the Project Oversight Escalation process are under review, and implementation is expected March 2016.

  • Response Type†: 1-Year
  • Estimated Completion Date: July 2016
  • Response Date: February 2016

California State Auditor's Assessment of Status: Partially Implemented


Agency Response*

The Department continues to have discussions with the Department of Finance about an appropriate way to augment the ITPOD staffing allocation when new projects are expected to begin.

See the above status report (Attachment 2) for recommendation # 6 regarding discussing significant concerns during meetings with departments. The escalation process is planned for issuance September 2015.

  • Response Type†: 6-Month
  • Estimated Completion Date: September 2015
  • Response Date: August 2015

California State Auditor's Assessment of Status: Partially Implemented


Agency Response*

60 Day Update:

CalTech will engage the Department of Finance in a discussion about an appropriate way to augment the ITPOC staffing allocation when new projects are expected to begin.

Over the next 30 days, ITPOC will establish a new procedure for documenting significant concerns and agreed upon actions discussed during meetings with departments. This process will include follow-up and escalation procedures.

This new procedure is expected to be in place by June 2015.

  • Response Type†: 60-Day
  • Estimated Completion Date: 06/30/2015
  • Response Date: April 2015

California State Auditor's Assessment of Status: Pending


Recommendation #13 To: Technology, California Department of

To ensure that IT projects have the oversight needed to better position them for success, CalTech should require state departments to appropriately address deficiencies identified in CalTech's approval of any SPRs as conditions that must be met, and to do so according to specified timelines. If the deficiencies are not adequately addressed within the specified timelines, CalTech should take action to suspend the project until the department has either resolved the identified deficiencies or adequately documented its justification for not addressing the deficiencies.

Agency Response*

Written procedures were developed and implemented.

  • Response Type†: Annual Follow Up
  • Completion Date: March 2017
  • Response Date: November 2017

California State Auditor's Assessment of Status: Fully Implemented

The technology department revised SIMM Section 45 - Appendix G - by including a section in the IPO report template focused on the respective state department's actions to address conditions identified in the Technology Department's approval of any SPRs. Further, the technology department revised its Issue Escalation Process to more fully describe how issues or risks, including stagnant remediation of any conditions in the approval of an SPR, will be escalated to state department officials and to management of the technology department. Further, the Issue Escalation Process includes when and what the technology department should require as remedial actions, or the sanctions that it will impose when departments do not comply with these remedial actions.


Agency Response*

ITPOD publishes the most recent project approval conditions and the status of each in the IPOR. If the project fails to make progress on the project approval conditions by the dates established in the approval letter, the IPOD manager notifies the Branch Chief and Deputy Director for discussion with the department and project leadership. This process has been successfully used in 2016, but with the publication of the Escalation and Corrective Action Plan policies referenced above, departments will have greater understanding of the importance of addressing the project approval conditions.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: December 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Not Fully Implemented


Agency Response*

Due to funding constraints and workload, the anticipated Project Planning & Lifecycle Automation system will be delayed until next year. But, ITPOD is now using a manual process to track compliance with project approval conditions. This includes documenting the conditions in the IPOR and reporting on progress each month. If the project fails to make progress on the project approval conditions, the IPO manager will follow the new draft Escalation Process that is expected to be finalized by March 2016. We are also developing a follow-on process for issuing a "Corrective Action" (CAP) requirement on an IT project. This will take the form of both policy and formal ITPOD process. Because the CAP will take the form of published policy, we expect to finalize it by June 2016.

Together the Escalation process and the CAP policy will contribute to the framework necessary to evaluate project problems and ensure appropriate attention and accountability are employed so that the problems can be resolved.

  • Response Type†: 1-Year
  • Estimated Completion Date: June 2016
  • Response Date: February 2016

California State Auditor's Assessment of Status: Pending


Agency Response*

This capability has been included in the requirements for the Department's new IT system project: Project Planning and Lifecycle Automation. Target date for the new system is September 2016. Until then, a manual process is required. The manual process is expected to be completed by September 2015.

  • Response Type†: 6-Month
  • Estimated Completion Date: September 2015
  • Response Date: August 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

60-Day Update:

CalTech ITPOC will develop a standardized process to track the results of a state entities compliance with any/all conditions that CalTech places on approval of a Special Project Report. In the event a state entity fails to respond adequately to the conditions, the IPO will follow Project Oversight Escalation Process and ITPOC management will determine and/or recommend the appropriate remedial action.

The new process is expected to be implemented by July 2015.

  • Response Type†: 60-Day
  • Estimated Completion Date: 0731/2015
  • Response Date: April 2015

California State Auditor's Assessment of Status: Pending


Recommendation #14 To: Technology, California Department of

To ensure that IT projects have the oversight needed to better position them for success, CalTech should develop thresholds relating to IT project cost increases and schedule delays to inform and better justify its decision to allow an IT project to continue. If a department's IT project reaches or exceeds these thresholds, CalTech should require the department to conduct a cost-benefit analysis for the project and include this analysis in an SPR. CalTech should consider the results of this analysis in its decision to approve or deny the SPR and, if warranted, take action to suspend or terminate the project so that it does not allow projects with significant problems to continue without correction.

Agency Response*

A new SPR template has been developed, and is ready for rollout at the end of January 2018.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: January 2018
  • Response Date: November 2017

California State Auditor's Assessment of Status: Pending


Agency Response*

The ITPOD will modify our internal SPR analysis process documents and will add a new section to the SPR Analysis Template to compare the relevant information from the financial worksheets of the original or most recently approved project plan to the proposed new project plan included in the SPR. The financial implications of various courses of action will continue to be one of the factors considered in determining appropriate actions on troubled projects. The revision to the SPR analysis template is expected to be finalized by December 2016.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: December 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Not Fully Implemented


Agency Response*

The ITPOD will modify our internal SPR analysis process documents and will add a new section to the SPR Analysis Template to capture the relevant information concerning cost benefits analysis. The financial implications of various courses of action will continue to be one of the factors considered in determining appropriate actions on troubled projects. The revised SPR analysis template is anticipated to be issued by April 2016.

  • Response Type†: 1-Year
  • Estimated Completion Date: April 2016
  • Response Date: February 2016

California State Auditor's Assessment of Status: Pending


Agency Response*

This recommendation is currently under development and we expect to be able to report progress for the next scheduled update.

  • Response Type†: 6-Month
  • Estimated Completion Date: tbd
  • Response Date: August 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

60-Day Update:

CalTech will add a section to the Special Project Report Template that will require the state entity to provide sufficient information on the cost-benefit or risk avoidance (as applicable) assessment that the state entity used to justify the approval of the SPR.

Progress on this initiative will be reported in our 6 month status update.

  • Response Type†: 60-Day
  • Estimated Completion Date: 12/31/2015
  • Response Date: April 2015

California State Auditor's Assessment of Status: Pending


Recommendation #15 To: Consumer Affairs, Department of

To make certain that Consumer Affairs has a project team that consists of staff trained in managing IT projects, it should ensure that all the BreEZe project team leads obtain all required project management training.

Agency Response*

The Department has completed a comprehensive project management training schedule to ensure that the project leads have the required project management training, in compliance with Statewide Information Management Manual Section 17E (California Project Management Methodology (CA PMM)). See Attachment 1 for complete training schedule.

  • Response Type†: 1-Year
  • Completion Date: January 2016
  • Response Date: February 2016

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

To fully implement this recommendation, the Department has put in place a project management training schedule to ensure that the project leads obtain the required project management training, in compliance with Statewide Information Management Manual Section 17E (California Project Management Methodology (CA PMM)). Two of the four identified team leads have been scheduled for the required CA PMM Basic training, which will be completed by the end of September 2015. The remaining two identified team leads' required CA PMM Basic training should be completed by January 2016. Additionally, the Project Director and Project Manager have Project Management Professional (PMP) certifications. For the new lead staff identified in the Department's SPR 3.1, the Department has included additional funding to ensure the project is prepared to train those staff.

  • Response Type†: 6-Month
  • Estimated Completion Date: January 1, 2016
  • Response Date: August 2015

California State Auditor's Assessment of Status: Partially Implemented


Agency Response*

To fully implement this recommendation, the Department has put in place a project management training schedule to ensure that the project leads obtain the required project management training, in compliance with Statewide Information Management Manual Section 17 (California Project Management Methodology). Additionally, the Project Director and Project Manager have Project Management Professional (PMP) certifications. For the new lead staff identified in the Department's SPR 3.1, the Department has included additional funding to ensure the project is prepared to train those staff. (Attachment 1)

  • Response Type†: 60-Day
  • Estimated Completion Date: TBD
  • Response Date: April 2015

California State Auditor's Assessment of Status: Pending


Recommendation #16 To: Consumer Affairs, Department of

Consumer Affairs should develop a process to ensure that it prepares all required project management documents and undertakes all oversight activities related to BreEZe as CalTech requires so that it can prevent or identify and monitor future problems as they arise. This includes taking steps to sufficiently respond to any concerns that the IPO specialist and the IV&V consultant raise.

Agency Response*

The Department has developed a process to ensure that its IT projects meet all project management documentation requirements during their planning phase. Additionally the Department, in collaboration with Department of Technology (CalTech), has implemented a process to clearly document all activities the project undertakes in response to oversight findings related to BreEZe as identified by CalTech's IPO and IV&V oversight staff. This documentation identifies the steps taken to sufficiently respond to any concerns that the IPO specialist and the IV&V consultant raise and is reviewed in its entirety with CalTech oversight staff each month at a minimum. (Attachments 2 & 3)

  • Response Type†: 60-Day
  • Completion Date: January 2015
  • Response Date: April 2015

California State Auditor's Assessment of Status: Fully Implemented


Recommendation #17 To: Consumer Affairs, Department of

To the extent that Consumer Affairs chooses to implement BreEZe at the phase 3 regulatory entities, it should first complete a formal cost-benefit analysis to ensure that BreEZe is a cost-effective solution to meet these regulatory entities' business needs. To make certain this analysis is complete, it should include an assessment of the potential changes these regulatory entities may require to be made of the BreEZe system and the associated costs. Consumer Affairs should complete the cost-benefit analysis before investing any more resources into the implementation of BreEZe at the phase 3 regulatory entities, and it should update this analysis periodically as significant assumptions change.

Agency Response*

The Department of Consumer Affairs, in conjunction with phase 3 regulatory entities, has developed a Business Modernization Plan (Plan) to effectively facilitate the analysis, approval, and potential transition to program specific licensing and enforcement platforms. The Plan outlines our approach to identifying board/bureau specific business needs, determining cost effective options, and proposes initial schedules. The Plan also considers the impacts and requirements of the Project Approval Lifecycle (PAL), the project approval process required by the California Department of Technology. Each board/bureau will perform a cost benefit analysis, or determine the cost effectiveness of each alternative platform, during Stage 2 of the PAL process. The Financial Analysis Worksheets (FAW) facilitate fiscal planning and comparison, and will be used to determine cost benefit on a board by board basis.

Currently there are four boards/bureaus actively documenting their current business processes to identify needs and requirements. The remaining boards/bureaus are working to inventory their business processes and schedule documentation workshops with departmental Organizational Change Management staff. One program has submitted Stage 1 of the PAL process and a second is scheduled to submit theirs within 90 days.

The thorough planning, business analysis, and program-specific nature of this effort will ensure success for our remaining boards and bureaus (formerly Release/Phase 3).

Implementation Timeframe: Proposed timelines are included and maintained in a board/bureau's individual Business Modernization Reports and may change due to business resource contention.

Individual board/bureau progress through business modernization will vary based on individual program resources, current legislative/business landscape, complexity, fiscal landscape, and the PAL process.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: TBD
  • Response Date: November 2017

California State Auditor's Assessment of Status: Pending


Agency Response*

The Department will perform a formal cost benefit analysis after Release 2 is completed. Part of this formal evaluation will include a gap analysis that will compare existing BreEZe functionality as delivered at the completion of Release 2, to Release 3 boards and bureaus' business needs and current systems' functionality to determine if BreEZe is a cost-effective IT solution for Release 3 boards. The results of this analysis will guide the Department's future efforts to implement BreEZe. In addition, the Department will ensure that the formal cost-benefit analysis is updated periodically as significant assumptions change.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: Unknown
  • Response Date: October 2016

California State Auditor's Assessment of Status: Not Fully Implemented


Agency Response*

The Department will perform a formal cost benefit analysis after Release 2 is completed. Part of this formal evaluation will include a gap analysis that will compare existing BreEZe functionality as delivered at the completion of Release 2, to Release 3 boards and bureaus' business needs and current systems' functionality to determine if BreEZe is a cost-effective IT solution for Release 3 boards. The results of this analysis will guide the Department's future efforts to implement BreEZe. In addition, the Department will ensure that the formal cost-benefit analysis is updated periodically as significant assumptions change.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: TBD
  • Response Date: October 2016

California State Auditor's Assessment of Status: Not Fully Implemented


Agency Response*

The Department will perform a formal cost benefit analysis after Release 2 is completed. Part of this formal evaluation will include a gap analysis that will compare existing BreEZe functionality as delivered at the completion of Release 2, to Release 3 boards and bureaus' business needs and current systems' functionality to determine if BreEZe is a cost-effective IT solution for Release 3 boards. The results of this analysis will guide the Department's future efforts to implement BreEZe. In addition, the Department will ensure that the formal cost-benefit analysis is updated periodically as significant assumptions change.

  • Response Type†: 1-Year
  • Estimated Completion Date: February 18, 2018
  • Response Date: February 2016

California State Auditor's Assessment of Status: Pending


Agency Response*

The Department will perform a formal cost benefit analysis after Release 2 is completed. Part of this formal evaluation will include a gap analysis that will compare all existing BreEZe functionality as delivered at the completion of Release 2, to the Release 3 boards and bureaus' business needs and current systems' functionality to determine if BreEZe would be the most cost-effective IT solution for Release 3 boards. The results of this analysis will guide the Department's future efforts to implement BreEZe. In addition, the Department will ensure that the formal cost-benefit analysis is updated periodically as significant assumptions change.

  • Response Type†: 6-Month
  • Estimated Completion Date: February 18, 2018
  • Response Date: August 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

The Department will perform a formal cost benefit analysis after Release 2 is completed. Part of this formal evaluation will include a gap analysis that will compare all existing BreEZe functionality as delivered at the completion of Release 2, to the Release 3 boards and bureaus' business needs and current systems' functionality to identify necessary BreEZe enhancements. The results of this analysis will guide the Department's future efforts to implement BreEZe. In addition, the Department will ensure that the formal cost-benefit analysis is updated periodically as significant assumptions change.

  • Response Type†: 60-Day
  • Estimated Completion Date: TBD
  • Response Date: April 2015

California State Auditor's Assessment of Status: Pending


Recommendation #18 To: Consumer Affairs, Department of

If Consumer Affairs determines that a new IT project is warranted in the future, it should develop a process to ensure the success of that project that includes, at a minimum, system requirements that are specific to each regulatory entity as applicable.

Agency Response*

The Department, working with the California Department of Technology, will continue to follow all statewide IT policy applicable to the Systems Development Life Cycle. For all future IT projects, the Department has developed a formal policy so that any future IT project will capture and utilize the business/system requirements specific to each regulatory entity in the selection or development of an application. See Attachment 3 for aforementioned policy.

  • Response Type†: 1-Year
  • Completion Date: January 2016
  • Response Date: February 2016

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

The Department, working with the California Department of Technology, will continue to utilize/follow the Systems Development Life Cycle model/process (the Systems Development Life Cycle model describes the process for planning, creating, testing, and deploying an information system).

For all future IT projects, the Department will capture and utilize the business/system requirements specific to each regulatory entity in the selection or development of an application.

  • Response Type†: 6-Month
  • Completion Date: July 2015
  • Response Date: August 2015

California State Auditor's Assessment of Status: Pending

Until Consumer Affairs develops a formal process to ensure that, at a minimum, it includes system requirements that are specific to each regulatory entity as part of a new IT project in the future, we will not consider this recommendation fully implemented. According to Consumer Affairs, it plans to provide us with a status update on its implementation of this recommendation in its one-year response.

  • Auditee did not substantiate its claim of full implementation

Agency Response*

The Department, working with the California Technology Agency, will continue to utilize/follow the Systems Development Life Cycle process for all future IT projects. The Department will capture and utilize the business/system requirements specific to each regulatory entity in the selection or development of an application.

  • Response Type†: 60-Day
  • Estimated Completion Date: TBD
  • Response Date: April 2015

California State Auditor's Assessment of Status: No Action Taken

We followed-up with Consumer Affairs on the status of this recommendation as reported in its 60-day response, and learned that it intends on developing a plan that will include activities that address this recommendation after it implements BreEZe at the phase 2 regulatory entities.


Recommendation #19 To: Consumer Affairs, Department of

If Consumer Affairs determines that a new IT project is warranted in the future, it should develop a process to ensure the success of that project that includes, at a minimum, a project team that is qualified and experienced.

Agency Response*

The California Department of Technology has implemented a revised Project Approval Lifecycle (SIMM 19), which includes a four stage project review and approval process. Stage 4 of this approval process requires Department of Technology review and approval of project resources to ensure that they are available and properly qualified, experienced, and trained (SAM 4928 - Information Technology). As the Department's IT projects fall under the Department of Technology's purview, the revised Project Approval Lifecycle will ensure that any future IT project at the Department includes a project team that is qualified and experienced. With the release of Technology Letter 16-07 in August 2016, all four Stages of the Project Approval Lifecycle are complete and in place.

  • Response Type†: Annual Follow Up
  • Completion Date: August 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

The California Department of Technology has implemented a revised Project Approval Lifecycle (SIMM 19), which includes a four stage project review and approval process. Stage 4 of this approval process requires Department of Technology review and approval of project resources to ensure that they are available and properly qualified, experienced, and trained (SAM 4928 - Information Technology). As the Department's IT projects fall under the Department of Technology's purview, the revised Project Approval Lifecycle will ensure that any future IT project at the Department includes a project team that is qualified and experienced.

  • Response Type†: 1-Year
  • Completion Date: October 2015
  • Response Date: February 2016

California State Auditor's Assessment of Status: Pending

According to the Department of Technology's one-year response to recommendations eight through ten regarding its four stage project review and approval process, Stage 4 of that approval process is scheduled for final release in July 2016. Thus, until that stage is finalized and includes steps to ensure that any future IT project includes a project team that is qualified and experienced, we will continue to report the status of this recommendation as pending.

  • Auditee did not substantiate its claim of full implementation

Agency Response*

The Department has implemented additional procedures for IT projects based on lessons learned. For any new IT project, a project staffing plan will be required prior to embarking on the project. At a minimum, the components of the project staffing plan will include the number of staff required, how staff will be acquired, length of time the staff will be needed, qualifications, skills, and training needs.

  • Response Type†: 6-Month
  • Completion Date: July 2015
  • Response Date: August 2015

California State Auditor's Assessment of Status: Pending

Until Consumer Affairs develops a formal process to ensure that, at a minimum, its project team that oversees any new IT project is qualified and experienced, we will not consider this recommendation fully implemented. According to Consumer Affairs, it plans to provide us with a status update on its implementation of this recommendation in its one-year response

  • Auditee did not substantiate its claim of full implementation

Agency Response*

The Department will develop a project staffing plan prior to embarking on any new IT project. Items that will be included in the project staffing plan are the number of staff required, how staff will be acquired, length of time the staff will be needed, qualifications, skills, and training needs.

  • Response Type†: 60-Day
  • Estimated Completion Date: TBD
  • Response Date: April 2015

California State Auditor's Assessment of Status: No Action Taken

We followed-up with Consumer Affairs on the status of this recommendation as reported in its 60-day response, and learned that it intends on developing a plan that will include activities that address this recommendation after it implements BreEZe at the phase 2 regulatory entities.


Recommendation #20 To: Consumer Affairs, Department of

If Consumer Affairs determines that a new IT system is warranted in the future, it should develop a process to ensure the success of that project that includes, at a minimum, development of and adherence to all project management plans.

Agency Response*

The Department has developed a process to ensure that its IT projects meet all project management documentation requirements during their planning phase. (Attachment 2)

  • Response Type†: 60-Day
  • Completion Date: March 2015
  • Response Date: May 2015

California State Auditor's Assessment of Status: Fully Implemented


Recommendation #21 To: Consumer Affairs, Department of

If Consumer Affairs determines that a new IT project is warranted in the future, it should develop a process to ensure the success of that project that includes, at a minimum, planning and implementation of effective organizational change management.

Agency Response*

The Department has deployed a skilled Organizational Change Management (OCM) team to assist the Release 2 boards and bureaus with their transition to BreEZe. This OCM team is charged with working with the Release 2 boards and bureaus to develop and assist in the implementation of new or revised business processes specific to each entity's business. At the conclusion of BreEZe Release 2 OCM activities the OCM documentation will be available. The Department has developed a process to ensure that its IT projects meet all project management documentation requirements during their planning phase (Attachment 2). This Department process requires any future IT project to include appropriate organizational change management activities as a project critical success factor.

  • Response Type†: 1-Year
  • Completion Date: March 2015
  • Response Date: February 2016

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

The Department has deployed a skilled Organizational Change Management (OCM) team to assist the Release 2 boards and bureaus with their transition to BreEZe. This OCM team is charged with working with the Release 2 boards and bureaus to develop and assist in the implementation of new or revised business processes specific to each entity's business. At the conclusion of BreEZe Release 2 OCM activities the OCM documentation will be available. Prior to any new IT project in the future, the Department will initiate this new organizational change management process making any refinements based on additional lessons learned and on the needs of the new project.

  • Response Type†: 6-Month
  • Completion Date: July 2015
  • Response Date: August 2015

California State Auditor's Assessment of Status: Pending

Until Consumer Affairs develops a formal process to ensure that, at a minimum, it includes planning and implementation of effective organizational change management as part of a new IT project in the future, we will not consider this recommendation fully implemented. According to Consumer Affairs, it plans to provide us with a status update on its implementation of this recommendation in its one-year response.

  • Auditee did not substantiate its claim of full implementation

Agency Response*

The Department has deployed a skilled Organizational Change Management (OCM) team to assist the Release 2 boards and bureaus with their transition to BreEZe. This OCM team is charged with working with the Release 2 boards and bureaus to develop and execute training that is specific to each entity's business processes. This process will be used for all future IT projects.

  • Response Type†: 60-Day
  • Completion Date: November 2014
  • Response Date: April 2015

California State Auditor's Assessment of Status: Partially Implemented

We followed-up with Consumer Affairs on the status of this recommendation as reported in its 60-day response, and learned that it intends on developing a plan that will include activities that address this recommendation after it implements BreEZe at the phase 2 regulatory entities. Thus, although Consumer Affairs has taken action to assist the phase 2 regulatory entities as it relates to organizational change management, not until it develops and finalizes a plan for future IT projects will we be able to fully assess its progress in implementing this recommendation.

  • Auditee did not substantiate its claim of full implementation

Recommendation #22 To: Consumer Affairs, Department of

If Consumer Affairs determines that a new IT project is warranted in the future, it should develop a process to ensure the success of that project that includes, at a minimum, timely response to concerns that the IV&V consultant and the IPO specialist raise.

Agency Response*

The Department, in collaboration with Department of Technology, implemented a process to document and report activities the project undertakes in response to Project Oversight findings related to BreEZe. These reports identify the steps taken to sufficiently respond to concerns that the IPO specialist and the IV&V consultant raise and are reviewed with Project Oversight staff no less than once per month. The Department has developed a standard response template (Attachment 1) that will ensure future IT projects have the tool to facilitate clear communication with oversight staff and provides timely documented response to IV&V and/or IPO identified concerns.

  • Response Type†: Annual Follow Up
  • Completion Date: July 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

The Department, in collaboration with Department of Technology, implemented a process to document and report activities the project undertakes in response to Project Oversight findings related to BreEZe. These reports identify the steps taken to sufficiently respond to concerns that the IPO specialist and the IV&V consultant raise and are reviewed with Project Oversight staff no less than once per month. The Department is developing a standard process and response template that will ensure future IT projects provide timely documented response to IV&V and/or IPO identified concerns.

  • Response Type†: 1-Year
  • Estimated Completion Date: July 1, 2016
  • Response Date: February 2016

California State Auditor's Assessment of Status: Pending


Agency Response*

The Department, in collaboration with Department of Technology (CalTech), implemented a process to document and report activities the project undertakes in response to oversight findings related to BreEZe as identified by CalTech's IPO and IV&V oversight staff. These reports identify the steps taken to sufficiently respond to concerns that the IPO specialist and the IV&V consultant raise and are reviewed with CalTech oversight staff at least once a month. These reports are available and can be requested from CalTech's IPO and IV&V staff. Prior to any new IT project in the future, the Department will follow this or a similar model to ensure timely responses to oversight staff concerns.

  • Response Type†: 6-Month
  • Completion Date: July 2015
  • Response Date: August 2015

California State Auditor's Assessment of Status: Pending

Although Consumer Affairs indicates that it collaborated with CalTech and implemented a process to document and report activities the project undertakes in response to concerns raised by the IV&V consultant and IPO specialist, this process is specific to BreEZe. As a result, not until Consumer Affairs develops a standard process that staff are required to use on any new IT projects it may undertake in the future will we consider this recommendation fully implemented. According to Consumer Affairs, it plans to provide us with a status update on its implementation of this recommendation in its one-year response.

  • Auditee did not substantiate its claim of full implementation

Agency Response*

The Department, in collaboration with Department of Technology (CalTech), has implemented a process to clearly document all activities the project undertakes in response to oversight findings related to BreEZe as identified by CalTech's IPO and IV&V oversight staff. This documentation identifies the steps taken to sufficiently respond to any concerns that the IPO specialist and the IV&V consultant raise and is reviewed in its entirety with CalTech oversight staff each month at a minimum. This process will be used for all future IT projects.

  • Response Type†: 60-Day
  • Completion Date: January 2015
  • Response Date: April 2015

California State Auditor's Assessment of Status: Partially Implemented

We followed-up with Consumer Affairs on the status of this recommendation as reported in its 60-day response, and learned that it intends on developing a plan that will include activities that address this recommendation after it implements BreEZe at the phase 2 regulatory entities. Thus, although it reported that it has developed a process for responding to oversight findings related to BreEZe, not until it develops a plan for future IT projects that adresss this recommendation will we be able to fully assess its progress in implementing this recommendation.

  • Auditee did not substantiate its claim of full implementation

Recommendation #23 To: Consumer Affairs, Department of

If Consumer Affairs determines that a new IT project is warranted in the future, it should develop a process to ensure the success of that project that includes, at a minimum, sufficient staffing.

Agency Response*

The California Department of Technology has implemented a revised Project Approval Lifecycle (SIMM 19), which includes a four stage project review and approval process. Stage 4 of this approval process requires Department of Technology review and approval of project resources to ensure that they are available, sufficient, and properly qualified, experienced, and trained (SAM 4928 - Information Technology). As the Department's IT projects fall under the Department of Technology's purview, the revised Project Approval Lifecycle (fully implemented as of August 29, 2016 per Technology Letter 16-07) will ensure that any future IT project at the Department includes a project team of sufficient size.

  • Response Type†: Annual Follow Up
  • Completion Date: August 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

The California Department of Technology has implemented a revised Project Approval Lifecycle (SIMM 19), which includes a four stage project review and approval process. Stage 4 of this approval process requires Department of Technology review and approval of project resources to ensure that they are available, sufficient, and properly qualified, experienced, and trained (SAM 4928 - Information Technology). As the Department's IT projects fall under the Department of Technology's purview, the revised Project Approval Lifecycle will ensure that any future IT project at the Department includes a project team of sufficient size.

Additionally, the Department has developed a process to ensure that its IT projects meet all project management documentation requirements during their planning phase (Attachment 2). This Department process requires any future IT project to include a Human Resources (Staffing) Plan as a project critical success factor.

  • Response Type†: 1-Year
  • Completion Date: October 2015
  • Response Date: February 2016

California State Auditor's Assessment of Status: Partially Implemented

According to the Department of Technology's one-year response to recommendations eight through ten regarding its four stage project review and approval process, Stage 4 of that approval process is scheduled for final release in July 2016. Thus, until that stage is finalized and includes steps to ensure that any future IT project includes a project team that is qualified and experienced, we will continue to report this recommendation as not fully implemented. However, because Consumer Affairs has developed a project management plan development procedure, which includes a Human Resources Management plan, we will report this recommendation as partially implemented.


Agency Response*

As stated in the response to Recommendation #19, a project staffing plan will be required prior to embarking on any new IT project. Further to ensure sufficient staffing, the plan will include obtaining commitments from key staff (boards and bureaus) before actual work begins.

  • Response Type†: 6-Month
  • Completion Date: July 2015
  • Response Date: August 2015

California State Auditor's Assessment of Status: Pending

Until Consumer Affairs develops a formal process to ensure that, at a minimum, it has sufficient staffing as part of a new IT project in the future, we will not consider this recommendation fully implemented. According to Consumer Affairs, it plans to provide us with a status update on its implementation of this recommendation in its one-year response.

  • Auditee did not substantiate its claim of full implementation

Agency Response*

The Department will develop a project staffing plan prior to embarking on any new IT project. Items that will be included in the project staffing plan are the number of staff required, how staff will be acquired, length of time the staff will be needed, qualifications, skills, and training needs. Additionally, the plan will include obtaining commitments from key staff (boards and bureaus) before actual work begins.

  • Response Type†: 60-Day
  • Estimated Completion Date: TBD
  • Response Date: April 2015

California State Auditor's Assessment of Status: No Action Taken

We followed-up with Consumer Affairs on the status of this recommendation as reported in its 60-day response, and learned that it intends on developing a plan that will include activities that address this recommendation after it implements BreEZe at the phase 2 regulatory entities.


Recommendation #24 To: Technology, California Department of

To ensure that future IT project procurements do not jeopardize the State's financial interests, CalTech's procurement division should document its reasons for approving any deviations from standard contract language.

Agency Response*

In August 2015, the Department's Procurement division piloted its draft desk procedures for approving any deviations from standard contract language. Desk procedures finalization and full implementation for all IT reportable projects were completed May 2016.

  • Response Type†: Annual Follow Up
  • Completion Date: May 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

In August 2015, the Department's Procurement division piloted its draft desk procedures for approving any deviations from standard contract language. Desk procedures finalization and full implementation for all IT reportable projects is expected by March 2016

  • Response Type†: 1-Year
  • Estimated Completion Date: March 2016
  • Response Date: February 2016

California State Auditor's Assessment of Status: Pending


Agency Response*

The Department's Procurement division currently documents the reasons for any deviations from standard contract language (GSPD 401IT) utilizing formal approval threshold criteria. The Procurement division has developed draft desk procedures for this area, which are currently under internal review and expect to be finalized by September 2015.

  • Response Type†: 6-Month
  • Completion Date: August 2015
  • Response Date: August 2015

California State Auditor's Assessment of Status: Pending

Until CalTech finalizes its draft desk procedures for this area, we will continue to report the status of this recommendation as pending.


Agency Response*

CalTech is currently developing the below new criteria. The following is the GSPD 401IT Paragraph Modification Threshold Criteria that STPD has proposed:

To modify a "carve out" paragraph of the GSPD (i.e., "unless otherwise specified in the Statement of Work":

- The Department must show, and CalTech must agree, that there is added clarity to the contract or value added to an important State position in the agreement as a result of changing the contract term.

To modify a "non-carve out" paragraph of the GSPD:

- The Department must show, and CalTech must agree, that a critical Department objective cannot be met without changing the contract term.

  • Response Type†: 60-Day
  • Estimated Completion Date: 12/31/2015
  • Response Date: May 2015

California State Auditor's Assessment of Status: Pending


Recommendation #25 To: Technology, California Department of

To ensure that future IT project procurements do not jeopardize the State's financial interests, CalTech's procurement division should ensure that contract language gives departments the rights to the source code necessary to complete the project if a contract is terminated for convenience.

Agency Response*

Since April 2014, CalTech Procurement ensures the contract language gives the rights to the source code necessary to complete the project if a contract is terminated for convenience utilizing standard contract language (GSPD 401IT).

This action has been completed

  • Response Type†: 60-Day
  • Completion Date: April 2014
  • Response Date: April 2015

California State Auditor's Assessment of Status: Fully Implemented


Recommendation #26 To: Technology, California Department of

To ensure that future IT project procurements do not jeopardize the State's financial interests, CalTech's procurement division should ensure that contract language contains assurances that state funds will not be used for unauthorized purposes.

Agency Response*

Since April 2014, CalTech Procurement ensures contract language contains assurances that state funds will not be used for unauthorized purposes utilizing standard contract language (GSPD 401IT).

This action has been completed

  • Response Type†: 60-Day
  • Completion Date: April 2015
  • Response Date: April 2015

California State Auditor's Assessment of Status: Fully Implemented


Recommendation #27 To: Technology, California Department of

To ensure that future IT project procurements do not jeopardize the State's financial interests, CalTech's procurement division should ensure that contracting departments can appropriately manage IT maintenance support contracts by including service-level agreements in these contracts.

Agency Response*

Since April 2014, CalTech Procurement ensures that contracting departments can appropriately manage IT maintenance support contracts by including service level agreements in these contracts utilizing standard contract language (GSPD 401IT) or special contract provisions.

This audit recommended action has been completed.

  • Response Type†: 60-Day
  • Completion Date: April 2014
  • Response Date: April 2015

California State Auditor's Assessment of Status: Fully Implemented


Recommendation #28 To: Technology, California Department of

To ensure that contracting departments comply with requirements, CalTech should periodically monitor the status of contracts and enforce adherence to procurement requirements. For example, CalTech should ensure that contracting departments forward all change requests to it and that it conducts reviews of all change requests to determine whether the changes exceed the scope of the respective contract.

Agency Response*

CDT has updated the SPR preparation instructions to reflect potential alternatives if cost-benefit analysis is incomplete.

  • Response Type†: Annual Follow Up
  • Completion Date: March 2017
  • Response Date: November 2017

California State Auditor's Assessment of Status: Pending

In following up with CalTech regarding its implementation of this recommendation, we learned that it has not yet developed procedures to ensure that contracting departments forward all change requests to it and that it conducts reviews of all change requests to determine whether the changes exceed the scope of the respective contract. Until it develops such procedures, we will report this recommendation as pending.

  • Auditee did not substantiate its claim of full implementation

Agency Response*

ITPOD will examine change requests in the normal course of the oversight activity and if any are identified as exceeding the scope and intent of the current contract, ITPOD will raise that issue with the department and will bring the issue to the attention of the State Technology and Procurement Division (STPD), as needed. STPD will conduct a more detailed review to make the final determination whether or not the changes exceed the scope of the respective contract. Procedural documentation of these new oversight activities are expected to be completed by December 2016.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: December 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Not Fully Implemented


Agency Response*

It is important to recognize that change requests are often necessary and proper. ITPOD uses SIMM 45, Appendix F: Project Oversight Review Checklist to evaluate the conditions which necessitate changes to the contract as well as the processes for ensuring adequate decision making at the appropriate level for issuing a change request. This includes verifying that key specification documents (e.g. contracts, requirement specifications and/or contract deliverables) and software products are under formal configuration control, with items to be controlled and specific staff roles and responsibilities for configuration management identified in a configuration management plan. This also includes verification that a formal governance process is in place, which necessitates the documentation of the reasons for approval of change requests.

ITPOD will examine change requests in the normal course of the oversight activity and if any are identified as exceeding the scope and intent of the current contract, ITPOD will raise that issue with the department and will bring the issue to the attention of the State Technology and Procurement Division (STPD) as needed. STPD will conduct a more detailed review to make the final determination whether or not the changes exceed the scope of the respective contract.

  • Response Type†: 1-Year
  • Completion Date: January 2016
  • Response Date: February 2016

California State Auditor's Assessment of Status: Pending

In following up with CalTech regarding its implementation of this recommendation, we learned that it has not yet developed procedures to ensure that contracting departments forward all change requests to it and that it conducts reviews of all change requests to determine whether the changes exceed the scope of the respective contract. Until it develops such procedures, we will report this recommendation as pending.


Agency Response*

The Department's Procurement division will continue to work with ITPOD to identify if a Change Request has deviated outside a procurement requirement. The division will then be notified to conduct a more detailed review to determine if the changes exceed the scope of the respective contract.

We expect to be able to report progress for the next scheduled update.

  • Response Type†: 6-Month
  • Estimated Completion Date: TBD
  • Response Date: August 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

To address the audit recommendation that CalTech review all contract requests, STPD has formally requested staff augmentation. We will be able to provide an update on the progress of this additional resource request in our 6-month status update report.

  • Response Type†: 60-Day
  • Estimated Completion Date: 1/31/2016
  • Response Date: May 2015

California State Auditor's Assessment of Status: Pending


Recommendation #29 To: Consumer Affairs, Department of

To ensure that it complies with all terms of the BreEZe contract, Consumer Affairs should develop and follow a process for doing so, including documenting how it has met applicable contract terms, such as forwarding all change requests to CalTech.

Agency Response*

The Department has developed a process to ensure that it complies with all terms of the BreEZe contract. This process and documentation format has been added to the project's Contractual Obligations Management Plan (Attachment 4).

  • Response Type†: 1-Year
  • Completion Date: June 2015
  • Response Date: February 2016

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

The Department has a robust process in place to document how each system requirement within the contract has been met which is traceable from initial contract requirement to the implemented system. Following the pending execution of amendment 3 to the BreEZe implementation contract, the contract manager will use a similar robust process for all contract administrative requirements. This process shall document how all terms of the BreEZe contract have been satisfied, including contract terms such as forwarding all change requests to California Department of Technology.

  • Response Type†: 6-Month
  • Estimated Completion Date: August 28, 2015
  • Response Date: August 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

The Department has a robust process in place to document how each system requirement within the contract has been met which is traceable from initial contract requirement to the implemented system. Following the pending execution of amendment 3 to the BreEZe implementation contract, the contract manager will develop a similar robust process for all contract administrative requirements. This process shall document how all terms of the BreEZe contract have been satisfied, including contract terms such as forwarding all change requests to CalTech.

  • Response Type†: 60-Day
  • Estimated Completion Date: TBD
  • Response Date: April 2015

California State Auditor's Assessment of Status: Pending


Recommendation #30 To: Registered Nursing, Board of

To ensure that BRN has adequate data to effectively use its resources and manage its workload, it should formally track and monitor the timeliness of its processing of applications by type and the cause of any delays.

Agency Response*

As a result of the Auditor's January 2017 response, the BRN completed development of more than 15 reports to formally track and monitor applications by type and deficiency. Some reports are reviewed daily, weekly, and monthly by managers as noted in the Succession Plan: Licensing Management Manual (SPLMM). Instructions and guidance for usage of these reports by management are formally outlined in the SPLMM on pages 10-14. Management's review of the reports is to ensure applications are processed in accordance with regulatory timeframes, identify delays in application processing, and remedy any causes of such delay.

  • Response Type†: Annual Follow Up
  • Completion Date: September 2017
  • Response Date: November 2017

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

Beginning in May 2016 the BRN has been able to formally track and monitor applications by type by utilizing the new Quality Business Interactive Reporting Tool (QBIRT). The BRN has been tracking application deficiency periods and now can track the specific cause for the delay because the BRN further refined the QBIRT report to track deficiencies within BreEZe to identify specific causes of application delays. The BRN reviews this report on a bi-weekly basis to ensure applications are evaluated on a timely basis and to remedy any causes for delay within BRNs purview.

  • Response Type†: Annual Follow Up
  • Completion Date: December 2016
  • Response Date: January 2017

California State Auditor's Assessment of Status: Partially Implemented

Although BRN was able to provide reports demonstrating that it now has the capability to track and monitor the timeliness of its processing of applications by type and the cause of any delays, it has not yet formalized its process for reviewing these reports. Thus, until BRN sets forth this review process in its policies and procedures, we will continue to report this recommendation as partially implemented.

  • Auditee did not substantiate its claim of full implementation

Agency Response*

BRN staff worked with the Department of Consumer Affairs (DCA) and initiated a system enhancement to track application processing at specified points within the application life cycle and the data has been used to track the timeliness of application processing by type. This enhancement to track the postmark date, initial review date and deficiency period was implemented in BreEZe for the BRN starting July 1, 2015.

Additionally, starting in May 2016 the BRN has been able to formally track and monitor applications by type by utilizing the new Quality Business Interactive Reporting Tool (QBIRT). The BRN can track application deficiency periods, but cannot currently track the specific cause for the delay. The BRN is working with DCA BreEZe staff to update and create new specific deficiency tracking points within BreEZe to allow for more detailed application deficiency tracking which will allow the BRN the ability to identify specific causes of application delays.

The BRN anticipates implementation of these changes by July 1, 2017 and once completed this recommendation will be considered fully implemented.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: July 1, 2017
  • Response Date: October 2016

California State Auditor's Assessment of Status: Not Fully Implemented


Agency Response*

BRN staff worked with the Department of Consumer Affairs (DCA) and initiated a system enhancement request to track application processing at specified points within the application life cycle and the data will be used to track the timeliness of application processing by type.

The enhancement to track the postmark date, initial review date and deficiency period was implemented in BreEZe for use starting July 1, 2015.

Reports were released into production on January 19, 2016, when release 2 was deployed in BreEZe. BRN staff is verifying data to determine the accuracy of information provided in the reports.

The BRN continues to use data extracts that were created to help ensure applications are processed in a timely manner.

  • Response Type†: 1-Year
  • Estimated Completion Date: 7/1/2016
  • Response Date: February 2016

California State Auditor's Assessment of Status: Partially Implemented


Agency Response*

BRN staff worked with the Department of Consumer Affairs (DCA) and initiated a system enhancement request to track application processing at specified points within the application life cycle and the data will be used to track the timeliness of application processing by type.

The enhancement to track the postmark date, initial review date and deficiency period was implemented in BreEZe for use starting July 1, 2015.

The report is still under development for use by boards and bureaus utilizing BreEZe and is expected to be available by July 1, 2016.

In the interim, staff members are working with DCA to design and develop adhoc queries to ensure applications are processed in a timely manner.

  • Response Type†: 6-Month
  • Estimated Completion Date: 7/1/2016
  • Response Date: August 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

BRN staff is working with Department of Consumer Affairs (DCA) and has initiated a system enhancement request to track application processing at specified points within the application life cycle and the data will be used to track the timeliness of application processing by type.

The enhancement request was presented at the March 24, 2015 Change Control Board (CCB) for review and approval. The CCB approved the system enhancement and expects to have the enhancement in BreEZe for use by July 1, 2015.

In addition, the BRN is requesting an additional system enhancement to track the initial review date for each application type as this was not included in the DCA's original enhancement request. It is hoped this additional system enhancement will be included in the release by July 1, 2015.

In the interim, staff members are working with the DCA to design and develop adhoc queries to ensure applications are processed in a timely manner.

  • Response Type†: 60-Day
  • Estimated Completion Date: 07/01/2015
  • Response Date: April 2015

California State Auditor's Assessment of Status: Pending


Recommendation #31 To: Registered Nursing, Board of

To ensure that BRN has adequate data to effectively use its resources and manage its workload, it should formally track and monitor its pending workload of applications by type and original receipt date.

Agency Response*

BRN staff worked with the Department of Consumer Affairs (DCA) and initiated a system enhancement request to capture the deficiency start and end date which provides us with statistical information for incomplete applications. The enhancement to track the deficiency period was implemented for use in BreEZe on July 1, 2015.

Beginning April 1, 2015, the BRN began utilizing the "postmark date field" in BreEZe to track the original receipt date of all initial licensing application types. BRN staff members worked with DCA to design and develop data extracts to help ensure applications are processed in a timely manner. Additionally, starting in May 2016, the BRN has been able to create application specific reports and formally track and monitor applications by utilizing the Quality Business Interactive Reporting Tool(QBIRT).

The BRN continues to maintain the initial licensing application workload sorted by month and week to determine age of pending applications in process. This allows us to monitor the number of pending applications in concise groupings to help ensure applications are processed within the regulatory time frame.

  • Response Type†: Annual Follow Up
  • Completion Date: May 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

BRN staff worked with the Department of Consumer Affairs (DCA) and initiated a system enhancement request to capture the deficiency start and end date which will provide us with statistical information for incomplete applications.

The enhancement to track the deficiency period was implemented for use in BreEZe on July 1, 2015.

Beginning April 1, 2015, the BRN began utilizing the "postmark date field" in BreEZe to track the original receipt date of all initial licensing application types. BRN staff members worked with DCA to design and develop data extracts to help ensure applications are processed in a timely manner.

Reports were released into production on January 19, 2016, when release 2 was deployed in BreEZe. BRN staff is verifying data to determine the accuracy of information provided in the reports.

The BRN continues to maintain the initial licensing application workload sorted by month and week to determine age of pending applications in process. This allows us to monitor the number of pending applications in concise groupings to help ensure applications are processed within the regulatory time frame.

  • Response Type†: 1-Year
  • Estimated Completion Date: 7/1/2016
  • Response Date: February 2016

California State Auditor's Assessment of Status: Pending

Although BRN indicated it has released reports into production that address this recommendation, based on additional follow-up with BRN regarding its response, the assistant executive director explained that BRN is not comfortable with the data in the report regarding its pending workload and is working with Consumer Affairs to resolve its concerns. Given the uncertainty surrounding the accuracy of these reports, we will continue to report this recommendation as pending until these concerns are alleviated.


Agency Response*

BRN staff worked with the Department of Consumer Affairs (DCA) and initiated a system enhancement request to capture the deficiency start and end date which will provide us with statistical information for incomplete applications.

The enhancement to track the deficiency period was implemented for use in BreEZe on July 1, 2015.

Beginning April 1, 2015, the BRN began utilizing the "postmark date field" in BreEZe to track the original receipt date of all initial licensing application types. BRN staff members are working with DCA to design and develop adhoc queries to ensure applications are processed in a timely manner.

The addition of temporary staff, including an additional supervisor, has allowed the licensing supervisor to more closely monitor the age of each pending application.

Formal reports are being developed to capture this information by participating boards/bureaus and DCA to be available by July 1, 2016. Until the reports are available the BRN maintains the initial licensing application workload sorted by month and week to determine age of pending applications in process. This allows us to monitor the number of pending applications in concise groupings to ensure applications are processed within the regulatory time frame.

  • Response Type†: 6-Month
  • Estimated Completion Date: 7/1/2016
  • Response Date: August 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

BRN staff is working with the Department of Consumer Affairs (DCA) and has initiated a system enhancement request to capture the deficiency start and end date which will provide us with statistical information for incomplete applications.

The enhancement request was presented at the March 24, 2015 Change Control Board (CCB) for review and approval. The CCB approved the system enhancement and expects to have the enhancement in BreEZe for use by July 1, 2015.

Beginning April 1, 2015, the BRN is utilizing the "postmark date field" in BreEZe to track the original receipt date of all initial licensing application types. BRN staff members are working with the DCA to design and develop adhoc queries to ensure applications are processed in a timely manner.

The addition of temporary staff, including an additional supervisor, has allowed the licensing supervisor to more closely monitor the age of each pending application.

Until the system enhancements are complete the BRN maintains the initial licensing application workload sorted by month and week to determine age of pending applications in process. This allows us to monitor the number of pending applications in concise groupings to ensure applications are processed within the regulatory time frame.

  • Response Type†: 60-Day
  • Estimated Completion Date: 07/01/2015
  • Response Date: April 2015

California State Auditor's Assessment of Status: Pending


Recommendation #32 To: Registered Nursing, Board of

To ensure that BRN has adequate data to effectively use its resources and manage its workload, it should conduct an analysis no later than June 30, 2015, of its application processing since implementing BreEZe in order to identify the workload capability of each of its units, such as the licensing support unit; to the extent it determines additional resources are necessary, BRN should submit a request for these resources that is appropriately justified.

Agency Response*

The BRN has analyzed the information from the SOLID report to assess efficiency and workload capability which was included with our data collection to request additional resources in a BCP for FY 17/18 which was included in the Governor's budget that was released on January 10, 2017.

  • Response Type†: Annual Follow Up
  • Completion Date: January 2017
  • Response Date: January 2017

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

The Department of Consumer Affairs' SOLID developed and delivered a final report that detailed and documents the licensing business processes, average processing times, and recommendations prior to June 30, 2015.

The BRN is analyzing the information from the SOLID report to assess efficiency and workload capability to include in our existing request for additional resources.

  • Response Type†: 1-Year
  • Completion Date: July 2015
  • Response Date: February 2016

California State Auditor's Assessment of Status: Partially Implemented

Until BRN has finalized its analysis of the information in the SOLID report and determined whether additional resources are necessary to manage its workload, we will continue to report this recommendation as partially implemented.

  • Auditee did not substantiate its claim of full implementation

Agency Response*

The Department of Consumer Affairs' SOLID developed and delivered a final report that detailed and documents the licensing business processes, average processing times, and recommendations prior to June 30, 2015.

The BRN is analyzing the information from the SOLID report to assess efficiency and workload capability to include in our existing request for additional resources.

  • Response Type†: 6-Month
  • Completion Date: July 2015
  • Response Date: August 2015

California State Auditor's Assessment of Status: Partially Implemented

  • Auditee did not substantiate its claim of full implementation

Agency Response*

The Board of Registered Nursing (BRN) is currently doing the following to meet the recommendation:

- The BRN met with the Department of Consumer Affairs (DCA) SOLID Training Solutions to outline the project on February 12, 2015. BRN subject matter experts met with SOLID and designed licensing process maps for each license type in March 2015.

- SOLID developed and delivered business process flowcharts and summary procedures for each licensing type to BRN for review. SOLID and BRN staff will meet on April 8, 2015 to discuss and finalize the flowcharts.

The BRN will do the following to meet the recommendation:

- SOLID will deliver a template that details instructions and the process to develop average time frames to process applications for each license type.

- BRN staff will conduct time trials to process all application types to develop average processing time frames.

- SOLID will develop and deliver a final report that details and documents the licensing business processes, average processing times, and any recommendations no later than June 30, 2015.

- The BRN will analyze the information from the SOLID report to assess efficiency and workload capability to include in a request for additional resources.

  • Response Type†: 60-Day
  • Estimated Completion Date: 06/30/2015
  • Response Date: April 2015

California State Auditor's Assessment of Status: Pending


Recommendation #33 To: Registered Nursing, Board of

To ensure that BRN continues to process applications within regulatory time frames, it should continue its efforts to refine its business processes to increase efficiency and reduce the amount of time applications are pending its review.

Agency Response*

The BRN has implemented changes related to SOLID and staff recommendations and on an ongoing basis has taken additional steps to improve and refine our business processes. The BRN has refined the application review process for California schools eliminating the need to review an application multiple times to determine eligibility to take the exam and obtain a license. An additional business process change in October 2016 to accept transcripts electronically via the DCA Cloud system from California approved RN schools has contributed to the improved processing times. As a result, the BRN has conducted the majority of initial application evaluations within the regulatory timeframe since July 2016.

  • Response Type†: Annual Follow Up
  • Completion Date: October 2016
  • Response Date: January 2017

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

The BRN has worked with the Department of Consumer Affair's (DCA) SOLID Training Solutions to document all of its business processes and reviewed the recommendations. The BRN has implemented changes related to the recommendations and on an ongoing basis has taken additional steps to improve and refine our business processes. The BRN will continue to analyze and accept staff feedback regarding its business processes to develop and implement new and revised procedures for the initial licensure application process. The BRN continues to assess efficiency and workload capability and has identified the need for additional resources. Requests for additional staff have been submitted.

This recommendation is considered implemented, but is also considered ongoing as business processes evolve due to ongoing enhancements to the BreEZe system.

  • Response Type†: Annual Follow Up
  • Completion Date: September 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Partially Implemented

Based on information provided by BRN, it continues to face significant delays in processing license applications. As a result, until it processes applications within regulatory time frames, we will continue to report this recommendation as partially implemented.


Agency Response*

The BRN has finished working with the Department of Consumer Affairs' (DCA) SOLID Training Solutions to document all of its business processes and reviewed the recommendations. The BRN is moving forward to implement changes related to the recommendations and taking additional steps to improve and refine our business processes.

The BRN continually analyzes and accepts staff feedback regarding its business processes to develop new and implement revised procedures for the initial licensure application process.

The BRN is assessing efficiency and workload capability and will identify additional resources as needed by the end of 2015 and ongoing.

The BRN will submit requests for additional staff when adequate funding is available to sustain the request.

  • Response Type†: 1-Year
  • Estimated Completion Date: 7/1/2016
  • Response Date: February 2016

California State Auditor's Assessment of Status: Pending


Agency Response*

The BRN has finished working with the Department of Consumer Affairs' (DCA) SOLID Training Solutions to document all of its business processes and reviewed the recommendations. The BRN is moving forward to implement changes related to the recommendations and taking additional steps to improve and refine our business processes.

The BRN continually analyzes and accepts staff feedback regarding its business processes to develop new and implement revised procedures for the initial licensure application process.

The BRN is assessing efficiency and workload capability and will identify additional resources as needed by the end of 2015 and ongoing.

  • Response Type†: 6-Month
  • Estimated Completion Date: 7/1/2016
  • Response Date: August 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

The BRN continuously analyzes and accepts staff feedback regarding its business processes to develop new and implement revised procedures for the initial licensure application process.

Once the BRN has finished working with the Department of Consumer Affairs' (DCA) SOLID Training Solutions in early July 2015 to document all of its business processes and review any recommendations, the BRN will assess efficiency and workload capability and will identify additional resources as needed by the end of 2015 and ongoing.

  • Response Type†: 60-Day
  • Estimated Completion Date: 12/31/2015
  • Response Date: April 2015

California State Auditor's Assessment of Status: Pending


Recommendation #34 To: Consumer Affairs, Department of

Consumer Affairs should continue to work with the phase 1 regulatory entities to ensure that the issues they are facing with BreEZe are being resolved in a timely manner, with particular attention to understanding their reporting needs and improving the system's reporting capabilities.

Agency Response*

The Department has a maintenance and operation (M&O) contract in place which provides dedicated resources to resolving issues for Release 1 entities. Additionally, we have established three user groups (Licensing, Enforcement, and Reports) and multiple forums for Release 1 executives and their staff to ensure the issues are clearly communicated to the Department and addressed in a timely manner.

In addition to the M&O contract, the Department established a dedicated reports development team in March 2013 to address the difficulties and deficiencies with the available BreEZe reports. This team works with the Reports User Group and is charged with developing, prioritizing, and standardizing reports.

  • Response Type†: 60-Day
  • Completion Date: November 2014
  • Response Date: April 2015

California State Auditor's Assessment of Status: Fully Implemented


Recommendation #35 To: Consumer Affairs, Department of

To ensure that future training for BreEZe system rollouts is timely and effective, Consumer Affairs should provide training on the BreEZe system as close to the rollout date as possible to ensure that staff retain the information for using the system as it is implemented.

Agency Response*

For phase 3 regulatory entities, the planning of training activities will be part of the required project planning included in the California Department of Technology's Project Approval Lifecycle (PAL). The Department will ensure planned training activities occur as close to the rollout date within each entity's approved project schedule that is baselined through the PAL process.

Implementation Timeframe: The dates may vary as each entity proceeds through the planning stages of the PAL.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: TBD
  • Response Date: November 2017

California State Auditor's Assessment of Status: Partially Implemented

In its one-year response to this recommendation, Consumer Affairs explained and provided supporting documentation that it had successfully executed a comprehensive multi-stage training curriculum for all BreEZe regulatory entities included in phase 2, which resulted in providing BreEZe system training up to the penultimate business day before system rollout. Because of this, we assessed the status of this recommendation as partially implemented. Until Consumer Affairs provides training activities as close to the rollout date as possible for the phase 3 regulatory entities, we will continue to report this recommendation as partially implemented.


Agency Response*

The Department successfully executed a comprehensive multi-stage training curriculum for all BreEZe Release 2 clients that resulted in providing BreEZe system training up to the penultimate business day prior to system rollout, ensuring the greatest retention of information for using the system as implemented. This multi-stage training consisted of system functional training 2-3 months prior to the scheduled go-live date, which was immediately followed, and reinforced, by business process practice wherein staff were able to utilize the skills gained during functional training and apply it to their board or bureau unit specific business processes with live practice in a "sandbox" copy of the to-be production BreEZe system. As mentioned, this practice continued daily up until the actual rollout date on January 19, 2016. This response remains as partially implemented as the training plan for the programs formerly in Release 3 is yet to be developed.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: TBD
  • Response Date: October 2016

California State Auditor's Assessment of Status: Not Fully Implemented


Agency Response*

The Department successfully executed a comprehensive multi-stage training curriculum for all BreEZe Release 2 clients that resulted in providing BreEZe system training up to the penultimate business day prior to system rollout, ensuring the greatest retention of information for using the system as implemented. This multi-stage training consisted of system functional training 2-3 months prior to the scheduled go-live date, which was immediately followed, and reinforced, by business process practice wherein staff were able to utilize the skills gained during the functional training and apply it to their board or bureau unit specific business processes with live practice in a "sandbox" copy of the to-be production BreEZe system. As mentioned, this practice continued daily up until the actual rollout date on January 19, 2016. This response remains as partially implemented as the training plan for the programs formerly in Release 3 is yet to be developed.

  • Response Type†: 1-Year
  • Estimated Completion Date: TBD
  • Response Date: February 2016

California State Auditor's Assessment of Status: Partially Implemented


Agency Response*

The Department has worked with the BreEZe implementation vendor to ensure that Release 2 BreEZe system training is scheduled as close to the Release 2 go-live date as possible, and continues beyond go-live as needed. This will provide staff a greater opportunity to retain the information for using the system as it is implemented. In the event that the planned Release 2 go-live date is moved, the training will be moved accordingly to ensure it remains as close to the Release 2 go-live date as possible.

  • Response Type†: 6-Month
  • Estimated Completion Date: December 14, 2015
  • Response Date: August 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

The Department has worked with the BreEZe implementation vendor to ensure that Release 2 BreEZe system training is scheduled as close to the Release 2 go-live date as possible, and continues beyond go-live as needed. This will provide staff a greater opportunity to retain the information for using the system as it is implemented. In the event that the planned Release 2 go-live date is moved, the training will be moved accordingly to ensure it remains as close to the Release 2 go-live date as possible.

  • Response Type†: 60-Day
  • Estimated Completion Date: TBD
  • Response Date: April 2015

California State Auditor's Assessment of Status: Pending


Recommendation #36 To: Consumer Affairs, Department of

To ensure that future training for BreEZe system rollouts is timely and effective, Consumer Affairs should work with the regulatory entities to develop training that is specific to each entity's business processes.

Agency Response*

For phase 3 regulatory entities, the planning of training activities will be part of the required project planning included in the California Department of Technology's Project Approval Lifecycle (PAL). The Department will ensure planned training activities include business-process specific training elements.

Implementation Timeframe: For phase 3 regulatory entities, the dates may vary as each entity proceeds through the planning stages of the PAL.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: TBD
  • Response Date: November 2017

California State Auditor's Assessment of Status: Partially Implemented

In its one-year response, Consumer Affairs explained and provided supporting documentation that it deployed an Organizational Change Management team to assist the phase 2 regulatory entities with their transition to BreEZe. Specifically, the Organizational Change Management team was charged with working with the phase 2 regulatory entities to develop and execute training that was specific to each entity's business processes. Because of this, we assessed this recommendation as partially implemented. However, until Consumer Affairs works with the phase 3 regulatory entities to develop training that is specific to each entity's business processes, we will continue to report this recommendation as partially implemented.


Agency Response*

The Department deployed a skilled Organizational Change Management (OCM) team to assist the Release 2 boards and bureaus with their transition to BreEZe. This OCM team was charged with working with the Release 2 boards and bureaus to develop and execute training that was specific to each entity's business processes. This training was the last stage of the multi-stage training approach detailed in response to Recommendation #35. This response remains as partially implemented as the training plan for the programs formerly in Release 3 is yet to be developed.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: TBD
  • Response Date: October 2016

California State Auditor's Assessment of Status: Not Fully Implemented


Agency Response*

The Department has deployed a skilled Organizational Change Management (OCM)team to assist the Release 2 boards and bureaus with their transition to BreEZe. This OCM team was charged with working with the Release 2 boards and bureaus to develop and execute training that was specific to each entity's business processes. This training was the last stage of the multi-stage training approach detailed in response to Recommendation #35. This response remains as partially implemented as the training plan for the programs formerly in Release 3 is yet to be developed.

  • Response Type†: 1-Year
  • Estimated Completion Date: TBD
  • Response Date: February 2016

California State Auditor's Assessment of Status: Partially Implemented


Agency Response*

The Department has deployed a skilled Organizational Change Management (OCM) team to assist the Release 2 boards and bureaus with their transition to BreEZe. This OCM team is charged with working with the Release 2 boards and bureaus to develop and execute training that is specific to each entity's business processes.

  • Response Type†: 6-Month
  • Estimated Completion Date: January 6, 2016
  • Response Date: August 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

The Department has deployed a skilled Organizational Change Management (OCM) team to assist the Release 2 boards and bureaus with their transition to BreEZe. This OCM team is charged with working with the Release 2 boards and bureaus to develop and execute training that is specific to each entity's business processes.

  • Response Type†: 60-Day
  • Estimated Completion Date: TBD
  • Response Date: April 2015

California State Auditor's Assessment of Status: Pending


All Recommendations in 2014-116

Response Type refers to the interval in which the auditee is providing the State Auditor with their status in implementing recommendations made in an audit report. Auditees must submit a response regarding their progress in implementing recommendations from our reports at three intervals from the release of the report: 60 days, six months, and one year or subsequent to one year.

*Agency responses received after June 2013 are posted verbatim.


Report type

Report type
















© 2013, California State Auditor | Privacy Policy | Conditions of Use | Download Adobe PDF Reader