Report 2014-602 All Recommendation Responses

Report 2014-602: High Risk Update—California Department of Technology: Lack of Guidance, Potentially Conflicting Roles, and Staffing Issues Continue to Make Oversight of State Information Technology Projects High Risk (Release Date: March 2015)

Recommendation #1 To: Technology, California Department of

By December 2015 CalTech should develop and adopt criteria to guide the type and degree of intervention it will take to prevent IT projects with significant problems from continuing without correction, including when and how IPO analysts should recommend corrective action and escalate issues to CalTech's management.

Agency Response*

Reflected in SIMM 45, CDT developed and adopted criteria to guide the type and degree of intervention it will take to prevent IT projects with significant problems from continuing without correction.

  • Response Type†: Annual Follow Up
  • Completion Date: March 2017
  • Response Date: November 2017

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

ITPOD has developed a draft escalation and correction action plan. The final version is expected to be completed in December 2016.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: December 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Not Fully Implemented


Agency Response*

The Information Technology Project Oversight Division ITPOD has developed an escalation process which includes a reporting template and examples of conditions that warrant escalation (See attached). In the course of drafting the escalation process, ITPOD decided to also develop a "Corrective Action Plan" process which will formally notify departments that certain conditions or negative trends must be remediated, and if allowed to continue, could result in suspension or termination of the project. This process will become official State policy which necessitates allowing for consultation with Agency Information Officers and drafting State Administrative Manual updates. The additional time needed for this recommendation is due to the anticipated time needed to appropriately articulate and vet the conditions which warrant this action.

  • Response Type†: 1-Year
  • Estimated Completion Date: July 2016
  • Response Date: March 2016

California State Auditor's Assessment of Status: Partially Implemented


Agency Response*

Activity is progressing in developing the criteria for future use in determining project intervention.

  • Response Type†: 6-Month
  • Estimated Completion Date: Dec 2015
  • Response Date: September 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

- Communicate the criteria for suspending or terminating a project to ITPOD - complete

- Develop high level conditions which can be observed and documented - in process

- Document ITPOD's internal escalation process - in process

  • Response Type†: 60-Day
  • Estimated Completion Date: 12/31/2015
  • Response Date: May 2015

California State Auditor's Assessment of Status: Pending

Although CalTech provided us documentation that it met with its staff to communicate its criteria for suspending or terminating a project, the criteria provided was the same as that included in its response to our report, which we indicated was not sufficient. Specifically, we noted that the criteria provided in its response was a good start; however, the criteria only speaks to circumstances where the need to intervene is apparent. We believe guidance is important in situations where professional judgment is needed—as we note on page 13 of our report—such as before IT projects reach the situations CalTech describes. Furthermore, CalTech's response does not address our concerns discussed on page 15 of our report that, beyond the description in state law, it has not defined the remedial measures, short of suspension or termination, that it may pursue, or the fact that CalTech lacks criteria to guide independent project oversight (IPO) analysts in recommending such measures. According to our IT expert, without such guidance, CalTech cannot meaningfully defend its decisions about whether and when to intervene in a troubled IT project.


Recommendation #2 To: Technology, California Department of

By December 2015 CalTech should develop and adopt criteria to guide the type and degree of intervention it will take to prevent IT projects with significant problems from continuing without correction, including when and what CalTech should require that sponsoring agencies perform as remedial actions, and what sanctions CalTech will impose for noncompliance with these remedial actions.

Agency Response*

Reflected in SIMM 45, CDT developed and adopted criteria to guide the type and degree of intervention it will take to prevent IT projects with significant problems from continuing without correction.

  • Response Type†: Annual Follow Up
  • Completion Date: March 2017
  • Response Date: November 2017

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

ITPOD has developed a draft escalation and correction action plan. The final version is expected to be completed in December 2016.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: December 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Not Fully Implemented


Agency Response*

The Information Technology Project Oversight Division ITPOD has developed an escalation process which includes a reporting template and examples of conditions that warrant escalation (See attached). In the course of drafting the escalation process, ITPOD decided to also develop a "Corrective Action Plan" process which will formally notify departments that certain conditions or negative trends must be remediated, and if allowed to continue, could result in suspension or termination of the project. This process will become official State policy which necessitates allowing for consultation with Agency Information Officers and drafting State Administrative Manual updates. The additional time needed for this recommendation is due to the anticipated time needed to appropriately articulate and vet the conditions which warrant this action.

  • Response Type†: 1-Year
  • Estimated Completion Date: July 2016
  • Response Date: March 2016

California State Auditor's Assessment of Status: Partially Implemented


Agency Response*

Activity is progressing in developing the criteria for future use in determining project intervention.

  • Response Type†: 6-Month
  • Estimated Completion Date: Dec 2015
  • Response Date: September 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

- Develop a "progressive discipline" framework to communicate the need for corrective action to departments - in process

  • Response Type†: 60-Day
  • Estimated Completion Date: 13/31/2015
  • Response Date: May 2015

California State Auditor's Assessment of Status: Pending


Recommendation #3 To: Technology, California Department of

By December 2015 CalTech should develop and adopt criteria to guide the type and degree of intervention it will take to prevent IT projects with significant problems from continuing without correction, including what conditions could trigger CalTech to consider suspending or terminating an IT project.

Agency Response*

Reflected in SIMM 45, CDT developed and adopted criteria to guide the type and degree of intervention it will take to prevent IT projects with significant problems from continuing without correction.

  • Response Type†: Annual Follow Up
  • Completion Date: March 2017
  • Response Date: November 2017

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

ITPOD has developed a draft escalation and correction action plan. The final version is expected to be completed in December 2016.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: December 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Not Fully Implemented


Agency Response*

The Information Technology Project Oversight Division ITPOD has developed an escalation process which includes a reporting template and examples of conditions that warrant escalation (See attached). In the course of drafting the escalation process, ITPOD decided to also develop a "Corrective Action Plan" process which will formally notify departments that certain conditions or negative trends must be remediated, and if allowed to continue, could result in suspension or termination of the project. This process will become official State policy which necessitates allowing for consultation with Agency Information Officers and drafting State Administrative Manual updates. The additional time needed for this recommendation is due to the anticipated time needed to appropriately articulate and vet the conditions which warrant this action.

  • Response Type†: 1-Year
  • Estimated Completion Date: July 2016
  • Response Date: March 2016

California State Auditor's Assessment of Status: Partially Implemented


Agency Response*

Activity is progressing in developing the criteria for future use in determining project intervention.

  • Response Type†: 6-Month
  • Estimated Completion Date: Dec 2015
  • Response Date: September 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

- Draft policy for SAM or SIMM - not started

- Incorporate all content into ITPOD training program - not started

  • Response Type†: 60-Day
  • Estimated Completion Date: 12/31/2015
  • Response Date: May 2015

California State Auditor's Assessment of Status: No Action Taken


Recommendation #4 To: Technology, California Department of

To clarify and reinforce its oversight authority with sponsoring agencies, by December 2015 CalTech should develop a method to formally document and communicate its expectations with the sponsoring agencies whose projects are under CalTech's oversight.

Agency Response*

The new SIMM 45, with a project oversight fact sheet was developed and posted.

  • Response Type†: Annual Follow Up
  • Completion Date: April 2017
  • Response Date: November 2017

California State Auditor's Assessment of Status: Fully Implemented

CalTech provided us its revised IT Project Oversight Fact Sheet, which more clearly outlines its expectations for departments when performing oversight on IT projects. These expectations clearly state that IPO analysts will have access to all documents and meetings for an IT project, and that departments are to take appropriate actions to respond to recommendations that IPO analysts make. Therefore, we consider this recommendation fully implemented.


Agency Response*

ITPOD has developed a draft "Project Oversight Fact Sheet", which will be referenced in and enclosed with the approval letter for exiting each stage of the Project Approval Lifecycle. The final version is expected to be completed in December 2016.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: December 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Not Fully Implemented


Agency Response*

ITPOD has developed a draft "Project Oversight Fact Sheet" which will be referenced in and enclosed with the approval letter for exiting each stage of the Project Approval Lifecycle. (See attached draft Fact Sheet and S1BA Approval Letter Template)

  • Response Type†: 1-Year
  • Completion Date: March 2016
  • Response Date: March 2016

California State Auditor's Assessment of Status: Partially Implemented

While Caltech has spelled its general framework for authority, it needs to include in this framework the importance of it being independent when performing independent project oversight.

  • Auditee did not address all aspects of the recommendation

Agency Response*

Activity is progressing in developing an expectation document for agencies.

  • Response Type†: 6-Month
  • Estimated Completion Date: Dec 2015
  • Response Date: September 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

Plan for Implementation:

- Develop a script for ITPO managers to use at the beginning of each stage of the project approval process which defines the purpose of the stage, the roles and responsibilities of CalTech and the state entity during the stage and the high level conditions for completing the stage - In process

- Develop an approval template to be used at the end of Stage 2 and Stage 4 which includes a citation of CalTech's authority to oversee projects - Not started

  • Response Type†: 60-Day
  • Estimated Completion Date: 12/31/2015
  • Response Date: May 2015

California State Auditor's Assessment of Status: Pending


Recommendation #5 To: Technology, California Department of

To help ensure the independence and objectivity of IPO analysts working in the oversight and consulting division, CalTech should develop a policy outlining expectations for independence and objectivity while performing oversight of IT projects.

Agency Response*

In working with California State Auditor (CSA), ITPOD had completed an updated version of the Principles and Expectations document with additional content on the definition and subject of "independence". This updated document will be forwarded to CSA for closure of this recommendation.

  • Response Type†: Annual Follow Up
  • Completion Date: October 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

ITPOD has established a Principles & Expectations document for all staff. (See attachment) Among a number of important expectations, this document states that ITPO managers must understand and maintain their independence while performing project oversight work. This is now provided to every new staff member and signed by both the staff person and their manager. ITPOD has also included the topic of "independence" as a standing agenda item in our regular weekly staff meetings on Friday. (See attachment) We encourage the staff to share their successes and challenges in maintaining their independence.

  • Response Type†: 1-Year
  • Completion Date: November 2015
  • Response Date: March 2016

California State Auditor's Assessment of Status: Partially Implemented

Caltech's "Principles & Expectations" document lacks a definition of independence, and it does not address how IPO analysts will maintain their independence while performing oversight of IT projects.

  • Auditee did not address all aspects of the recommendation

Agency Response*

Activity is progressing in developing a new policy of expectations for IPO analysts.

  • Response Type†: 6-Month
  • Estimated Completion Date: Dec 2015
  • Response Date: September 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

- Change the name of the IT Project Oversight and Consulting Division (ITPOC) to IT Project Oversight Division (ITPOD) - partially complete

- Develop a policy for ITPOD staff to follow - in process

- Add the concept of independence as a standing item for discussion on the ITPOD staff meeting agenda - completed

- Incorporate the independence concept into the scripts for each stage referenced above - not started

  • Response Type†: 60-Day
  • Estimated Completion Date: 12/31/2015
  • Response Date: May 2015

California State Auditor's Assessment of Status: Partially Implemented


Recommendation #6 To: Technology, California Department of

To help ensure the independence and objectivity of IPO analysts working in the oversight and consulting division, CalTech should provide regular training regarding maintaining independence while conducting project oversight.

Agency Response*

The Quarterly Training Plan needs to be updated to cover Independence including providing examples of situations and coaching to staff. Expected completion date is March 2018.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: March 2018
  • Response Date: November 2017

California State Auditor's Assessment of Status: Pending


Agency Response*

In working with California State Auditor (CSA), ITPOD had completed an updated version of the Principles and Expectations document with additional content on the definition and subject of "independence". This updated document will be forwarded to CSA for closure of this recommendation.

  • Response Type†: Annual Follow Up
  • Completion Date: October 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Partially Implemented

Although CalTech indicates that it includes the topic of independence at each weekly staff meeting and has adopted policy on independence, we believe that IPO analysts need regular training on independence since it is a critical and challenging aspect of them providing independent project oversight on IT projects.

  • Auditee did not address all aspects of the recommendation

Agency Response*

ITPOD has established a Principles & Expectations document for all staff. (See attachment) Among a number of important expectations, this document states that ITPO managers must understand and maintain their independence while performing project oversight work. This is now provided to every new staff member and signed by both the staff person and their manager. ITPOD has also included the topic of "independence" as a standing agenda item in our regular weekly staff meetings on Friday. (See attachment) We encourage the staff to share their successes and challenges in maintaining their independence.

  • Response Type†: 1-Year
  • Completion Date: November 2015
  • Response Date: March 2016

California State Auditor's Assessment of Status: Partially Implemented

As noted for recommendation 5, the "Principles & Expectations" document lacks a definition for independence while performing IT project oversight.


Agency Response*

Activity is progressing in developing a new training program for IPO analysts.

  • Response Type†: 6-Month
  • Estimated Completion Date: Dec 2015
  • Response Date: September 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

- Incorporate the independence policy into the new employee orientation module of the training program - not started

  • Response Type†: 60-Day
  • Estimated Completion Date: 12/31/2015
  • Response Date: May 2015

California State Auditor's Assessment of Status: No Action Taken


Recommendation #7 To: Technology, California Department of

To better track its oversight actions and sponsoring agencies' responses to these actions, CalTech should retain the briefing documents created for the State CIO's portfolio meetings and the project status reports that sponsoring agencies submit while project oversight is ongoing.

Agency Response*

In working with CSA, ITPOD successfully developed a new folder system in SharePoint to capture and retain each month's Briefing Document for all projects subject to CDT oversight. Screen copies of this new folder system and a sample of non-confidential information will be forwarded to CSA this month for closure of this recommendation.

  • Response Type†: Annual Follow Up
  • Completion Date: October 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

Briefing Documents are being saved in ITPOD Share Point system. Copies of saved briefing documents and recorded action items from portfolio meetings were previously provided to the auditors as evidence of completion. (Recent examples are attached)

a. The expectation that every staff member must maintain and update the briefing document is included in an ITPOD Principles and Expectations document.

  • Response Type†: 1-Year
  • Completion Date: July 2015
  • Response Date: March 2016

California State Auditor's Assessment of Status: Partially Implemented

Caltech's response does not address the retention of project status reports that agencies submit while project oversight is ongoing.

  • Auditee did not address all aspects of the recommendation

Agency Response*

ITPOD staff are saving the briefing documents and recording action items from portfolio meetings in the SharePoint e-File room.

  • Response Type†: 6-Month
  • Completion Date: July 2015
  • Response Date: September 2015

California State Auditor's Assessment of Status: Pending

During our follow up with CalTech to obtain documentation to support its response, CalTech informed us that its e-File room in SharePoint needs to be modified to implement version control so that when a new briefing document is uploaded it does not overwrite the prior version. Also, CalTech believes that its PPM Central System that is used to publish project status reports on its public website has a bug that may affect the presentation of the reports' information. As a result, CalTech is still working to fully implement our recommendation to retain briefing documents and project status reports.

  • Auditee did not substantiate its claim of full implementation

Agency Response*

Plan for Implementation:

- Develop a written process for ITPOD staff to follow to retain all briefing documents and project status reports while the projects are in an active status - in process

- Develop a location in SharePoint for retention of action items from Portfolio meetings - not started

  • Response Type†: 60-Day
  • Estimated Completion Date: 07/31/2015
  • Response Date: May 2015

California State Auditor's Assessment of Status: Pending


Recommendation #8 To: Technology, California Department of

To better track its oversight actions and sponsoring agencies' responses to these actions, CalTech should record action items from all portfolio meetings.

Agency Response*

Briefing Documents are being saved in ITPOD Share Point system. Copies of saved briefing documents and recorded action items from portfolio meetings were previously provided to the auditors as evidence of completion. (Recent examples are attached)

b. ITPOD established a process of recording action items from portfolio meetings.

  • Response Type†: 1-Year
  • Completion Date: July 2015
  • Response Date: March 2016

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

ITPOD staff are saving the briefing documents and recording action items from portfolio meetings in the SharePoint e-File room.

  • Response Type†: 6-Month
  • Completion Date: July 2015
  • Response Date: September 2015

California State Auditor's Assessment of Status: Pending

During our follow up with CalTech to obtain documentation to support its response, CalTech informed us that its e-File room in SharePoint needs to be modified to implement version control so that when a new briefing document is uploaded it does not overwrite the prior version. As a result, CalTech is still working to fully implement our recommendation.

  • Auditee did not substantiate its claim of full implementation

Agency Response*

- Develop a process for capturing action items from Portfolio meetings and sharing these with the ITPOD manager for follow-up and retention - not started

  • Response Type†: 60-Day
  • Estimated Completion Date: 07/31/2015
  • Response Date: May 2015

California State Auditor's Assessment of Status: No Action Taken


Recommendation #9 To: Technology, California Department of

To ensure that the sponsoring agencies' project status reports provide a reliable and consistent assessment of an IT project's progress, CalTech should develop and adopt specific standards that describe how to calculate and report the project's current status.

Agency Response*

The revised Project Status Report has been drafted and it is expected that this will be included in an update to SIMM 45 and an associated Technology letter in December 2017.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: December 2017
  • Response Date: November 2017

California State Auditor's Assessment of Status: Pending


Agency Response*

ITPOD has drafted a new Project Status Report format, which includes specific instructions for measuring and reporting of project performance that is related to the schedule, scope and cost. Effort is still ongoing with the Agency Chief Information Officers in implementing this new process and publishing this new policy in the State Administrative Manual.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: April 2017
  • Response Date: October 2016

California State Auditor's Assessment of Status: Not Fully Implemented


Agency Response*

Completion of this recommendation was delayed while the new CA Project Management Framework was being developed. That effort is nearly complete and CDT recently discussed the concerns related to the CA-PMM instructions for completing project status reports with the Agency Information Officers and is considering two options.

a. Create a new Project Status Report format, which includes specific instructions for measuring and reporting of project performance related to schedule, scope and cost. This would be completed by the Project Manager.

b. Utilize the project status dashboard in a new version of the Independent Project Oversight Report (IPOR) which is completed by the ITPOD IPO manager.

The AIOs agreed to work with ITPOD to determine the best approach with the goal of publishing a new policy in SAM by July 2016.

  • Response Type†: 1-Year
  • Estimated Completion Date: July 2016
  • Response Date: March 2016

California State Auditor's Assessment of Status: Pending


Agency Response*

Activity is progressing in developing and adopting specific standards for project status reports.

  • Response Type†: 6-Month
  • Estimated Completion Date: Dec 2015
  • Response Date: September 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

Plan for Implementation:

- Determine if PSRs should continue to be required and if not, identify an appropriate alternative to providing public information about the status of all reportable IT Projects - in process

- Develop standards for measuring project progress and the project's status - not started

- Publish the new standards - not started

  • Response Type†: 60-Day
  • Estimated Completion Date: 12/31/2015
  • Response Date: May 2015

California State Auditor's Assessment of Status: Pending


Recommendation #10 To: Technology, California Department of

To attract and retain employees with appropriate experience and qualifications to perform IT project oversight, CalTech should continue its efforts to gain approval to modify and use the project manager classification for the IPO analyst role.

Agency Response*

The administration is revisiting the IT classification category.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: January 2019
  • Response Date: November 2017

California State Auditor's Assessment of Status: Pending


Agency Response*

It was determined in October 2015 by CalHR that the existing Project Management classification is not suitable for use by ITPOD. CalHR has taken the lead role to modernize a variety of state classifications including IT classifications, and CDT is an active participant in this reform initiative. As stated previously, until better options become available; ITPOD will continue to use the Data Processing Manager (DPM) classification and work with CalHR to gain approval to use the project management classification.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: June 2017
  • Response Date: October 2016

California State Auditor's Assessment of Status: Not Fully Implemented


Agency Response*

It was determined in October 2015 that the existing Project Management (PM) classification is not suitable for use by ITPOD. Because the audit recommendation was based on the assumption that the existing PM classification would assist ITPOD in improving staff retention, we ask that this recommendation be classified as completed. That said, CalHR is leading a significant effort to modernize a variety of state classifications including IT classifications. CDT is an active participant in this reform initiative. Until better options become available, ITPOD will continue to use the Data Processing Manager (DPM) classification.

  • Response Type†: 1-Year
  • Completion Date: October 2015
  • Response Date: March 2016

California State Auditor's Assessment of Status: Pending


Agency Response*

Activity is progressing and a determination regarding the use of PM classification is expected to be made by or before December 2015.

  • Response Type†: 6-Month
  • Estimated Completion Date: Dec 2015
  • Response Date: September 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

Plan for Implementation:

- Evaluate the current classification of PM to determine suitability for ITPOD - in process

- Identify any needed changes and determine if these can be made to the existing PM classification or if a new classification will be required - in process

- Work with CalHR to confirm findings and evaluate options - not started

- Obtain CalHR approval of the best option - not started

  • Response Type†: 60-Day
  • Estimated Completion Date: 12/31/2015
  • Response Date: May 2015

California State Auditor's Assessment of Status: Pending


Recommendation #11 To: Technology, California Department of

To ensure that it provides the appropriate level of oversight for IT projects under development, by December 2015 CalTech should conduct a workload assessment to determine the level of staffing and expertise required for each IT project it oversees.

Agency Response*

ITPOD has established three processes to capture and evaluate workload information:

a) Stage Gate Tracking Spreadsheet (attached) - this provides the status and expectations for completing the work in each stage of the Project Approval Lifecycle. It will also provide earlier insight as to what level and when oversight work will be needed when the project is approved.

b) ITPOD established a monthly report of work activity by activity type, by manager and by branch (attached). This provides Branch managers and the Deputy Director with information about current capacity of individuals and groups within the Division. It will also facilitate better planning for future staff augmentation needs.

c) Effective April 1, every IPO manager will develop a work plan with their supervisor. (attached) This plan will cover a period to be determined by the supervisor depending on the experience and performance of the IPO manager. The work plan will include all anticipated work and relative priorities and, at the end of the period, will document accomplishments and explain any variances to the plan.

  • Response Type†: 1-Year
  • Completion Date: March 2016
  • Response Date: March 2016

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

Activity is progressing in developing the workload assessment.

  • Response Type†: 6-Month
  • Estimated Completion Date: Dec 2015
  • Response Date: September 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

Plan for Implementation:

- Capture quarterly updates of workload activity by ITPOC manager and by Branch - in process

- Determine what new projects will begin in 2015-16 - completed

- Request additional staff needed for 2015-16 projects through provisional budget language - in process

- Develop a BCP for 2016-17 staff augmentation accounting for new projects and other workload activity in the justification - in process

- Document a process for on-going workload tracking and workload projections - not started

  • Response Type†: 60-Day
  • Estimated Completion Date: 12/31/2015
  • Response Date: May 2015

California State Auditor's Assessment of Status: Pending


Recommendation #12 To: Technology, California Department of

Using the workload assessment that we recommended CalTech complete by December 2015, CalTech should make decisions to assign staffing to oversee each IT project. This staffing could include contracted IPO consultants in those situations when CalTech staff either are unavailable or lack the expertise needed.

Agency Response*

ITPOD has established three processes to capture and evaluate workload information:

a) Stage Gate Tracking Spreadsheet (attached) - this provides the status and expectations for completing the work in each stage of the Project Approval Lifecycle. It will also provide earlier insight as to what level and when oversight work will be needed when the project is approved.

b) ITPOD established a monthly report of work activity by activity type, by manager and by branch (attached). This provides Branch managers and the Deputy Director with information about current capacity of individuals and groups within the Division. It will also facilitate better planning for future staff augmentation needs.

c) Effective April 1, every IPO manager will develop a work plan with their supervisor. (attached) This plan will cover a period to be determined by the supervisor depending on the experience and performance of the IPO manager. The work plan will include all anticipated work and relative priorities and, at the end of the period, will document accomplishments and explain any variances to the plan.

  • Response Type†: 1-Year
  • Completion Date: March 2016
  • Response Date: March 2016

California State Auditor's Assessment of Status: Fully Implemented


Agency Response*

Activity is progressing in developing the workload assessment.

  • Response Type†: 6-Month
  • Estimated Completion Date: Dec 2015
  • Response Date: September 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

Once Caltech has completed the workload assessment, and receives a decision for its staffing resource request, it will be in a better position to make decisions to assign the appropriate staffing resources.

  • Response Type†: 60-Day
  • Estimated Completion Date: 02/29/2016
  • Response Date: May 2015

California State Auditor's Assessment of Status: No Action Taken


Recommendation #13 To: Technology, California Department of

To improve its oversight training, by June 2015 CalTech should continue to implement a consistent and repeatable training plan for IPO analysts, which includes contract management, project assessment, IT systems engineering, and maintaining independence.

Agency Response*

The new onboarding process training has been developed. IT Projects Quarterly Training Plan will be updated to cover areas identified by the California State Auditor (CSA). Estimated completion date is June 2018.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: June 2018
  • Response Date: November 2017

California State Auditor's Assessment of Status: Partially Implemented


Agency Response*

In conjunction with CSA'a advice, the ITPOD's plan for implementation includes:

- Develop training priorities for new staff and for continuing staff-in process.

- Identify existing training courses that can be leverage-in process.

- Develop customized content for any additional training course that are needed.

- Finalize curriculum and schedule.

- Seek additional funding, if needed.

  • Response Type†: Annual Follow Up
  • Estimated Completion Date: June 2017
  • Response Date: October 2016

California State Auditor's Assessment of Status: Not Fully Implemented


Agency Response*

As of April 2016, ITPOD will began using a new employee onboarding checklist which will ensure every new IPO manager is provided with the information to perform their work functions and which will be used to plan for the training they will need. (See attachment)

  • Response Type†: 1-Year
  • Completion Date: March 2016
  • Response Date: March 2016

California State Auditor's Assessment of Status: Partially Implemented

The information CalTech provided does not address how IPO analysts will be trained on an ongoing basis, such as the extent of and timing of required ongoing training and continuing education.

  • Auditee did not address all aspects of the recommendation

Agency Response*

Workload and vacancies have impeded progress on the development of this plan. Additional time will be needed to complete it.

  • Response Type†: 6-Month
  • Estimated Completion Date: Jan 2016
  • Response Date: September 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

Plan for Implementation:

- Develop training priorities for new staff and for continuing staff - in process

- Identify existing training courses that can be leveraged - in process

- Develop customized content for any additional training courses that are needed - not started

- Finalize curriculum and schedule - not started

- Seek additional funding if needed - not started

  • Response Type†: 60-Day
  • Estimated Completion Date: 06/30/2015
  • Response Date: May 2015

California State Auditor's Assessment of Status: Pending


Recommendation #14 To: Technology, California Department of

By June 2015 CalTech should put in place a system to track IPO analysts' training hours to ensure that all IPO analysts have completed the necessary California Project Management Methodology training curriculum.

Agency Response*

The CA-PMF replaced the CA-PMM policy in June 2016. The CalQ requirement was not included in the new policy. The individual courses that were previously included in the CalQ curriculum have been included in the ITPOD Staff Competency Checklist. This checklist will be forwarded to CSA this month for closing out this recommendation.

  • Response Type†: Annual Follow Up
  • Completion Date: October 2016
  • Response Date: October 2016

California State Auditor's Assessment of Status: Fully Implemented

The combination of CalTech's two actions--implementation of software in January 2016 to track staff training and adoption of a curriculum policy in June 2016--satisfies our recommendation.


Agency Response*

ITPOD managers now have ready access to training courses taken by ITPOD staff. In January 2016, the CDT Office of Professional Development implemented a major PeopleSoft upgrade to the Training Registration System (TRS). The new system was renamed the Learning Request System (LRS). This system provides several new benefits:

o Improved user-friendly dashboard and screen environment

o Integrated access to the Training & Education Center Catalog and automated enrollment for TEC classes

o Allow administrators to mass-enroll employees into required courses such as Driver Training, Sexual Harassment Prevention, and Security Training

o Reminders feature—staff can easily see approved upcoming training; requested training is easily visible in manager's console

o Upgraded email notification system

In June 2016, the CA Project Management Manual (CA-PMM) will be replaced by the CA Project Management Framework (CA-PMF). The CDT CA Project Management Office (CA-PMO) will be providing CA-PMF training on two levels: for those beginning to learn project management practices and methodologies and for those who would benefit from more in-depth practice in developing tools and templates. The former CalQ requirement will be replaced with a new policy which will recognize experience and credentials from other legitimate sources. ITPOD will take advantage of these opportunities as they become available.

  • Response Type†: 1-Year
  • Estimated Completion Date: June 2016
  • Response Date: March 2016

California State Auditor's Assessment of Status: Partially Implemented


Agency Response*

Workload & vacancies have impeded progress to implement a tracking system for staff training. Additional time will be needed to complete it.

  • Response Type†: 6-Month
  • Estimated Completion Date: Jan 2016
  • Response Date: September 2015

California State Auditor's Assessment of Status: Pending


Agency Response*

Plan for Implementation:

- Collect information on staff completion of CA-PMM training - in process

- Maintain the information in a Division wide spreadsheet - in process

  • Response Type†: 60-Day
  • Estimated Completion Date: 06/30/2015
  • Response Date: May 2015

California State Auditor's Assessment of Status: Pending


All Recommendations in 2014-602

Response Type refers to the interval in which the auditee is providing the State Auditor with their status in implementing recommendations made in an audit report. Auditees must submit a response regarding their progress in implementing recommendations from our reports at three intervals from the release of the report: 60 days, six months, and one year or subsequent to one year.

*Agency responses received after June 2013 are posted verbatim.


Report type

Report type
















© 2013, California State Auditor | Privacy Policy | Conditions of Use | Download Adobe PDF Reader