​​​​​​​​​​​​​​​​​​​​Correspondence with excerpts: 

Waxman letter, March 10, 2008, Subject: Award Fees, Contract Oversight and Lockheed Martin

  •  Both DoD and the U.S, Navy issued policy memoranda which disclose that EVM is not working for acquisition management. The DoD memorandum states that:

             1. EVM is not serving its intended function in the internal control process

             2. DoD is committed to resolve systemic,  DoD-wide weaknesses

  • The Navy memo is explicit regarding contractor issues. It states that reviews have found broad deficiencies across multiple contractor sites including intentionally masking cost and schedule variances and failure to document and manage changes to the baseline 

Zients OMB letter, Dec.13, 2009, Subj: Section 302 of the Weapon System Acquisition Reform Act (WSARA) (Earned Value Management)

Please consider acquisition reform to improve the effectiveness of EVM for the acquisition of capital assets.

DoD Report to Congress excerpts:

  • Contractors may circumvent proper EVM practices to keep EVM metrics favorable and problems hidden
  • Engineering community should establish technical performance measures that enable objective confirmation that tasks are complete
  •  Various subsystems that make up many contractors' EVMS are not integrated, resulting in inconsistent portrayals of status

​​​​​​​​​​​​Skelton letter, 3/28/10, Subject: Panel on Defense Acquisition Reform 

  • Neither the DFARS EVMS clause 252.234-7002 nor its cited EVMS guidelines in EIA-748 require that contractors report progress toward achieving quality or technical goals that are specific and measurable. 
  • In fact, ANSI/EIA-748 states that earned value is a “measurement of only the quantity of work” and that “quality and technical content of work performed are controlled by other means” (Section 3.8). Guideline 2.2.b describes the use of technical performance goals to measure progress as an option, not a requirement.
  • DoD’s Report to Congress, DoD Earned Value Management: Performance, Oversight, and Governance, stated that contractors “keep EVM metrics favorable and problems hidden.” The deficiency in DFARS and EIA-748 enables contractors to report metrics that are more favorable than actual conditions and to defer reporting of real problems. 
  • ​Today, neither the acquisition managers nor the PARCA office can be assured that a contractor’s performance metrics are valid or accurate. To my knowledge, neither the GAO nor any other agency ever validated that EVMS truly integrates cost, schedule and quality/technical performance or that it provides accurate status and Estimate at Completion. There is a need to transform EVMS into a more valuable acquisition management tool that will provide early warning of performance problems on a consistent basis.


McKeon letter, 9/13/11, Subject: Defense Acquisition Reform

  • Contractors must submit monthly Contract Performance Reports (CPR) that show cost and schedule performance based on EVMS guidelines in the industry standard, EIA-748. This standard, maintained by the NDIA, has loopholes and deficiencies that enable contractors to submit flawed, inaccurate data to the military. Even if a DCMA compliance review determines that a contractor is compliant with the guidelines, the loopholes enable a contractor to overstate progress and understate final costs. Consequently, a DCMA compliance review can provide false assurance to the Program Manager. This is like relying on Standard & Poor’s “no risk” ratings of mortgage-backed securities before the financial collapse.
  • A key deficiency is the lack of a requirement for contractors to link earned value to technical performance or quality. The FAR, DFARS, and EIA-748 enable contractors to report earned value that is based only on the quantity of work performed and to ignore technical performance or quality (Quality Gap). For example, earned value is often based on the quantity of drawings completed or software code written. Although technical performance measures (TPM) are used, such as weight, payload, or software functionality, contractors often ignore these TPMs when calculating earned value. As a result, the CPRs fail to provide early warnings of cost overruns and behind schedule conditions. More accountability and transparency is needed. 

Sen. Collins  said the following about EVM in the WSARA conference report: “GAO observed that contractor reporting on EVM often lacks consistency, leading to inaccurate data and faulty application of the EVM metric. In other words, garbage in, garbage out.”

As required by WSARA and the 2009 NDAA, DoD submitted a report to Congress in September 2009. It concluded the "Utility of EVM has declined to a level where it does not serve its intended purpose." It stated that:

  • Contractors may circumvent proper EVM practices to keep EVM metrics favorable and problems hidden.
  • Engineering community should establish technical performance measures (TPM) that enable objective confirmation that tasks are complete.
  • Systems Engineering and EVM should be integrated, not stove-piped.​

The Skelton provisions in NDAA for FY 2011Sections 864 a, b, and c) require DoD to:

  • Review acquisition guidance, including DoD Instruction 5000.02, to “consider whether measures of quality and technical performance should be included in any EVM system.
  • Submit a report to the Armed Services Committees.   


McCain letter, 10/25/11, Cost Controls on the F-35 and Need for Acquisition Reform of EVM​

The acquisition regulations and process enable contractors to submit invalid, misleading information to the Government on all capital asset acquisitions, not just weapon systems. We need to improve transparency and accountability when contractors use a contractually-required Earned Value Management System (EVMS) on cost-reimbursement contracts.

The EVMS guidelines are similar to Generally Accepted Accounting Principles (GAAP). However, GAAP protects investors but EVMS often fails to protect taxpayers. The standard has ambiguities and loopholes that should be removed.

The two most important deficiencies are:

  1. Contractors are able to overstate progress and understate final costs (called Estimate at Completion or EAC) by basing earned value on the quantity of work performed, not on technical performance or quality.

  2. Contractors routinely use “Management Reserve” (MR) as a slush fund for additional budget to compensate for poor planning, to reduce reported cost overruns, and to provide additional budget for tests and rework to designs that fail to meet technical performance or quality objectives.

McCain letter, 12/20/11, Proposed Amendment to NDAA for 2012 for

EVM Acquisition Reform

The current DFARS enables contractors to submit flawed, monthly Contract Performance Reports that overstate cost and schedule performance and understate the final costs (called Estimate at Completion or EAC). 

The proposed amendment would require the Secretary of Defense to modify DFARS to force contractors to base earned value on technical performance or quality, instead of just the quantity of work performed. The new amendment would be a follow-up to NDAA for 2011, Section 864. That provision addressed including measures of quality and technical performance in any EVM system.

Specifically, please propose an amendment that would add requirements to DFARS 234.201 and 252.2347002.

The specific DFARS revisions should state that, in addition to compliance with the guidelines in EIA-748, the contractor shall be in compliance with the following augmentations to those guidelines:

 Guideline 2.1(a): Add "product scope" to work scope.

 Guideline 2.2(b): Require mandatory, not optional, identification of "technical performance goals" and measurement of technical.​​


Thornberry letter, 11/17/13Expanded NDAA Defense Acquisition Reform - Earned Value

Compliance with the EVMS guidelines that are required by DFARS is costly to the programs and the taxpayers. However, compliance does not ensure that Contract Performance Reports provide timely, reliable, or verifiable information, as required by DFARS 252.234-7002. Nor does compliance ensure that performance-based acquisition management meet the OMB objective to measure progress towards milestones, cost, capability to meet specified requirements, timeliness, and quality (OMB Circular No. A-11, Sec. 300-5).

Recommendations:

  • Tailor 3 EVMS guidelines to meet OMB objectives to measure progress towards capability to meet specified requirements and quality.
  • ​ Remove compliance with 12 guidelines to reduce costs for contractor labor, DCMA oversight, training, consultants, and software.


​ In my opinion, the cost savings by eliminating compliance with eight guidelines will offset any cost increases that may be incurred because of the tailored guidelines.


McCaskill letter, 4/2/14, Request for Earned Value Management Acquisition Reform

McCain letter, 1/21/15, Cost Controls on the F-35 and Need for Acquisition
Reform of EVM


Letter of Appreciation from Sen. John McCain, March 5, 2015, for continuing efforts to improve acquisition reform.


 

McCain letter, 11/5/16, Cost Overruns and Delays on the F-35 Program and Need for Acquisition Reform

  • DoD failed to implement improvements that were discussed in its 2010 DoD EVM Report to Congress that was required by NDAA and WSARA.
  • Failures of DoD to implement the objective of its 2004 Policy for Systems Engineering (SE Policy).
  • DCMA assessment that contractors are not integrating Technical Performance Measurement (TPM) with EVM. 

Trump letter, 11/13/16, DoD Acquisition Reform - Under Budget and Ahead of Schedule  

Unfortunately for taxpayers, contractors often exploit permissive EVMS guidelines and submit monthly contract performance reports (CPR) that overstate cost and schedule performance and understate the estimated completion costs and schedule. The CPRs often fail to provide an early warning of pending funding shortfalls and delays to providing working weapons to the warfighters.

The biggest loophole in the EVMS guidelines allows contractors to report progress in terms of percent complete that is based on flawed, misleading metrics. In construction terms, contractors may report cost and schedule performance based on the number of actual vs. planned architectural drawings, or floors built, even if the rising structure will not meet building codes, seismic risks etc. Contractors are not required by the EVMS guidelines to assess and report progress that is based on achieved vs. planned technical performance or quality (Quality Gap).

McCain letter, 1/4/18, Fraud on the F-35 Program and Need for Acquisition Reform 

McCaskill letter, 1/11/18 , Fraud on the F-35 Program and Need for Acquisition Reform 

Mulvaney letter, 1/17/18 ,​​​​ Project Management of High Risk DoD Acquisitions 

The Program Management Improvement and Accountability Act of 2015 (PMIAA) calls for adoption of government-wide standards, policies, and guidelines for program and project management for executive agencies. Mr. Gene Dodaro, in discussing the GAO’s 2017 high risk list, stated that the statute, if implemented effectively, will help foster progress on high-risk issues government-wide.

However, PMIAA is not applicable to DoD “to the extent that the provisions…are substantially similar to or duplicative of policy, guidance, or instruction of the Department related to program management.” In my opinion, DoD does not have similar or duplicative policy, guidance, or instruction. Consequently, DoD’s high risk acquisitions are off the hook. They will continue to incur cost overruns and schedule slips. Please consider proposing remedial legislation to require that PMIAA be applicable to DoD.

The primary Program Management (PM) standard used by DoD is the Earned Value Management System Standard (EVMS), EIA-748. The most widely accepted standard for project management, planning, and delivery is the “PMI Project Management Body of Knowledge” (PMBOK Guide). In comparing the two standards, it is obvious that EVMS has fatal deficiencies. The characteristics of EVMS that fail to meet the needs of a PM are:
1.    EVMS addresses only the work scope but is silent on product.
2.    The use of technical performance measures (TPM) is optional, not mandatory.
3.    Earned value is a direct measurement of the quantity of work accomplished.  The quality and technical content of work performed is controlled by other processes. I call this the “Quality Gap.”

In April 2016, DCMA made a presentation to the National Defense Industrial Association (NDIA) PM Systems Committee. It reported a common, EVM finding of a lack of objective measures to assess performance, including “Measurement does not indicate technical accomplishment.”

A PM needs a project management tool that integrates cost, schedule, and technical performance. However, the primary DoD PM instruction, “DoD Instruction 5000.02, Operation of the Defense Acquisition System,” does not fix the EVMS standard’s failures and close the Quality Gap. Its discussion of TPMs and metrics, in terms of progress against established plans, fails to discuss any linkage with EVM.​

Thornberry letter, 1/28/18 ,​​​​ Subject: Updated NDAA Defense Acquisition Reform - Earned Value

The Program Management Improvement and Accountability Act of 2015 (PMIAA) calls for adoption of government-wide standards, policies, and guidelines for program and project management for executive agencies. Mr. Gene Dodaro, in discussing the GAO’s 2017 high risk list, stated that the statute, if implemented effectively, will help foster progress on high-risk issues government-wide.

However, PMIAA is not applicable to DoD “to the extent that the provisions…are substantially similar to or duplicative of policy, guidance, or instruction of the Department related to program management.” In my opinion, DoD does not have similar or duplicative policy, guidance, or instruction. Consequently, DoD’s high risk acquisitions are off the hook. They will continue to incur cost overruns and schedule slips. Please consider proposing remedial legislation to require that PMIAA be applicable to DoD.

The primary Program Management (PM) standard used by DoD is the Earned Value Management System Standard (EVMS), EIA 748. The most widely accepted standard for project management, planning, and delivery is the “PMI Project Management Body of Knowledge” (PMBOK Guide). In comparing the two standards, it is obvious that EVMS has fatal deficiencies. The characteristics of EVMS that fail to meet the needs of a PM are:
1.    EVMS addresses only the work scope but is silent on product.
2.    The use of technical performance measures (TPM) is optional, not mandatory.
3.    Earned value is a direct measurement of the quantity of work accomplished.  The quality and technical content of work performed is controlled by other processes. I call this the “Quality Gap.”

In April 2016, DCMA made a presentation to the National Defense Industrial Association (NDIA) PM Systems Committee. It reported a common, EVM finding of a lack of objective measures to assess performance, including “Measurement does not indicate technical accomplishment.”
A PM needs a project management tool that integrates cost, schedule, and technical performance. However, the primary DoD PM instruction, “DoD Instruction 5000.02, Operation of the Defense Acquisition System,” does not fix the EVMS standard’s failures and close the Quality Gap. Its discussion of TPMs and metrics, in terms of progress against established plans, fails to discuss any linkage with EVM.

Revise DFARS to Replace EIA-748 with a Project Management Standard (PMIAA)    
An alternative to implementing the recommendations in my 2013 letter is to abandon the requirement for compliance with EIA-748. I now recommend that DoD implement the Program Management Improvement and Accountability Act of 2015 (PMIAA) instead of having a waiver. The Act calls for adoption of government-wide (excluding DoD) standards, policies, and guidelines for program and project management for executive agencies. The Project Management Institute (PMI) Project Management Body of Knowledge (PMBOK Guide) is the only standard that meets PMIAA criteria.​​

Email to Undersecretary of Defense Ellen Lord, 2/3/18:

Subject:Section 809 Panel Report on Streamlining Acquisition Confirms shortcomings of EVM

Ms. Lord,
The Section 809 Panel Report on Streamlining and Codifying Acquisition Regulations, issued January 31, 2018, confirms what I have been asserting: the limited value of Earned Value Management. EVM does not measure product quality.  

Excerpt from Sec. 809 Report:
“Limited Value of EVM  
Another substantial 
shortcoming of EVM is that it does not measure product quality. A program could perform ahead of schedule and under cost according to EVM metrics, but deliver a capability that is unusable by the customer...Traditional measurement using EVM provides less value to a program than an Agile process in which the end user 

continuously verifies that the  product meets the requirement.” 

The Report’s Recommendation 19 is to “Eliminate the EVM mandate for software programs using  Agile methods.”


Previous Findings and Recommendations


In my email to you, dated Jan. 14, I cited an April 14, 2014 letter to Sen. McCaskill which stated that “Contractors are permitted to report progress in terms of the percent of work completed and may ignore reporting progress towards meeting the technical performance requirements or quality.”
The quality gap was reported in the May, 2004 article in Defense AT&L, “Integrating Systems Engineering with EVM.
Excerpt from 2004 article:  

“EVM Limitations
With regard to a PM’s needs, there are several limitations of EVMS that can be overcome by integrating EVM with robust systems engineering.
EVMS states that EV is a  measure of the quantity, not quality, of the work accomplished. A PM should also ensure that EV also measures the quality and technical maturity of technical work products instead of just the quantity of work. Robust systems engineering processes should provide TPM and exit criteria for assessing technical maturity that are quantitatively linked to EV.”

The May 2011 article in Defense AT&L, "Path to EVM Acquisition Reform," reported that DoDI 5000.02 provides no guidance to link measures of quality and technical performance with EVM.” The article includes recommended regulatory changes that would require that EV be linked to quality. The article also cites the NDAA for FY 2011 which addresses this issue in Ike Skelton’s markup provision.

My 2012 consulting report and slides for PARCA/NAVAIR included this issue:          









Again, as stated portrayed in yesterday’s email, from the 2004 article:








Unfortunately for taxpayers, contractors often exploit permissive EVMS guidelines and submit monthly contract performance reports (CPR) that overstate cost and schedule performance and understate the estimated completion costs and schedule. The CPRs often fail to provide an early warning of pending funding shortfalls and delays to providing working weapons to the warfighters.


Then Why Use EVM on non-Software Programs?
The Section 809 Panel recommends revising DFARS Subpart 234.201, DoDI 5000.02 Table 8, and OMB Circular A‐11 to eliminate the EVM mandate for specified software programs. So why not do the same for all EMD programs for which compliance with the EVMS Standard is required? 

My PARCA white paper and many articles and tutorials on my website provide specific acquisition reforms and recommendations for implementing the needed fixes.

Incidentally, several articles on the website specifically address software development, including Agile projects and the use of EVM for commercial IT development at Samsung IT. 


Letter to Ms. Weichert (OMB) and Mr. Fahey (DoD), 7/26/18

Subj: Successful Implementation of PMIAA by all Agencies, including DoD


Letter to Ms. Weichert (OMB) and Mr. Fahey (DoD), 7/31/18
Subj: How Commercial IT Companies Use Earned Value Management with P/PM

DoD Documents:

New DODI 5000.02, 1/1/15, includes TPM but no link to EV:

The technical planning documented in the Systems Engineering Plan (SEP) will guide the details in the program’s schedule. Program managers should include the SEP in the RFP as either guidance or a compliance document depending on the maturity of the plan and the acquisition strategy.


​The SEP should address the following: software unique risks; inclusion of software in technical reviews; identification, tracking, and reporting of metrics for software technical performance, process, progress, and quality. 
​Technical risks should be quantified and implications reflected in the program’s Integrated Master Schedule and Integrated Master Plan.  ​The Program Manager will use technical performance measures and metrics to assess program progress. 


Needed acquisition reform: Revise Dodi 5000.02 to link EV to TPMs.  

Letter to Ms. Weichert (OMB) and Mr. Fahey (DoD), 7/26/18

Defense Acquisition Guidebook (DAG) 11/2/17

CH 3–2.7 Systems Engineering Role in Contracting

Incentives such as award fee may be tied to program performance and progress that may be evaluated during technical reviews, or more frequently the incentive is tied to the completion of a technical review. 
Another area to which incentives are tied is the EVMS. The PM should ensure that the EVMS, tied to any incentive, measures the quality and technical maturity of technical work products instead of just the quantity of work. If contracts include earned value (EV) incentives, the criteria should be stated clearly and should be based on technical performance. EV incentives should be linked quantitatively with:

  • Technical performance measurement (TPM)
  • Progress against requirements
  • Development maturity
  • Exit criteria of life-cycle phases
  • Significant work packages and work products


Note: For, additional guidance to implement DAG Ch 3-2.7, see the source, Defense AT&L article, "Systems Engineering (SE) and EVM Support for Performance-Based Awards," Jan. 2007 Link to PDF
Excerpts:

EV can also be a valid basis for award fee determination if it is tied to technical performance, not just to work accomplished.

Proper scoping of the technical tasks in the Statement of Work (SOW), Statement of Objectives (SOO), or Performance Work Statement (PWS) is necessary to ensure that the final system meets end user’s needs.

Another area to which incentives are tied is the EVMS. The PM should ensure that the EVMS, tied to any incentive, measures the quality and technical maturity of technical work products instead of just the quantity of work. If contracts include earned value (EV) incentives, the criteria should be stated clearly and should be based on technical performance. EV incentives should be linked quantitatively with:

  • Technical performance measurement (TPM)
  • Progress against requirements
  • Development maturity
  • Exit criteria of life-cycle phases
  • Significant work packages and work products


DCMA Report to NDIA:: "DCMA Update", April 14, 2016, slide 18:

Common, EVM finding: a lack of objective measures to assess performance, including “Measurement does not indicate technical accomplishment.”

Link to article, "Path": AT&L May 2011 path.pdf

Performance-Based Earned Value