Was MHS a FAILURE? SIG vs. Glock, an In-Depth Analysis

[Zarvan]

After 13 years of searching for the US Army’s next pistol, a successor to the Beretta M9 has been selected. However, with the selection of the SIG P320 as the M17 and M18 Modular Handgun System, many individuals in the industry have been compelled to cry foul and demand the Army retry the competition between the two finalists, Glock and SIG.

Although I personally prefer Glock pistols to those made by SIG, and I believe Glock’s pistol was very likely the better of the two weapons, I am going to have to come down against the side that believes the competition should be retried. Doing so, I believe, would be a risky waste of time and money on what is essentially known quantity. Re-opening the problem would extend an already shamefully long effort to find a successor to the Beretta M9 handgun, as well as make the program vulnerable to a significant risk of cancellation. If the latter happened, there would be no new handgun at all, until a new program could be begun.

The video embedded below was released by firearms expert Chris Bartocci, and serves as a basic summary of the complaints against the Modular Handgun System program’s conclusion, and the arguments for re-opening the competition:

A brief summary of the points raised in his argument are as follows:

1. The US Army did not conclude the testing laid out in the MHS RFP, and in doing so selected a winner prematurely after conducting a 12,500 round test. Because testing was not concluded, the Army does not know which pistol is the best one
2. Also, for the same reason, maintenance schedules and spare parts packages cannot be developed for the handguns
3. SIG’s Performance (sic) Verification Testing was conducted on the full-size pistol, with only 500 rounds through the compact variant
4. No environmental/harsh conditions testing was conducted
5. The military could see issues with the pistol after adoption, which testing would have uncovered
6. SIG’s price is likely unsustainable, and he is skeptical about their spare parts bid as they submitted two separate pistols
7. There is a lawsuit out on SIG from Steyr regarding the pistol’s chassis design, which is likely to be decided in Steyr’s favor – royalties from a favorable decision for Steyr would cost SIG even more on price
8. The Government selected for “value” over “performance”
9. The Army has a responsibility to complete this test
10. It’s unfair to the companies who competed
11. The US Army needs to provide the US serviceman with the finest weapon available, even if the weapon in question is just a pistol

Before I get into these arguments, I want to point out a couple of pieces of confusing language in the MHS RFP, the first being the difference between an Offeror and a Contractor. An Offeror is a competitor who has not yet won a contract. A Contractor has already won a contract. The MHS RFP has provision for up to 3 Offerors to become Contractors via contract award; but less than that could be chosen (in the event, only 1 was).

Second is the difference between the Bid Sample Test (BST) and the Product Validation Test (PVT) – which Bartocci accidentally refers to as the “Performance Verification Test”. The BST is a 49,300 round test involving sixteen sample firearms (out of a total of 36 sample firearms). Three of those firearms were each subjected to the 12,500 round test which Bartocci describes in his video, but also 3 more were each subjected to a 3,600 round high temperature test, and 10 more each subjected to a 1,000 round user evaluation.

Bartocci believes that the PVT was a “Phase II” of the program, however the RFP does not reflect this. Instead, it makes clear that the PVT is a test to ensure that the initial production handguns match the performance of the guns tested in the BST. From Section C of the RFP:

STATEMENT OF WORK FOR MODULAR HANDGUN SYSTEM

C.1.1. Objectives

The objective of this statement of work (SOW) is to define the requirements for the production and delivery of the Modular Handgun System (MHS) to the Government in accordance with the Governments MHS Purchase Descriptions, this SOW, and the associated product and data deliverables. The Contractor shall support and sustain the proposed handgun system to meet the Governments quantities required for the U.S. Army to test, operate, maintain and sustain the proposed MHS.

This SOW is broken out into 4 parts to support handgun and ammunition Production Verification Testing (PVT) / Down-Select and Evaluation (DSE) and Production. The layout is as follows:

HANDGUN

Part A Handgun PVT/DSE

Part B Handgun Production / Compact PVT

AMMUNITION

Part A Ammunition PVT/DSE

Part B Ammunition Production

NOTE: The entire statement of work (parts A & B) is applicable to the single Contractor selected as a result of the down-select evaluation.

[emphasis mine]

The confusion regarding the BST versus the PVT is probably the result of this section. The Down-Select Evaluation (DSE) was a secondary evaluation that would only occur in the event that more than one Offeror was awarded a contract. Note that the Statement of Work (SOW) only references “the Contractor”, and states that the SOW applies to a single Contractor as well.

With that out of the way, I will address Bartocci’s individual points as briefly as I can while doing them justice. Note that, while I disagree with Bartocci’s overall conclusions about the program, I do not always disagree with each individual point.

1. The MHS RFP did not guarantee a downselect to more than one pistol. The program manager had – as laid out in the RFP – a choice between downselecting to multiple pistols and running PVTs on both, or downselecting to just one, adopting the handgun, and then performing PVT. We can see that in the language of the RFP, below. From the Executive Summary:
   The Government intends to award up to three (3) Firm Fixed Price (FFP), Indefinite Delivery/Indefinite Quantity (IDIQ) contracts based on the results of the initial evaluation of the proposal submission by following the evaluation procedure contained in section M of this RFP. The Government will then make a final down-selection to a single contractor by following the evaluation procedures contained in section H of this RFP. The period of performance of the base contract(s) will be ten (10) years for the handgun, accessories and spares and five (5) years for the ammunition. Each contract that is awarded will be issued a delivery order in the base year to meet the minimum guarantee for the contract. The minimum contract guarantee will consist of the weapon system component package (CLIN 0001), as described in the statement of work. The weapon system component package items, which will be due 60 days from receipt of order will be used as part of the down-select evaluation as described in section H of this RFP. The Contractors not selected to provide production units, as a result of the down-select evaluation will have their contracts considered complete with no further obligation required by the Government. All bid sample hardware received from unsuccessful Offerors, with the exception of ammunition, will be returned in an as tested condition. These items will be returned to the respective Offeror per the same address from which it was received.
   
   From Section M:
   
   SECTION M – EVALUATION FACTORS FOR AWARDM.1 BASIS FOR AWARDM.1.1 The Government intends to make up to three (3) base awards as a result of this RFP. Each contractor will receive an order for the Weapon System Component Package requirements in accordance with CLIN 1001 of the base contract and Statement of Work C.3.1 which will satisfy the minimum quantity guarantee of each of the ID/IQ contract. The Government will select for award the proposals that are most advantageous and represent the best value to the Government using the trade-off method, with the Source Selection Authority (SSA) giving the appropriate consideration to the nine (9) evaluation factors: Bid Sample Test – Technical, Bid Sample Test – Other, Written Technical, License Rights Ammunition, License Rights Handgun and Accessories, Production/Manufacturing, Price, Past Performance, and Small Business Participation. The Government will weigh the relative benefits of each proposal and award will be made based on an integrated assessment of the results of the evaluation. In making the integrated assessment of the evaluation results, the SSA will give due consideration to all of the Factors and Sub-Factors and their relative order of importance. Offerors that receive a final rating of Red/Unacceptable at any Factor/Sub-factor level are ineligible for award.
   
   These passages demonstrate that the contract allowed the Government both to downselect to just 1 Offeror, and that a contract could be awarded on the basis of the initial Bid Sample Test (BST) and not the Product Validation Test (PVT).This fact is reflected in the opinion presented by the GAO in their rejection of Glock’s protest on the MHS contract.Further, the idea that the 12,500 round per gun (49,300 round per system) test was insufficient to determine which system was “the best” is unsupported. Bartocci cites the 35,000 rounds fired in the 1984 handgun trials that led to the Beretta 92FS being adopted as evidence that the MHS trials were insufficient. However, Bartocci fails to note that the 35,000 round figure was per system, not per gun. From the GAO report on the selection of the XM9:
   
   As shown by the table, the majority of malfunctions were class I, minor Class II malfunctions were generally not a problem. Army systems analysts noted that none of the class III malfunction rates was high considering that about 35,000 rounds had been fired on each system.
   
   In fact, in the XM9 trials, no individual pistol was subjected to firing more than 10,000 rounds:
   
   
   
   
2. Spare parts packages and maintenance schedules are the responsibility of the Contractor, as per the MHS RFP. From Section C:
   C.3.2.11. MHS Rework and Repair
   Rework and Repair Procedures, along with the associated inspection and acceptance procedures, shall be documented by the Contractor and submitted to the Government, DI-MGMT-81910 (CDRL A039) for review and written approval by the PCO prior to implementation.
   
   Also, spare parts packages and maintenance schedules were required to be provided by the Offerors during Bid Sample Testing (BST), as per Section L:
   
   L.1.5.6. Replacement Barrels and Spare and Repair Parts: Each Offeror shall provide replacement barrels, spare and repair parts, magazines, and spare supplies for the cleaning kits adequate to support the evaluation.
   
   NOTE: A sufficient number of replacement barrels, magazines, spare and repair parts as determined by the Offeror, is required to support the following tests.
   
   
3. Bartocci is referring to the BST, not the PVT. Whether only 500 rounds were fired through the compact SIG XM18 or not is unknown, and probably irrelevant. The XM18 is just an XM17 with a shorter slide and barrel, and probably does not have significant performance differences.
4. High temperature testing was conducted as part of the BST, as per Page 317 of the RFP. While no dust tests or other such tests were conducted, it is unlikely that these tests were needed. Every pistol submitted to the competition shared the same basic design, and in the harsh conditions testing conducted during the M9 trials, no pistol displayed an exceptional degree of performance in these conditions. All performed well. From the GAO report on the M9 trials:
   Given that the designs tested in the MHS program were even more similar to each other than those of the XM9 trials, the absence of mud or salt water testing in the MHS program does not seem concerning.
5. While it is possible that the military could see issues with the M17 post-adoption, these issues would almost certainly be due to manufacturing issues or easily changed design details, not due to issues with the fundamental design of the handgun. The fundamental operating mechanisms of all handguns tested in the MHS program are over 100 years old, their architecture over 40 years old, and they were all made by reputable manufacturers. This does not necessarily mean that these handguns will not have issues in service, but does suggest that any issues will be soluble and will not result in program termination.Further, it is the purpose of the PVT and First Article Testing (FAT) to establish that production articles are up to the standards established during the selection process.
6. SIG’s bid is indeed very low, and the possibility that this price might come up in the future represents Bartocci’s strongest argument, by far. However, it is unclear how the Government would be supposed to justify the selection of the higher-priced competitor, over SIG, especially given that SIG’s bid was $100 million lower than Glock’s. If SIG’s price does rise in the future, that could be a matter of contention, but it does not seem to be a sound basis for selection of a different pistol, or for awarding more than one contract.
7. I agree with Bartocci that Steyr’s lawsuit against SIG is – at least technically – on fairly firm ground. However, Steyr’s lawsuit came months after the MHS decision, so while it could possibly be justification for opening up bidding again, it is not a reason to determine that the MHS selection process was a failure.
8. There does not appear to be any evidence that the Government selected for value overperformance. The Government selected for value in the absence of any major performance differences – which is not a shock, given how similar the competitors were, and how mature modern handgun design is.
9. The US Government will very likely complete both PVT and FAT testing, as per the requirements for the MHS program. If they did not, there would indeed be a scandal. Neither of these tests are intended to precede a contract award.
10. Whether we like it or not, Glock competed, and lost. The US Government is under no obligation to award more than one contract.
11. The US Army has an obligation to the taxpayer not to waste their money, as well. In the face of a $100 million price difference between the two bids, any performance differences between the SIG and Glock MHS submissions seem downright trivial, unless a major defect with one or the other is uncovered.

All this rebuttal is not to suggest that the MHS program was not in some way unusual. In June of 2015, the US Army released its second draft of the MHS RFP, and with it, Colonel Scott Armstrong, then Project Manager of Soldier Weapons, was quoted as saying:

We expect to release the final solicitation in 2016; this will be followed by a phased down-select process that will run through 2017.

So in mid-2015, the MHS program was expected to run until 2018 before a selection was made. Yet, selection was in fact made in mid-January of 2017, virtually a full year ahead of schedule. Those familiar with military procurement know that this virtually never happens without major program restructuring, suggesting that something very unusual occurred during MHS’s selection process. Conveniently, I have a theory as to what that was, and it has to do with the timing of the announcement of SIG’s win:



The announcement was made the day before the inauguration of the new administration, which was expected to carry with it a massive change leadership. Specifically, Secretary of Defense Ash Carter would be replaced with General James Mattis. Mattis, a Marine known for being a tenacious and hard-nosed but fair leader, was already by that time expected to shake up the defense landscape as Secretary. With MHS already on the chopping block and Army Chief of Staff Mark Milley having publicly denounced the program’s lagging time schedule, it is not unreasonable to suggest that program management feared it would be cancelled outright. In that event, it’s likely that the program’s officers were searching for ways to end the program early and declare a winner, before it could be cancelled by a porkfat-boiling Secretary Mattis.

Assuming this theory is true (and it may not be), soliciting final bids and down-selecting to just one contract would be the perfect way to close out the program early while still having done due diligence. If this is how it went down, then I applaud the MHS team for making a selection and salvaging a 13 year old effort that might otherwise have been canceled with nothing to show for it. Although SIG would probably not have been my pick, it is very likely that the P320 will do just fine as a standard handgun, and I have nothing but congratulations for SIG for their big win.

It’s not all over for Glock, either. There will be other contracts – the other services are reportedly not entirely on-board with MHS yet – and Glock is still an extremely strong competitor on the market. And, if they’re listening, I hope they bring their XM17 to market, as it’s quite a slick little handgun.


http://www.thefirearmblog.com/blog/2017/07/09/mhs-failure-sig-vs-glock-depth-analysis/

 

[Zarvan]

OPINION: Did the Government fail in its evaluation process for the MHS ? Yes they certainly did!
Shares
Defense, Pistols by Phil White with No Comments

This article reflects the opinion of the author, not necessarily of The Firearm Blog.

Since 2006 the government has been attempting to acquire a new pistol to replace the now well worn M9. There have been many starts and stops, claims of lack of funds to continue and other excuses ad nauseum. The most recent failure not only has Glock livid over the “final” decision but other companies as well who still believe the guns they submitted passed the basic requirements in the first stage of testing and evaluation.

I’m not going to go over all of the documents that have been presented on the Internet over and over again. I will show you a few I doubt you’ve seen before. My intent is to explain what was done in this competition and why. I’ll refrain from expressing a personal opinion of which pistol should have been chosen. That answer is simple it should be the best pistol for our warfighters use to defend themselves when the primary weapon malfunctions.

Our government had a two part test to determine the answer to the question which is the best handgun for the conditions our warfighters operate in. During the first part of the evaluation, the testing was simple and straightforward. Did the 9 pistols presented by several well known companies meet the basic criteria set forth in the solicitation? The bottom line is SIG and the P320 passed as did the MHS Glock.

After these two pistols were chosen to continue both were tested for ergonomics which only required a number of testers hold the pistols in various stances and shooting grips. Further test were done firing the guns while mounted in a Ransom rest type device. The purpose of this test was to determine accuracy and reliability. Testing a pistol for reliability by using a machine doesn’t really give a realistic idea only testing by a human being with all of our differences in stance, grip etc will provide that answer only after firing many rounds.

At some point in this very basic testing, the bids were released. SIG bid was over $1.000,000 less that Glock. Mind you this includes pistols, magazines, parts, training as well as other ancillary equipment. Sometime during this revelation, the government in it’s usual consistency in decision making chose the lowest bidder and called a halt to the second and most revealing portion of the testing. The reasoning behind the governments decision was based on the first set of test where the government declared that both guns were so close in performance there was no need to continue with the second phase of testing and awarded the contract to SIG because of the much lower price that they believed Glock could never match or come close to.

GlockPro

This is when the government made it’s biggest mistake calling off phase two testing even though both the Glock and SIG were chosen to go head to head in the second phase. The second phase should have been conclusive but unless things change we’ll never know who would have won the realistic phase two testing. At this point and considering the pressure being applied it’s unknown if phase two will ever go forward.

Before I continue let me address something that has caused more controversy than anything else and that’s the manual thumb safety. Many people like the idea of a manual safety while many others deplore the idea. Let me clarify this straight from Glock. The solicitation called for a manual thumb safety but the winner would do away with the manual safety. The actual issue pistols would not have a manual safety of any kind. So why did the government ask for a manual safety on the test guns when they knew full well the issued pistols would not have one!

If you’ve studied what would be done in phase two you realize this would be close to real world use. Actual soldiers of every shape and size would fire approximately 35,000 issued 9mm rounds. Over a longer period, the ergonomics would be further tested as would the life expectancy of each entry. The expected figure before breakdown was right at 20,000 rounds. There would be the usual water immersion, mud test, sand, drop test etcetera which when done should have provided conclusive proof of which pistol was correct for our soldiers. Would the low bidder have been the best pistol for the job or the high bidder? We’ll never know unless the test are continued and completed with both companies guns involved.

GAO Redacted MHS Protest Decision (4)

So far with the track record of the past decade, I’m skeptical the government will get back on track in the testing and not base a decision on money as it appears they have done to this point. Which pistol is better? Let’s find out and go forward with phase two and give the soldiers the pistol they deserve to protect their lives and our countries freedom.

Glock’s Protest MHS — Army Contracting Command – 02 24 17

http://www.thefirearmblog.com/blog/...nt-fail-evaluation-process-mhs-yes-certainly/