Third-Party Evaluations - What to Expect and What May Not be Apparent
Hiring a company to do a third-party evaluation may be required by a customer or deemed prudent rather than having the work done in-house. If you have never been down this road before, what considerations should you have, and what can you expect?
Qualifications
Obviously, the company or consultant hired to perform the third-party evaluation should be qualified to do so. They should have the technical expertise and experience to independently review and critique the evaluation or analysis that was performed. The prospective engineer should have performed the same types of evaluations and analyses, be familiar with any software used (as necessary), and be well-versed with the requirements of the applicable Codes or Standards.
Verification or Review?
Once the company or consultant has been selected, what can you expect with respect to the thoroughness and extent of their review and critique?
The first thing that should be understood is whether the third-party evaluation is a verification of the work or a review of the work. The difference between a verification and a review is what can be stated about the accuracy of the work based on the amount of information available.
Verification
If a person performing the third-party evaluation has access to all of the information and tools that were required to do the original work, he or she can verify the work is accurate with a high level of certainty.
An example of this would be someone performing a third-party evaluation of ASME Section VIII, Division 1 calculations. With access to the pertinent drawings, specifications, and Code, he or she can check the accuracy of every input and calculated result. This would be considered a verification.
Review
If the person performing the third-party evaluation does not have access to all of the information and tools that were required to do the original work, the level of confidence that the work is accurate will not be as high as with a verification.
An example of this would be someone conducting a third-party evaluation of a stress analysis that was performed using finite element analysis (FEA). He or she will likely not have access to the actual finite element model, analysis, and results. Rather, he or she will be relying on information on these things taken from a report that details what was done. In these cases, the engineer assumes the details provided in the report are accurate. The engineer will not be able to independently confirm the accuracy of all of these details, but rather make an assessment on whether they are appropriate and seem reasonable. This is considered a review rather than a verification.
With a review, the engineer performing the third-party evaluation will not have the same level of confidence about the accuracy of the work as with a verification.
While a verification is preferred to a review, third-party evaluations of finite element analyses are almost always a review. An outside firm considers the finite element model they created as their property and unless contractually obligated usually doesn’t provide this to their customer. This is usually not a problem because the customer often does not have the capabilities to do the FEA themselves. The computer files associated with the model, analysis, and results will be in a format that makes the files unusable without the FEA software.
What to Expect
There are several other issues to be aware of with a third-party evaluation. One is with a review, the thoroughness of the report on the analysis that is being evaluated can vary greatly. Depending on the author and contractual requirements, the report can be extensively detailed or extremely streamlined providing only the most basic information on the analysis. Even with an extensively detailed report, it is unlikely the report will have all of the information the reviewer is interested in. As a result, the reviewer performing the third-party evaluation will have to ask questions to confirm the appropriateness of the analysis and the accuracy of the results and their evaluation.
Questions and comments will be made by the reviewer even if the analysis and evaluation is well documented and completely accurate and correct. There will always be questions and comments. Part of this is because the outside firm performing the third-party evaluation has to justify their costs. A response of “Everything looks great! I have no questions and comments. Here’s my bill” gives the appearance that the reviewer didn’t do their due diligence. So, expect questions and comments.
The main thing to keep in mind when looking at the reviewer’s questions and comments is their relevance to the accuracy of the results and evaluation. It is not uncommon for a majority of the questions and comments to be editorial comments, requests for clarifications, or asking for information on something that will not have any significant impact on the accuracy of the analysis and evaluation.
Summary
To summarize, a third-party evaluation will be dependent on the amount of information available to review. If all of the information is available such that the person conducting the third-party evaluation could reproduce the original work (as in the case of reviewing hand calculations) the confidence level that the original work is accurate will be high. If any of the information is not available, as in the case of evaluating a FEA without having access to the finite element model, the confidence level that the original work is accurate will be less. In these cases, the person performing the third-party evaluation will be relying on a report detailing what was done. The thoroughness of the report will often determine how many questions and comments the reviewer will have on the original work. Lastly, expect questions and comments. That the reviewer has questions and comments does not necessarily mean there is a problem with the original work. Once the reviewer has his or her questions answered to their satisfaction a final assessment can be made as to the accuracy of the work being evaluated.