|Services Offered this Issue
Why Fannie Mae Shouldn’t Keep Data Secret
By Jim L. Sanders & Teresa Huff
Is appraising science? In many ways it is. Believe it or not, the methods for studying science and residential appraising are closely related. Here’s how:
- A good scientific study involves research, data collection, analysis and testing of results to ensure validity and reliability. Conclusions are based on careful study and consistent methods.• A good residential appraiser spends time analyzing, comparing, and calculating the countless variables that contribute to a property’s value. Just as with a scientific study, the appraiser’s final opinion of value is not a number pulled out of a hat; rather, it is based on well calculated research and analysis.
In both science and appraising, others are looking to the expert for a well-grounded, educated conclusion. The professional develops an opinion based on education, experience, firsthand observation, and objective methods with controlled variables. In order to make a credible claim, this estimation must be supported with clear data.
The reverse is true as well; when there is a flaw anywhere in the process – data, methods, analysis, or otherwise – the results will not be accurate but rather an estimate based upon data collected and the method used to analyze the data. The only way to validate the study is to repeat the process to ensure correct models and data.
Appraisal Tools and Methods
Residential appraisers must provide a transparent basis for their opinion of property value when submitting an appraisal report to a lender. These appraisals are then sent to Fannie Mae for review and the underwriting process begins. Richard Hagar, SRA who teaches classes on the topic, notes that reviewers may select an underwriting program provided by Fannie Mae or other vendors.
The latest tool of choice, Collateral Underwriter™ (CU™), is “a proprietary appraisal risk assessment application developed by Fannie Mae to support proactive management of appraisal quality.” This tool allows Fannie Mae to use a proprietary statistical model for real estate analysis. The lender compares the appraiser’s adjustments against this model’s results to see if they agree. The output of the CU analysis is read by the reviewer, pointing them to areas that should be further analyzed by a human. After further scrutiny, including reading the appraisal report and comments, Hagar shares that the reviewer has four options:
1) Accept the original report as written,
2) Ask questions of the appraiser,
3) Ask for clarification or more detailed explanations, or
4) Reject the appraisal and order another.
Appraisers and other professionals are invited to learn more about CU through Fannie Mae’s training resources. They cannot, however, access CU. The software is only available to lenders and their underwriters.
Today’s lenders have access to massive amounts of data. According to Hagar, government lenders have access to every recorded sale in most every county across the U.S. via information providers like CoreLogic. He believes appraisal adjustments should be cross-checked against sales metrics of the five million home sales that occur each year.
However, appraisers typically use Multiple Listing Service (MLS) for residential appraising. Data provided by MLS may vary from information shown by providers like CoreLogic. MLS has more detailed information that county records don’t contain. Oftentimes, though, MLS contains errors or missing data. In other words, both types of systems have errors, but not necessarily the same errors.
Increasingly, appraisers indicate that underwriters are rejecting appraisal reports for vague reasons based on ambiguous results from CU analysis. Problems arise when these methods are not accessible or replicable for others to validate results. Mr. Hagar emailed us his opinion:
Many appraisers do not understand how the CU system works. When there is a lack of understanding, people tend to reject the results and fight back. When the CU system spits out, “Your adjustments are different from what the model indicates or other appraisers in your area are using,” many appraisers don’t understand how the lender or CU arrived at this conclusion. What algorithm was used? What data were analyzed? All are legitimate questions that usually go unanswered.
By design, the CU output is only provided to the lender. The person reading the output is not supposed to simply cut and paste the comments into an email and send them to the appraiser. Unfortunately, many poorly trained people working for lenders and appraisal management companies do just that – cut and paste – without telling the appraiser, “Data indicates something different than what you indicated; how did you [the appraiser] determine the adjustment for x?” The CU isn’t necessarily telling the appraiser they are wrong; it is merely stating that their adjustments are different and therefore questionable.
Many appraisers fail to understand that they do indeed have access to the same sales data that the CU uses. Sales are sales, regardless of the storage method. The CU analyzes the sales data using their secret algorithm. Appraisers can use the same data along with a different method and produce a similar or different conclusion. For this reason, appraisers cannot guess at what an adjustment should be; they must have a valid method to determine the adjustment or their conclusions will be tested and questioned.
This brings us back to where we started; data must be properly analyzed using an acceptable, repeatable methodology. Otherwise, results are merely a guess.
(story continues below)
Transparency & Reproducibility
Transparency and reproducibility are key ingredients of good science and, we would argue, of appraising also. It requires that data and methods, including computer code, be made available. Whatever the information vessel – in science, appraisals, or otherwise – these elements are imperative to validate methods. Denying access to the methods used to form an opinion becomes a serious issue, such as with CU, when the results are viewed as “true.” Neither science nor appraising can be replicated on its own; people need the data to recreate the studies. They need the model to verify the methods.
This is from (Shuttleworth, M. Reproducibility. 14 June, 2009): “Reproducibility is regarded as one of the foundations of the entire scientific method, a benchmark upon which the reliability of an experiment can be tested. The basic principle is that, for any research program, an independent researcher should be able to replicate the experiment, under the same conditions, and achieve the same results.”
On the surface, CU guidelines align with these principles by promising, “Appraisers that make a good faith effort to use the most similar comparables, provide accurate and consistent data, and support their adjustments with market data and analysis can generally expect a minimum of CU feedback that would cause a follow-up request from the reviewer.” However, word on the street from appraisers shows quite the opposite. As mentioned previously, lenders are sending feedback such as:
- Do you have data to support this adjustment?
• We are asking that you take a class on how to support and prove your adjustments.
• The appraiser’s net adjustments for the comparable sales are materially different from the model net adjustments.
• The appraiser-provided comparables are materially different than the model-selected comparables.
How is the appraiser to understand feedback like this and understand the models when they aren’t made available? The process doesn’t lend itself to either transparency or reproducibility.
Fannie Mae emphasizes, “An accurate description of the physical condition and quality of the subject property is a critical element in arriving at a supportable opinion of market value, as well as in the prudent underwriting of a mortgage loan.” Lenders initially rely on appraisals to give an accurate firsthand description of the subject since the appraisers are, after all, on the front lines of evaluating properties. However, lenders later rely on CU feedback to tell them whether or not the appraiser’s eyewitness assessment is correct.
According to CU guidelines outlined by Fannie Mae, specific factors contribute to successful CU implementation. Among these are considering the restrictions of automated analysis, awareness of possible property or neighborhood nuances, and ensuring educated appraiser opinions are a priority over computerized results.
Fannie Mae’s guidelines expect lenders to “use human due diligence in combination with the CU findings.” They go on, “Lenders should not, however, make demands or provide instructions to the appraiser based solely on automated feedback.” However, this appears to be what is happening. Appraisers are accountable for conclusions driven by unknown models and an obscure algorithm for selecting sales to analyze. Their common-sense opinions are being overridden by the all-knowing system.
CU legalities forbid providing appraisers access to the CU interface or reports containing CU findings. Yet Fannie Mae claims, “CU is model-based and performs a more comprehensive analysis of data integrity, comparable selection, adjustments, and reconciliation.” If this were the case, according to basic scientific principles, then outside parties (appraisers) would be allowed access in order to verify the data and analysis.
In short, residential appraisers may be unfairly criticized and condemned for their conclusions and have no method for rebuttal. Potential discrepancies are issued based upon this model without the appraiser being fully informed of the basis for criticism; the appraiser has no opportunity to examine the questionable sales and explain why adjustments were made. No clarification is offered regarding why the appraiser’s opinions are contested. The process caters to underwriters, often leaving appraisers frustrated and unable to defend their work.
Shine the Light
Once again, CU models may be perfectly good- or they may not be. We have no way of knowing since Fannie Mae won’t share them. This leaves ambiguity whether appraisers are being treated fairly.
We must ask ourselves:
- Is the CU tool a one-size-fits-all method?
• If so, is the implementation in keeping with its own design of, “Well-informed human judgement should take precedence over automated results”?
• More importantly, do CU methods align with the basic principles of good science?
Until the models are made available we don’t have answers. A disconnect exists between Fannie Mae guidelines, lender interpretations and feedback to the appraiser. Meanwhile, appraisers must continue responding to lender requests and defending their conclusions as thoroughly as possible with the limited tools available to them.
> CE Online – 7 Hours (approved in 40 states)
How To Support and Prove Your Adjustments
Presented by: Richard Hagar, SRA
Must-know business practices for all appraisers working today. Ensure proper support for your adjustments. Making defensible adjustments is the first step in becoming a “Tier One” appraiser, who earns more, enjoys the best assignments and suffers fewer snags and callbacks. Up your game, avoid time-consuming callbacks and earn approved CE today! Sign Up Now! $119 (7 Hrs)
OREP Insured’s Price: $99
About the Author
Jim L. Sanders, MBA, is the owner of Real Estate Appraisal Litigation in Tucson, Arizona (RealEstateAppraisalAndLitigation.com). An appraiser since 1975, Sanders delivers expert witness testimony in state and federal courts. Teresa Huff, MS Ed., provides copywriting, editing, and development services to small businesses and solopreneurs. Find out more at www.AdeoDevelopment.com
Send your story submission/idea to the Editor: firstname.lastname@example.org