Per the NYT on January 8, 2016:

““… Sales and marketing concerns influencing the production of ratings. Credit ratings agencies that didn’t have policies and procedures in place to manage issuer-pay conflicts. These are the exact same deficiencies that caused the 2008 financial crisis,” he added, “and that the Dodd-Frank Act was supposed to address.””

If only assessment (e.g., credit default risk) could be as independent and impartial as other assessments (e.g., those in academic settings)…. More than a scoring rubric seems required. Generally, assessment is not entirely automated and free of individual discretion and management override. However, management by policy and procedure is not a panacea for bias, especially where individual interpretation and discretion are required. Moreover, those who believe that an entirely automated process without the biasing hazards of the inspection and oversight provided by individuals would be superior to policies and procedures requiring interpretation, decision-making, and assessment by informed individuals are right and wrong. It’s just that we don’t know in advance when and how much individual inspection and oversight is necessary, though education, training, experience, and reflection help a great deal in this predicament.

Consider an analyst reviewing audited financial reports of a publicly reporting entity: He/she does not know directly, specifically, and precisely how much, if any, the independent registered public accountant’s judgment was adversely influenced by management of the reporting entity. The analyst is also unaware of a generally applicable error / irregularity rate, and the analyst is unaware of the quantitative and collective measure of various incentive compensation plans in operation at the reporting entity. Expert opinion is not a warranty or guarantee. It results from a process akin to abduction; thus, neither wholly deductive (like maths) nor wholly inductive (like hard sciences).