Any one who turns on financial television should quickly realize that we like in a kind of “pundit nation.”  The question is do we need “scorecards for pundits”?

By that do we need to track the performance of predictions and security selections for those investment professionals who appear in print and on TV?  This idea has been raised by a number of savvy observers including:

That said, transparency is important and there should be regular reporting about what people have said in the past and what has transpired since.  More of that and maybe the prediction game would feature less inane bloviating about low-probability outcomes and more discussion of how uncertainty is being priced today. – Tom Brakke at the research puzzle

1.  Post the track records of each stockpicker/forecaster as they appear on CNBC TV.  Even if it’s only a record of what they’ve said on the network before. – Josh Brown at The Reformed Broker

2. Public Track Records of Pundits: I was astonished to see the same terrible advice on the TV during the entire market collapse. The spokesperson for a trillion dollar firm recommended buying the dip — the whole way down! Buy Dow 14k, buy 13k, buy buy buy at 12, 11, 10. All the way down. But there was never any mention of the prior horrific calls.

My impossible solution? Mandate that all TV and radio stations reveal the most recent appearance forecast, stock picks, and commentary. – Barry Ritholtz at The Big Picture

We agree with Tom Brakke that this kind of scrutiny would likely reduce the number of questionable predictions by pundits.  All that being said is this kind of scorecard sufficient?  The answer to that is likely no.

The financial media is not structured to both allow for both a prediction and the time necessary to lay out one’s analysis including the potential risks therein.  Daniel Indiviglio at The Atlantic looks at question of why the media continues to ask investors their opinion even though we all know they are talking their book.  He writes:

For better or for worse, investors will likely remain popular sources for business news. Reporters don’t like to stick to interviewing only academics, and readers will tire of their theory and hunger for opinions of those in the trenches. The hope, then, is that readers always take the opinions of investors with great skepticism, unless compelling analysis is also provided.

Whenever any one criticized the work of sell-side analysts when it came to their earnings projections, target prices or buy/sell/hold call the caveat always was:  we add value in our analysis not in the actual output.  There is a kernel of truth in that.  One can learn from looking at a piece of analysis without fixating on the end result:  a point estimate.

The same can be said for big picture pundits as well.  There is something to be said for some one who gets you to think, even if they don’t provide actionable advice.  Justin Fox at the Harvard Business Review took a look at this question in reviewing the ubiquity of star economist Nouriel Roubini.  Fox writes:

He’s a scenario-spinner, not a prophet. And now that his scenarios of the financial future are widely known and at least partially accepted, it’s hard for him to say much of anything that can surprise us.

All of this leaves us with two eternal truths.  The first is that everyone uses models in their predictions.  Jeff Miller at A Dash of Insight writes:

Anyone making a prediction is using a model!

The only question is whether than model has been constructed and tested with rigor and applied in a manner that it provides output with risk estimates.

The second eternal truth is that:  Everybody talks their book, everyone.

Given these two eternal truths, pundit scorecards are a necessary feature that would allow viewers to help judge the credibility of pundits.  However scorecards are not sufficient.  Until we are allowed to, as they say in math class, see the work, we will be somewhat at a loss when those predictions turn out true or false.