The difficulty of building counterfactuals

What are the effects of the monetary policy actions that central banks around the world have taken since the start of the crisis? How many jobs will be "saved or created" by the stimulus packages put forward by governments? These are crucial questions for policy makers but they are also very difficult to answer. The only way to provide a proper answer to these questions is by building a counterfactual: "what if those actions had not happened?". Easy to say, very difficult to do (at least in a credible manner). This leads to very interesting debates and, in some cases, confusion.

For example, the Obama administration has switched from talking about "jobs created" (by the stimulus packages) to "jobs created or saved". The reason was that they realized that over the first months (years?) of their mandate, we would only see negative growth for employment - i.e. jobs will only be destructed - so it did not make much sense to talk about "jobs created". To claim success of their policies, they need to talk about the jobs that would be saved and claim that without the stimulus packages there would have been even more jobs destructed. Wiliam McGurn in a recent Wall Street Journal article does not like this. Quoting from the article:

"If the "saved or created" formula looks brilliant, it's only because Mr. Obama and his team are not being called on their claims. And don't expect much to change. So long as the news continues to repeat the administration's line that the stimulus has already "saved or created" 150,000 jobs over a time period when the U.S. economy suffered an overall job loss 10 times that number, the White House would be insane to give up a formula that allows them to spin job losses into jobs saved."

I agree that it is difficult to hold governments accountable when we do not have a clear baseline on what would have happened without their actions, but this issue does not just apply to the Obama administration or to the "saved and created" formula. Even if we were in a period of job growth, any claim on "jobs created" would also have to be judged in relation to what would have happened if policies had been different. So whether politicians talk about "jobs created" or "jobs saved", we always need to question the assumptions in their economic projections.

Much of the empirical research in economics is about designing tests that can allow us to build counterfactuals and measure the true effects of policy actions. But in most cases this requires the use of historical time series and bundling together many episodes. And then the difficulty is to extrapolate what we learn from those historical episodes and see whether their lessons apply to the current scenario. As an example, our research might tell us that fiscal expansions have had in the past positive effects on output and employment and we can quantify those effects. But circumstances are always changing and it could be that under the current economic conditions those effects will be smaller or larger. As much as we have historical data on similar episodes, each episode is different from the previous one and it is impossible to control for all relevant factors. Having said that, this is not an excuse that can justify any claim on the number of "jobs created or saved". Baseline scenarios and counterfactuals can be built even if they are uncertain, and they should always be used to judge the impact of economic policies - in bad times and in good times.

Antonio Fatás