Beyond Outdated Economic Statistics

- Current data collection methods are outdated and inadequate for capturing the complexities of today's workforce and economy.
- A new approach to data infrastructure, leveraging administrative records and modern technologies, could revolutionize economic analysis and policymaking.
- International examples, particularly New Zealand's integrated data system, offer valuable lessons for modernizing U.S. data collection and analysis.
Our understanding of the workforce and economy is only as good as the data we collect. Yet, our methods for gathering this crucial information have not kept up in an era of rapid technological change and evolving work patterns.
"No one can tell you what data sources to use, and no one can tell you what data sources are any good," asserts Julia Lane, professor at NYU Wagner Graduate School of Public Service and the co-founder of the Coleridge Initiative, underscoring a critical issue: our data collection methods are woefully outdated.
Julia knows what she’s talking about, having worked with major statistical agencies like the U.S. Census Bureau to develop innovative approaches to data collection and analysis. She is also the author of "Democratizing Our Data: A Manifesto”.
Julia argues that the U.S. system for producing high-quality data is broken and in desperate need of change. "The economy has changed so rapidly over the past 20 years, and we're using a system that was developed a hundred years ago," Julia explains. This mismatch leads to significant gaps in our understanding of crucial economic trends, from employment patterns to the impact of emerging technologies like AI.
Julia's proposed solution? A fundamental rethinking of how we collect, analyze, and use data. She advocates for moving beyond traditional surveys to harness the power of administrative records and other modern data sources. This approach, she argues, could provide a more timely, actionable, and comprehensive picture of our workforce and economy.
Read on to learn about the limitations of current data collection methods, the potential of new approaches, and the implications for economic analysis and policy-making.
The limitations of current data collection methods
When it comes to measuring the impact of transformative technologies like AI on our workforce, our current methods are just not advanced enough. Julia points to a recent Census Bureau questionnaire as a prime example of this inadequacy.
"It is a questionnaire that is sent. They ask a respondent in a firm to answer, and it says, ‘Tell me about the impact of AI. In your firm, which could be a thousand people, has it increased employment, decreased employment, or had it about the same?’ How is one person going to be able to answer that?" Julia wonders.
This approach, she argues, is fundamentally flawed. It relies on a single individual to provide complex information about large organizations, often leading to inaccurate or incomplete data. As Julia puts it, "If I'm a survey respondent, I'm going to skip that question, which may be why the Census Bureau is only saying only 5% of firms are using AI because the human being that had to answer it skipped it."
The problem extends beyond just AI. Julia points out that our current measures of employment and unemployment, developed in the 1920s and 30s, are no longer adequate for capturing the complexities of today's labor market.
"For example, the unemployment measure is actively looking for work. In the survey, they call people on a phone for employment. They say, ‘Were you paid for at least one hour in the survey week? Are you in paid work for one hour in the survey week?’ Those are not measures of what's going on in the labor market, which is much more complex than that," Julia explains.
These outdated methods are leading to a fundamental disconnect between our economic statistics and the realities of our modern economy. As businesses and policymakers grapple with rapid technological change, this disconnect becomes increasingly problematic, hindering our ability to make informed decisions about workforce development and economic policy.
A new approach to data collection
Julia advocates for a radical shift in how we gather and analyze economic data. Instead of relying on traditional surveys, she proposes leveraging the wealth of information already being generated through various digital and administrative sources.
"You don't have to ask people the answers to questions. What you have to be able to do is to figure out what data there are to ask," Julia explains. She suggests starting with the fundamental questions we want to answer about the economy and workforce, then identifying existing data sources that can provide insights.
Julia gives a concrete example of how this approach could work in practice, particularly for understanding the spread of new technologies like AI across different industries:
"The way in which you build that system is you trace who is funded to do science in a particular area—quantum computing, AI, synthetic biology, or whatever. You could, it's not just the principal in this debate, it's the graduate students, undergraduate students, postdocs, clinical scientists. And then you trace how those people who've been trained to think about a problem, where are they being hired?"
This method provides a more accurate and timely picture of how innovations are diffusing through the economy. It captures the movement of skilled workers and the transmission of ideas, offering a deeper understanding of economic trends than traditional industry classifications.
Julia also emphasizes the importance of using administrative records, such as unemployment wage records, to gather information about workers' skills and employment patterns. This approach, she contends, is "a much more powerful and timely way of understanding what's going on in the economy than asking people the question."
By harnessing these rich data sources, Julia believes we can create a more comprehensive and nuanced picture of our workforce and economy, one that better reflects the complexities of our modern, innovation-driven world.
Towards a new data infrastructure
Julia's vision for revolutionizing data collection and analysis extends beyond just changing methodologies. She advocates for the creation of a new, independent entity to spearhead this transformation.
"What I had recommended in the book was that a federally funded research data center be set up," Julia explains. However, her thinking has evolved since writing the book. "It could be an independent think tank like the National Bureau of Economic Research," she suggests. This approach would provide more flexibility and robustness in addressing our data challenges.
Julia outlines three key elements for this proposed institute:
- Closer connection to data users: The institute would prioritize direct engagement with states, businesses, and workers to ensure the data collected meets their needs and reflects real-world conditions.
- Development of new data standards: The focus would be on creating innovative standards that capture the complexity of modern data types, ensuring the resulting products are trustworthy and reliable.
- Community-driven approach: Julia envisions a democratized process that's more inclusive than her initial recommendations, emphasizing a bottom-up approach driven by community input and needs.
Maintaining political neutrality is crucial in this endeavor, Julia emphasizes. "The absolutely critical thing to generate trust from a operational point of view is set up a governance structure that is clearly nonpartisan," she explains.
Julia cites examples like the National Bureau of Economic Research, which "does not set up policy recommendations. They provide the facts; they provide the answers." This approach, she argues, is essential for maintaining credibility and trust in the data produced.
By creating this new infrastructure, Julia envisions a future where our understanding of the workforce and economy is not only more accurate but also more responsive to the needs of businesses, workers, and policymakers alike.
Learning from other countries
In her quest to revolutionize data systems, Julia draws inspiration from successful models around the world, with a particular focus on New Zealand's approach.
"Bill English saw that data that had been built in at the beginning in New Zealand could be used to inform the decision making of the New Zealand government," Julia explains. This experience opened her eyes to the possibilities of a more integrated, comprehensive data system.
The New Zealand model, according to Julia, demonstrates how integrated data can effectively inform government decision-making.
She notes, "Bill was absolutely brilliant at figuring out how it could get integrated across the government. And now with the new government, it's turned into a Ministry of Social Investment and much of the decision-making has to be made on a cost-benefit calculus and they have integrated it into the entire decision-making of New Zealand policy."
Julia sees this as an aspirational model, stating, "We looked at it a lot when, when I was sitting on those White House committees, what New Zealand has done."
But New Zealand isn't the only country making strides in this area. Julia points to progress elsewhere: "I think there are other countries that are starting to head that way. I hate to say this, but the Australians. But yeah, I think in the UK, I sit on the advisory committee for the UK data service. They're moving it."
Julia emphasizes that this shift towards more modern, integrated data systems is inevitable: "This is inevitable. The survey is just too slow, too expensive, not granular enough, not timely enough. So really the question is not whether it's going to happen, but when and how and how strategic we can be."
She sees potential for quick returns with each new administration, suggesting that modernizing data infrastructure could be a priority area for improvement. By learning from these international examples, Julia believes the U.S. can leap forward in its approach to data collection and analysis, creating a system that's more responsive, accurate, and useful for all stakeholders.
Implications for economic analysis and policymaking
Julia raises significant concerns about the reliability of current economic data, particularly those used by investors and policymakers. When asked about the confidence we should have in data like GDP statistics and unemployment figures, her response is candid and sobering.
"I don't think we can be confident," Julia states. This lack of confidence, coming from an expert of her stature, underscores the urgency of the problem.
Julia points out that even the experts at the Bureau of Economic Analysis are aware of these limitations. She references a recent conference at Stanford, where officials from the Bureau admitted they're "really struggling with how to measure the changes in the economy."
The issues are multifaceted: "Part of it is they don't have the resources. They don't have a framework within which to really operate." She highlights that the current system of national accounts has been "stitched together, not purposefully, but kind of haphazardly in order to meet the [requirements]."
To drive home the magnitude of the problem, Julia cites Diane Coyle's work,
“GDP: A Brief but Affectionate History”, which suggests that "four-fifths of economic activity is not measured. 80% is not properly measured." This statistic is staggering, suggesting that our current economic measures might be missing more than they're capturing.
Julia emphasizes the need for a systematic, standardized approach to building a new data infrastructure. She sees this as a crucial step in addressing the current shortcomings in economic measurement and analysis.
A call for data infrastructure
Julia's insights reveal a system ill-equipped to capture the realities of today's economy. From outdated survey methods to fragmented collection processes, the challenges are numerous and significant. Yet, Julia's vision for the future is both ambitious and hopeful.
She advocates for a fundamental reimagining of our data infrastructure, one that leverages modern technologies, administrative records, and community-driven approaches. This new approach could transform how businesses operate, how policies are crafted, and how we measure economic progress.
The question now is not if we'll transform our data systems, but when and how quickly we can rise to meet this critical challenge. The future of economic understanding depends on robust data infrastructure.
This is based on an episode of Top Traders Unplugged, a bi-weekly podcast with the most interesting and experienced investors, economists, traders and thought leaders in the world. Sign up to our Newsletter or Subscribe on your preferred podcast platform so that you don’t miss out on future episodes.
Most Comprehensive Guide to the Best Investment Books of All Time

Most Comprehensive Guide to the Best Investment Books of All Time
Get the most comprehensive guide to over 400 of the BEST investment books, with insights, and learn from some of the wisest and most accomplished investors in the world. A collection of MUST READ books carefully selected for you. Get it now absolutely FREE!
Get Your FREE Guide HERE!