Buyer Beware: An Audit Framework for the Market Data Your Firm Relies On
Photo: Joe Haupt from USA, CC BY-SA 2.0, via Wikimedia Commons
A mid-sized E&P will spend more this year on market intelligence subscriptions than on geotechnical sampling, yet almost no one in the C-suite can name the sample size behind the headline numbers in the quarterly report they cite to their board. The asymmetry is striking: we audit reserves with SEC-grade rigor and accept market data on faith.
This is not an indictment of data providers — most do honest work under real constraints. It is an argument that consumers of intelligence should behave like consumers. Below is the framework we use at PetroMar Survey when evaluating sources, including our own, followed by a candid assessment of the major categories of providers.
The Four Tests
1. Sample composition. Who actually answers the survey or populates the dataset? A producer sentiment index weighted toward financiers in Houston towers tells you something different than one weighted toward operations managers in Midland. Ask for the respondent breakdown. Reputable providers publish it; the rest deflect.
2. Methodological transparency. Can you obtain the questionnaire, the weighting scheme, and the revision history? Sources that publish methodology appendices — as the EIA and most academic-adjacent surveys do — earn a structural credibility premium over those that treat method as proprietary mystery.
3. Track record, honestly scored. Any provider can claim accuracy. Few publish a running ledger of past calls against outcomes. Demand dated predictions and score them yourself.
4. Incentive disclosure. Does the provider sell advisory services, take trading positions, or accept sponsorship from the companies it covers? Conflicts are not disqualifying, but undisclosed ones are.
Grading the Landscape
Government statistical agencies (EIA, BLS): The gold standard on transparency and methodology, with the caveat that weekly and monthly releases are inherently lagging. Grade: A on rigor, C on timeliness.
Established trade press surveys and indices: Strong access and long institutional memory, but samples skew toward larger, publicly reporting companies. Small operators are chronically underrepresented. Grade: B+.
Sell-side consensus polls: Fast, cheap, and widely quoted — and structurally biased toward the trading calendar rather than the capital cycle. Useful as a contrarian reference; dangerous as a planning input. Grade: C+.
Boutique operator surveys: Often the best-differentiated data available, but quality varies enormously. Apply the four tests without mercy. Grade: B, with high variance.
Proprietary AI-scraped datasets: The fastest-growing category and the least audited. Web-scraped signals can be genuinely useful, but few buyers understand the survivorship bias baked into any dataset assembled from what companies choose to publish. Grade: Incomplete.
A Renewal-Season Checklist
Before your next subscription renewal, ask the provider three questions: What is your respondent or source composition, and how has it changed in three years? Where can I review your methodology and revision history? And what calls have you made that turned out to be wrong?
The third question is the most revealing. Providers with nothing to hide will answer it readily. The industry's decisions are too consequential — and its margins too thin — to keep buying intelligence nobody has audited.
