MIT Technology Review is reporting that AI companies like Anthropic and OpenAI regularly publish reports on how people use their products, but these reports often present a limited view, according to AI researchers. Anka Reuel, a Computer Science PhD candidate at the Stanford Trustworthy AI Research (STAIR) Lab, said there is no independent source to corroborate the companies' data.
Reuel co-leads a new research project called the AI Observatory, a public platform designed to aggregate and analyze real AI conversations collected with user consent. The Observatory aims to provide independent information for researchers and policymakers to assess generative AI use, as stakeholders are currently making significant decisions based on limited data.
The AI Observatory found that AI use varies significantly across models and has changed over time. Its research revealed many more sensitive behaviors than typically captured in major AI company reports, which tend to focus on work-related uses. For example, when the AI Observatory team applied Anthropic's methods to their dataset, nearly half of the conversations, 48%, would have been filtered out. These non-work-related conversations were more likely to include health and relationships, adult or illicit topics, harassment and hate, and sexual content.
The Observatory also found distinct usage patterns for different models. Users frequently turned to Grok and Gemini for information retrieval, with Grok being particularly popular for news and politics, where misinformation tended to concentrate. Anthropic was more often used for coding, Gemini for social and roleplay, and ChatGPT for homework assistance. Conversations within one of the study's largest datasets, WildChat, became longer and more elaborate over time, with increased small talk suggesting growing AI companionship, while sensitive uses decreased, possibly indicating more effective platform safeguards.
Shayne Longpre, a recent PhD graduate from the MIT Media Lab who co-led the research, said no single company report tells the whole story. The AI Observatory aggregated 24,521 conversations from 5,000 users interacting with 52 different models between 2023 and 2025. While this dataset is smaller than those used by major labs, researchers caution that its findings, drawn from voluntarily provided sources, may still underrepresent sensitive uses. However, the project broadens access for the research community, as AI companies typically do not share their chat data for independent analysis.
Full Article: We still don’t know how people are really using AI