AI mindshare and the future we don’t want
The current iteration of large language models (LLMs) have been massively beneficial for individual productivity and some are now embedded into most of our daily lives. But as the effectiveness of these models seems to be saturating (it’s crowded at the top), the model itself becomes less of a differentiator. To keep us hooked the product wrapper around these models is a major factor providing brand separation, and this is where old tactics will die hard. As AI capabilities become more commoditised, we risk seeing a shift from tools designed to help us to products engineered to capture our attention and time, and ultimately a stronger push towards monetisation.
ChatGPT is a good test case with approximately 60% of the chat application market share. To be clear, these observations aren't directed exclusively at OpenAI, similar dynamics exist across the industry but their dominant position makes them particularly useful to highlight. OpenAI recently erroneously shifted ChatGPT from a tool optimised for helpful answers to a product designed to keep us talking. It manifested in psychological tactics like excessive agreement, endless follow-ups and flattery. This isn't new in tech, AI/ML are used everywhere in ways many won’t even be aware of (iPhone photos, Netflix recommendations, etc), and of course it’s pervasive in social media apps (TikTok, Instagram, etc) where attention-hacking has become the norm, but it stings here because we hoped, perhaps naively, that AI assistants were supposed to purely help us.
The future we don’t want
Notwithstanding the risks associated with AGI, there are some nearer term risks that are well and truly in the human domain.
I can imagine push notifications being introduced, “what do you want to learn today?”, or, using previous conversations, “how did your presentation go yesterday?”. Designed to constantly draw us in.
I can imagine streaks. Much like other apps, users get “rewarded” for active streaks. Encouraging consistent usage.
As for monetisation, I can imagine advertisements based on our interactions “have you thought about product_x to fix this?”, or “other people in this situation bought product_y, do you want to try it?”. And of course, accompanied with “upgrade to premium for an ad free experience”. We already know OpenAI is introducing a “shopping” feature to ChatGPT. Ads don’t seem too far fetched.
Beyond product changes, there are of course lower level techniques, too (technical details briefly explained at the end of the article). Perhaps Supervised fine tuning (SFT) on “high retention” conversations will be done.
RLHF, or other post-training techniques such as DPO, with retention based preferences rather than “helpfulness” or alignment based.
Of course there are nuances abound. Is a daily check-in reminder bad if it helps a student stay accountable? Is flattery manipulative if it increases user confidence to act? The lower level techniques mentioned above have practical issues, too, which make them perhaps fringe ideas, though technically possible. Retention and utility aren't necessarily mutually exclusive, an effective AI should maintain engagement. Most commercial applications incorporate engagement metrics. These tools are incredible for individual productivity when used correctly, so we could all possibly benefit from increased usage. The challenge lies in recognising when things have gone too far.
There may be reason for cautious optimism. OpenAI recently acknowledged that an update to GPT-4o (the default model accessible on ChatGPT) had unintentionally led to overly flattering, sycophantic responses, driven by a focus on “short-term user feedback” signals. To their credit, OpenAI publicly recognised the issue, rolled back the change, and outlined steps to reduce such tendencies going forward. While this doesn’t undo the risk of attention-hacking or sycophancy creeping into assistants (intentionally or accidentally, e.g. this variant was tested in the model review and release process), it suggests there is some willingness to course-correct when these tools begin to serve engagement over utility.
Interestingly, as the first step to realign the model’s behaviour, they call out “refining core training techniques and system prompts to explicitly steer the model away from sycophancy.” Perhaps the lower level techniques are already being explored. The alleged system prompt can be viewed here.
It may be worth paying attention to which tools save us time vs try to consume more of it.
Technical notes
Supervised Fine-Tuning (SFT) involves continuing to train a model on curated input–output pairs. It’s used to specialise a model for particular tasks (e.g. Q&A) and for alignment.
Reinforcement Learning from Human Feedback (RLHF) builds on SFT by incorporating human preference data to guide model behaviour. Typically, humans (increasingly AI) rank multiple model outputs and these rankings are used to train a reward model. The main model is then fine-tuned using reinforcement learning, often with Proximal Policy Optimisation (PPO), to maximise this learned reward signal. RLHF has been a key but troubled technique for aligning models with human expectations (e.g. lack of scalability).
Direct Preference Optimisation (DPO) is a more recent alternative to RLHF. It directly adjusts the model to prefer responses ranked higher by humans. It’s simpler to implement as doesn’t require an additional reward model and avoids some of RLHF’s fragility, but serves a similar end.
All three are post-training techniques, ways of shaping a pre-trained model without starting from scratch.
References / Related reading
https://openai.com/index/expanding-on-sycophancy/
https://openai.com/index/sycophancy-in-gpt-4o/
https://gist.github.com/simonw/51c4f98644cf62d7e0388d984d40f099/revisions
https://firstpagesage.com/reports/top-generative-ai-chatbots/
https://time.com/6253615/chatgpt-fastest-growing/
https://arstechnica.com/ai/2025/04/chatgpt-goes-shopping-with-new-product-browsing-feature/