proomt

Search

Search posts, papers, and topics

All posts

Hacker News front page22 min readintermediate

Chat-based Large Language Models replicate the mechanisms of a psychic's con

Read on softwarecrisis.devHN 220·300 comments

Summary

The article argues that chat‑based LLMs don’t think; the perceived intelligence is a user‑side illusion created by validation statements that exploit the Forer effect, much like a psychic’s cold‑reading con. Understanding this bias helps engineers set realistic expectations for AI products.

  • LLMs are statistical token predictors, not reasoning agents.
  • User perception of AI intelligence stems from validation statements that feel personal.
  • The Forer/Barnum effect makes generic statements appear accurate to individuals.
  • Recognizing this bias prevents overpromising LLM capabilities and misuse.

Product engineers and AI researchers should care because the intelligence illusion can mislead users and shape unrealistic expectations for LLM systems.

5/10
llmaipsychologycognitive biascold reading

Related reading

  1. Verifiable Social Reasoning for LLM Assistants

    The paper introduces Fuse, a multi‑agent simulation that gives LLM assistants a verifiable ground‑truth task for social reasoning by hiding a target agent’s motive and letting a user‑mediated conversation infer it. Experiments on 12 LLMs show user mediation makes reasoning harder, models are biased by user framing, need more detail than humans, and longer chats don’t always help.

    Hugging Face Daily Papersarxiv.org1 minpaper

  2. US Military had close call after using AI for hallucinated intelligence report

    A US Special Operations analyst used an AI chatbot to augment an intelligence report that falsely claimed a Chinese vessel was carrying nuclear‑related cargo. The report triggered a near‑miss military operation that was aborted once the hallucination was discovered. The incident highlights the lack of verification standards and the risk of relying on AI‑generated intel for targeting decisions.

    Hacker News front pagecnn.com4 minHN513388

  3. Mind2Dialogue: Training Human-Aware Language Models by Simulating User Mental States

    Mind2Dialogue introduces a psychology‑guided simulator that generates user mental states and uses privileged distillation to train LLM assistants on Oracle responses that know those states. The resulting models improve personalization and theory‑of‑mind metrics by up to 41 percentage points versus standard instruction‑tuned baselines.

    Hugging Face Daily Papersarxiv.org2 minpaper

  4. I Don't Like LLMs

    Martin Fowler shares a personal, skeptical take on LLMs, noting their usefulness but criticizing their hallucinations, cultural bias, and the discomfort of interacting with them as if they were human.

    Lobstersmartinfowler.com2 minHN238271lobste.rs95

  6. Do LLMs Have the Memory of a Goldfish?

    The article explains that LLMs don’t have persistent personal memory; all “memory” is supplied by the surrounding application via the context window, summaries, or external storage. It outlines the distinction between trained weights, working‑memory (token context), and persistent application memory, shows how to construct API calls to preserve conversation state, and discusses the cost and laten…

    ByteByteGobytebytego.com12 min