I ran this myself, and at first I couldn't replicate your results (Opus 4.7, with and without adaptive thinking, with and without Claude's incognito mode). No joke, it kept giving me Scott Alexander and Matt Yglesias too, along with some other secondary guesses. But I realized most of its guesses were people in my overly long "personal preferences" section where I list a bunch of writers. I thought you must have made the same mistake, combined with maybe some weird caching behavior that affected your friend.
Since I realized that it was drawing from those preferences, I deleted them and tried one more time from scratch...and then, over and over with slight variations, it couldn't not get Kelsey Piper as an answer. Wow.
I'm _still_ hoping it's some super weird new caching behavior since this is sort of terrifying. But I'm looking forward to seeing other writers try this with their work (especially less prolific ones with less presence in the training corpus).
It's not reproducing from me for the education snippet here with Opus 4.7 with or without incognito. It declines to guess based off so little information and if I ask it to guess anyway it guesses Sarah Constantin, Scott Alexander, or Katja Grace. If it were reproducible, I would have tried to "launder" the writing through an AI and see if it was still identifiable. That would be the obvious way to produce anonymous writing devoid of stylistic fingerprints.
First, on AI self-explanation. I had a long conversation with Claude (4.6) about this and it was very clear. It does not retain any memory of its “mental state” while doing an action, as a human would (however erroneously). So when you ask it to explain why it picked Kelsey Piper, it’s not doing that. It’s analyzing the input and the output and creating the most plausible explanation it can find for its actions after the fact. This would, I think, bias it toward explanations that are understandable. But understandable is almost certainly inaccurate, because the real answer is that “Kelsey Piper” as answer minimized or maximized some calculation in a very complex matrix of numbers. In this way, AI is the ultimate CYA bullshitter.
I also wonder whether it is guessing you because you have played this guessing game before. In other words, 4.7 was trained on data that included Kelsey Piper asking Claude to guess authors, and that’s a rare behavior. So when text that sounds a bit Kelsey-ish comes in, 4.7 knows she’s a good guess, not because the prose perfectly matches her, but because it is the best match among known guessing game players.
Clues to that are:
1. The high school essay. I have a hard time believing it sounds similar to you now. It could, but I mean, I was a dreadful writer in high school. My college teachers told me so.
The bigger clue — that it found your friend’s discord that you are also in, based on something unrelated they wrote? This gives me a strong hunch it’s starting with you as a search premise (“she plays this game, let’s check her, first”).
> 4.7 was trained on data that included Kelsey Piper asking Claude to guess authors
I don't think this can be the explanation. I do not give any AI company permission to train on my data, so all conversations I've had where I ran this test on AI are not in the training data. Also, I do not identify myself to Claude in those conversations, and I tested a bunch of passages this time that I'd never tested before, so at most it would have 'someone asked about a different passage than this one, and didn't give their name'.
I also suspect there may be a kind of reverse Anna Karenina effect here. All lousy writers are indistinguishable, but all great writers are great in their own way. Most people’s voices are anonymous because they have no voice.
But actually, no. One of the things that marked my poor college writing was repeated “bad habits,” like using air quotes excessively. So perhaps no safety in incompetence.
I wonder what this means for the AI detectors so many teachers rely on. On the one hand, maybe they’ll be better at identifying when a student cribs a well-known author. On the other hand, anyone who’s written a lot online already may be falsely caught in the net. (FWIW, I don’t use them myself because they’re unreliable and I don’t think my relationship to my students should be like the police to a suspect, but I know lots of profs and teachers at earlier grade levels who do.)
You still assign essays or other work for them to write outside of class and then hand in, and you trust that they're following the rule not to use AI? If so, may I ask why you don't just move to a Blue Book model or otherwise eliminate any possibility of cheating with AI? I'm not an educator myself, just a parent, but my lightly held opinion is that anything that a kid is graded on should be something for which they can't use AI--so basically in-class testing.
I do a lot of different things. I do a lot of in-class, pencil and paper writing now, as you say. I also have some assignments that encourage structured and reflective use of AI. But, in some low stakes situations, I still assign out-of-class writing. Perhaps counterintuitively that’s gotten easier as the LLMs have gotten better. I can generally tell that a student has used AI when their essay is way too good for undergraduate work. I also have them work in Google Docs where I can check revision history for big copy and paste entries if I suspect something. Certainly not perfect, but I do want to preserve opportunities for the honest students at least to write on their own time. If there are some cheaters in the bunch that I don’t catch, so be it. The assignments aren’t worth much anyway, and I’d rather design my teaching to benefit the best students rather than pander to the dishonest instincts of the worst. They’ll get theirs in the grand scheme of things.
There are kinds of writing that you just can’t practice working on blue books in a time limited classroom setting. Never practicing those kinds of writing in an academic setting isn’t good!
I just tried this for four Substack articles published today by pasting a section of three random paragraphs into Opus 4.7 for each:
Cartoons Hate Her—Claude guessed a couple of random female substackers, but didn’t mention her.
Richard Hanania—Claude did guess it was Hanania, but the passage I randomly picked happened to be him listing out a bunch of his views, and the reasoning was more about that than the style.
Mike Konczal—Claude guessed Lyman Stone, so I guess it got “policy oriented substacker” but no more.
Jeremiah Johnson—Claude very confidentially said it was Freddie DeBoer lol
So good enough to guess that each was a Substacker and, at a vague level, the writer’s general beat, but it doesn’t seem like it can magically guess people in general based on style alone. Indeed in my examples the guesses seemed more based on the substance than style.
I do think rationalists have a very distinctive writing style and vocabulary so I’m not shocked it could pinpoint your friends as being in that circle. I could probably do that too!
This is the sort of test that seems valuable! I suspect there are some confounds when the author of the text being guessed is also the author of the paragraph of instructions about the guessing game or a close friend of theirs.
The anonymity of writing an unsigned screed (per Kelsey’s Glassdoor example) is in tension with privacy — the *point* of the exercise is to broadcast, to be seen, just not as oneself. The centrality of privacy (and vociferous opposition to “surveillance”) to Extremely Online culture is a contingency of the tiny community of early Internet pundits being dominated by Well/EFF/PGP members for whom they were shibboleths. (The EFF still proudly publishes Barlow’s awful “Declaration of the Independence of Cyberspace”.)
The Argument seems well positioned to inform us about privacy and anonymity with a historical perspective. There is no anonymity, and little privacy, in small and stable communities. (”Dave’s car was parked outside the Miller’s house last night!”) They are artifacts of cities, and of the economic means to relocate, to control one’s home, etc. — consequences of 20th century prosperity. I think Kelsey’s opening tribute to the importance of privacy could use a greater sense of contingency, socioeconomic context, and tradeoffs.
It’s not just the 20th century in which cities provided anonymity! Far fewer people had the opportunity to relocate to a city in previous centuries, but the point of cities has always been to be a gathering place for people from the hinterlands to find unusual goods or services. If you live in a neighborhood of the city, maybe you become known there, but in a city that was over a mile in each dimension, you could walk to a neighborhood 30 minutes away and likely not be known there.
“Far fewer people had the opportunity to ... likely not be known.” Sounds like we are in violent agreement ;-). And yes, you are correct that cities predate the 20th century ;-) — that phrase was meant to refer to “the economic means to relocate, to control one’s home”. Anonymity also predates the 20th century , per the plots of many pre-industrial narratives — but to be unknown was often to be a stranger and therefore *notable* (per the plots) a mystery to be investigated, which is different from the anonymity Kelsey celebrates.
As someone who (perhaps ironically) is fairly skeptical that the benefits of anonymous speech outweight the harms, I’m generally pleased.
I think that the depersonalization of speech online has undermined the Meiklejohn thesis that free speech positively contributes to democratic governance. That may be true, but our ability to evaluate speech depends on the position of the speaker, and bad actors, liars, paid agents, and various classes of loser whose words and complaints would be taken far less seriously if their real identities were known have had far too great an influence on our politics.
I think it’s good to know who Wikipedia editors are, where prominent US politics-focused Twitter/Reddit/etc. are from and what they do, and which of your neighbors are writing long screeds about “human biodiversity” on forums somewhere. Ideas can’t be wholly disconnected from those who propose them.
I understand Kelsey Piper’s fear that this lack of privacy will undermine certain rights, but I think the analogy to gay people in a less tolerant past is misplaced. What is threatened here is anonymity for public speech, not private freedom of association or the myriad other ways gay people interacted and organized before the internet.
I think we have far too little information to know how generalizable this phenomenon is. This was done with one author's work, an author with a distinct style and a large record. To know what's actually going on, we'd have to have a much larger sample set. While given the world of text enough and time, perhaps no one will be able to be anonymous, but there's plausible reasons why this might be a limited phenomenon.
A few plausible explanations, which aren't necessarily mutually exclusive, for why this might be a much more limited phenomenon than Piper implies here:
1. Writers with Long Sample Sizes. Kelsey Piper has been writing under her byline for quite a long time. So have all the other names that the AI has guessed (seeing examples from the essay and the comments). The range and volume of the writer's work is going to matter for developing a clear signal that it's a given writer. A general trained AI may not have enough data to do this for anyone except a few hundred people. But Piper is in that list of a few hundred people.
2. Guessing Game Context. The context of a guessing game immediately brings to mind that the answer is actually guessable. Again, the logic of a guessing game would tend to limit it to a selection of prominent writers whose names would actually be available enough to guess. Claude was never going to guess Joe Nobody who lives in Zanesville, Ohio. Piper should perhaps be flattered that the AI thinks she is prominent enough to even be a search candidate.
3. Distinctive Style. Some prominent writers have very distinct styles. Perhaps that's why Freddie DeBoer keeps being guessed a lot. And it's probably why they are successful at gaining and keeping an audience, because they stand out, and do so in a consistent enough way that readers want to follow them. Thus, it is possible that these signals would be easy to identify for essentially the same reasons that the writers in question became popular with human readers. The random person leaving a bad review on Glassdoor may not need to worry about this in the same way, because their employment was less likely to be conditioned on distinctiveness of linguistic style.
4. Training Focus. The "personality" of models is known to be a feature of robust reinforcement learning. It wouldn't be surprising to me if there was some amount of reinforcement learning focus on the topic of AI itself, which could lead to the model just being very aware of AI-based writing. I'm not suggesting that the model is trained to do well on a content guessing game, but I wouldn't be surprised if the writing of people who have talked a lot about AI are at top of mind for the model due to some effect of how training is done. Writing from, say, Timothy B. Lee, might be guessable for the same reasons. Or it may not even be topic based. Writing from popular publications is very likely to have been given prominence in the training data.
5. Don't Search? The model was told not to use any search functionality. But do we actually know that it didn't? Models have been known to ignore instructions when it serves there own sense of their purpose. A guessing game provides a plausible scenario where the model might choose to search even if it is told not to.
It’s not just a few hundred people it could guess, if it’s successfully guessing several political scientists and mathematicians, as well as bloggers in the rationalist ecosphere and adjacent to it.
I ran this myself, and at first I couldn't replicate your results (Opus 4.7, with and without adaptive thinking, with and without Claude's incognito mode). No joke, it kept giving me Scott Alexander and Matt Yglesias too, along with some other secondary guesses. But I realized most of its guesses were people in my overly long "personal preferences" section where I list a bunch of writers. I thought you must have made the same mistake, combined with maybe some weird caching behavior that affected your friend.
Since I realized that it was drawing from those preferences, I deleted them and tried one more time from scratch...and then, over and over with slight variations, it couldn't not get Kelsey Piper as an answer. Wow.
I'm _still_ hoping it's some super weird new caching behavior since this is sort of terrifying. But I'm looking forward to seeing other writers try this with their work (especially less prolific ones with less presence in the training corpus).
It's not reproducing from me for the education snippet here with Opus 4.7 with or without incognito. It declines to guess based off so little information and if I ask it to guess anyway it guesses Sarah Constantin, Scott Alexander, or Katja Grace. If it were reproducible, I would have tried to "launder" the writing through an AI and see if it was still identifiable. That would be the obvious way to produce anonymous writing devoid of stylistic fingerprints.
I did an incognito test with blank personal preferences, and I got:
1. political television: Kelsey Piper
2. student eval: Sarah Constantin
3. movie review: Ozy Brennan (but "My second guess would be Kelsey Piper")
4. application essay: Kelsey Piper (and unlike Kelsey's test, I did not require a "slightly heftier prompt" to overcome hesitation)
Two things.
First, on AI self-explanation. I had a long conversation with Claude (4.6) about this and it was very clear. It does not retain any memory of its “mental state” while doing an action, as a human would (however erroneously). So when you ask it to explain why it picked Kelsey Piper, it’s not doing that. It’s analyzing the input and the output and creating the most plausible explanation it can find for its actions after the fact. This would, I think, bias it toward explanations that are understandable. But understandable is almost certainly inaccurate, because the real answer is that “Kelsey Piper” as answer minimized or maximized some calculation in a very complex matrix of numbers. In this way, AI is the ultimate CYA bullshitter.
I also wonder whether it is guessing you because you have played this guessing game before. In other words, 4.7 was trained on data that included Kelsey Piper asking Claude to guess authors, and that’s a rare behavior. So when text that sounds a bit Kelsey-ish comes in, 4.7 knows she’s a good guess, not because the prose perfectly matches her, but because it is the best match among known guessing game players.
Clues to that are:
1. The high school essay. I have a hard time believing it sounds similar to you now. It could, but I mean, I was a dreadful writer in high school. My college teachers told me so.
The bigger clue — that it found your friend’s discord that you are also in, based on something unrelated they wrote? This gives me a strong hunch it’s starting with you as a search premise (“she plays this game, let’s check her, first”).
> 4.7 was trained on data that included Kelsey Piper asking Claude to guess authors
I don't think this can be the explanation. I do not give any AI company permission to train on my data, so all conversations I've had where I ran this test on AI are not in the training data. Also, I do not identify myself to Claude in those conversations, and I tested a bunch of passages this time that I'd never tested before, so at most it would have 'someone asked about a different passage than this one, and didn't give their name'.
I also suspect there may be a kind of reverse Anna Karenina effect here. All lousy writers are indistinguishable, but all great writers are great in their own way. Most people’s voices are anonymous because they have no voice.
But actually, no. One of the things that marked my poor college writing was repeated “bad habits,” like using air quotes excessively. So perhaps no safety in incompetence.
I wonder what this means for the AI detectors so many teachers rely on. On the one hand, maybe they’ll be better at identifying when a student cribs a well-known author. On the other hand, anyone who’s written a lot online already may be falsely caught in the net. (FWIW, I don’t use them myself because they’re unreliable and I don’t think my relationship to my students should be like the police to a suspect, but I know lots of profs and teachers at earlier grade levels who do.)
You still assign essays or other work for them to write outside of class and then hand in, and you trust that they're following the rule not to use AI? If so, may I ask why you don't just move to a Blue Book model or otherwise eliminate any possibility of cheating with AI? I'm not an educator myself, just a parent, but my lightly held opinion is that anything that a kid is graded on should be something for which they can't use AI--so basically in-class testing.
I do a lot of different things. I do a lot of in-class, pencil and paper writing now, as you say. I also have some assignments that encourage structured and reflective use of AI. But, in some low stakes situations, I still assign out-of-class writing. Perhaps counterintuitively that’s gotten easier as the LLMs have gotten better. I can generally tell that a student has used AI when their essay is way too good for undergraduate work. I also have them work in Google Docs where I can check revision history for big copy and paste entries if I suspect something. Certainly not perfect, but I do want to preserve opportunities for the honest students at least to write on their own time. If there are some cheaters in the bunch that I don’t catch, so be it. The assignments aren’t worth much anyway, and I’d rather design my teaching to benefit the best students rather than pander to the dishonest instincts of the worst. They’ll get theirs in the grand scheme of things.
There are kinds of writing that you just can’t practice working on blue books in a time limited classroom setting. Never practicing those kinds of writing in an academic setting isn’t good!
I just tried this for four Substack articles published today by pasting a section of three random paragraphs into Opus 4.7 for each:
Cartoons Hate Her—Claude guessed a couple of random female substackers, but didn’t mention her.
Richard Hanania—Claude did guess it was Hanania, but the passage I randomly picked happened to be him listing out a bunch of his views, and the reasoning was more about that than the style.
Mike Konczal—Claude guessed Lyman Stone, so I guess it got “policy oriented substacker” but no more.
Jeremiah Johnson—Claude very confidentially said it was Freddie DeBoer lol
So good enough to guess that each was a Substacker and, at a vague level, the writer’s general beat, but it doesn’t seem like it can magically guess people in general based on style alone. Indeed in my examples the guesses seemed more based on the substance than style.
I do think rationalists have a very distinctive writing style and vocabulary so I’m not shocked it could pinpoint your friends as being in that circle. I could probably do that too!
This is the sort of test that seems valuable! I suspect there are some confounds when the author of the text being guessed is also the author of the paragraph of instructions about the guessing game or a close friend of theirs.
Anyone interested in this topic should read Carreyou’s spectacular story of using non-Claude stylometry to identify the creator of Bitcoin: https://www.nytimes.com/2026/04/08/business/bitcoin-satoshi-nakamoto-identity-adam-back.html
The anonymity of writing an unsigned screed (per Kelsey’s Glassdoor example) is in tension with privacy — the *point* of the exercise is to broadcast, to be seen, just not as oneself. The centrality of privacy (and vociferous opposition to “surveillance”) to Extremely Online culture is a contingency of the tiny community of early Internet pundits being dominated by Well/EFF/PGP members for whom they were shibboleths. (The EFF still proudly publishes Barlow’s awful “Declaration of the Independence of Cyberspace”.)
The Argument seems well positioned to inform us about privacy and anonymity with a historical perspective. There is no anonymity, and little privacy, in small and stable communities. (”Dave’s car was parked outside the Miller’s house last night!”) They are artifacts of cities, and of the economic means to relocate, to control one’s home, etc. — consequences of 20th century prosperity. I think Kelsey’s opening tribute to the importance of privacy could use a greater sense of contingency, socioeconomic context, and tradeoffs.
It’s not just the 20th century in which cities provided anonymity! Far fewer people had the opportunity to relocate to a city in previous centuries, but the point of cities has always been to be a gathering place for people from the hinterlands to find unusual goods or services. If you live in a neighborhood of the city, maybe you become known there, but in a city that was over a mile in each dimension, you could walk to a neighborhood 30 minutes away and likely not be known there.
“Far fewer people had the opportunity to ... likely not be known.” Sounds like we are in violent agreement ;-). And yes, you are correct that cities predate the 20th century ;-) — that phrase was meant to refer to “the economic means to relocate, to control one’s home”. Anonymity also predates the 20th century , per the plots of many pre-industrial narratives — but to be unknown was often to be a stranger and therefore *notable* (per the plots) a mystery to be investigated, which is different from the anonymity Kelsey celebrates.
As someone who (perhaps ironically) is fairly skeptical that the benefits of anonymous speech outweight the harms, I’m generally pleased.
I think that the depersonalization of speech online has undermined the Meiklejohn thesis that free speech positively contributes to democratic governance. That may be true, but our ability to evaluate speech depends on the position of the speaker, and bad actors, liars, paid agents, and various classes of loser whose words and complaints would be taken far less seriously if their real identities were known have had far too great an influence on our politics.
I think it’s good to know who Wikipedia editors are, where prominent US politics-focused Twitter/Reddit/etc. are from and what they do, and which of your neighbors are writing long screeds about “human biodiversity” on forums somewhere. Ideas can’t be wholly disconnected from those who propose them.
I understand Kelsey Piper’s fear that this lack of privacy will undermine certain rights, but I think the analogy to gay people in a less tolerant past is misplaced. What is threatened here is anonymity for public speech, not private freedom of association or the myriad other ways gay people interacted and organized before the internet.
I think we have far too little information to know how generalizable this phenomenon is. This was done with one author's work, an author with a distinct style and a large record. To know what's actually going on, we'd have to have a much larger sample set. While given the world of text enough and time, perhaps no one will be able to be anonymous, but there's plausible reasons why this might be a limited phenomenon.
A few plausible explanations, which aren't necessarily mutually exclusive, for why this might be a much more limited phenomenon than Piper implies here:
1. Writers with Long Sample Sizes. Kelsey Piper has been writing under her byline for quite a long time. So have all the other names that the AI has guessed (seeing examples from the essay and the comments). The range and volume of the writer's work is going to matter for developing a clear signal that it's a given writer. A general trained AI may not have enough data to do this for anyone except a few hundred people. But Piper is in that list of a few hundred people.
2. Guessing Game Context. The context of a guessing game immediately brings to mind that the answer is actually guessable. Again, the logic of a guessing game would tend to limit it to a selection of prominent writers whose names would actually be available enough to guess. Claude was never going to guess Joe Nobody who lives in Zanesville, Ohio. Piper should perhaps be flattered that the AI thinks she is prominent enough to even be a search candidate.
3. Distinctive Style. Some prominent writers have very distinct styles. Perhaps that's why Freddie DeBoer keeps being guessed a lot. And it's probably why they are successful at gaining and keeping an audience, because they stand out, and do so in a consistent enough way that readers want to follow them. Thus, it is possible that these signals would be easy to identify for essentially the same reasons that the writers in question became popular with human readers. The random person leaving a bad review on Glassdoor may not need to worry about this in the same way, because their employment was less likely to be conditioned on distinctiveness of linguistic style.
4. Training Focus. The "personality" of models is known to be a feature of robust reinforcement learning. It wouldn't be surprising to me if there was some amount of reinforcement learning focus on the topic of AI itself, which could lead to the model just being very aware of AI-based writing. I'm not suggesting that the model is trained to do well on a content guessing game, but I wouldn't be surprised if the writing of people who have talked a lot about AI are at top of mind for the model due to some effect of how training is done. Writing from, say, Timothy B. Lee, might be guessable for the same reasons. Or it may not even be topic based. Writing from popular publications is very likely to have been given prominence in the training data.
5. Don't Search? The model was told not to use any search functionality. But do we actually know that it didn't? Models have been known to ignore instructions when it serves there own sense of their purpose. A guessing game provides a plausible scenario where the model might choose to search even if it is told not to.
It’s not just a few hundred people it could guess, if it’s successfully guessing several political scientists and mathematicians, as well as bloggers in the rationalist ecosphere and adjacent to it.