During the height of Truman Capote’s career, he dined and partied with a group of New York socialites who, in his words, were “swans.” His favorite of these women was Barbara “Babe” Paley, a stylish woman who regularly appeared on “best dressed” lists and was married to CBS founder William Paley. Capote entertained Babe and the other swans with his humor and stories, and they in turn told him about their own lives and the lives of others in their social set.

That changed in 1975, when Capote published a story in Esquire called “La Côte Basque, 1965.” The fictional story is about a socialite and her male friend gossiping over lunch at La Côte Basque, the French New York City restaurant where Capote and the swans often dined in real life. Capote drew from real life to compose the scandalous tales. And even though he changed the names of some (but not all) of the people who appeared in his story, those people still knew he was talking about them.

“La Côte Basque, 1965” turned many New York socialites against Capote, and that included his favorite swan, Babe Paley. After reading the story, which contained a fictionalized account of her husband cheating on her, Paley stopped talking to Capote. The social backlash surprised the famous writer, who didn’t seem to anticipate that the people he wrote about would recognize themselves in his fiction.

In the late 1960s and early 1970s, Truman Capote was one of the most famous writers in the United States. His 1958 novella Breakfast at Tiffany’s and the 1961 movie adaptation had brought success, but it was his 1966 “non-fiction novel” In Cold Blood that really turned him into a nationally-recognized celebrity. The year In Cold Blood came out, he signed a contract with Random House to write what he hoped would be his best novel: Answered Prayers.

By 1975, Capote still hadn’t finished the novel, but he decided to publish a couple of chapters in Esquire. The first chapter, “Mojave,” appeared in the magazine’s June issue. Encouraged by the positive reaction, he published another chapter, “La Côte Basque, 1965,” in the November issue. This second story focused on members of New York’s high society, including some of his swans.

“Truman Capote invented this term ‘swans’ for a very specific group of women,” says Laurence Leamer, author of Capote’s Women: A True Story of Love, Betrayal, and a Swan Song for an Era. “They all were beautiful, they all were rich, they were elegant and they had a certain social savvy and wit about them.” The swans included Babe Paley, CZ Guest and Nancy “Slim” Keith, an American who married into British royalty.

“La Côte Basque, 1965” centers on Lady Ina Coolbirth, a character based on Slim Keith. Over lunch, Coolbirth tells her companion about Ann Hopkins, a woman who supposedly murdered her husband and made it look like an accident.

This was a thinly-veiled reference to Ann Woodward, a real-life socialite who fatally shot her husband in 1955. Woodward claimed she shot him because she’d mistaken him for a burglar, and never faced legal charges. In Capote’s fictional story, Coolbirth speculates that Ann shot her husband on purpose.

The story also contains descriptions of infidelity that rung familiar to tales Capote had heard about his friends. In one passage, a famous TV personality answers a phone call from his wife while in bed with another woman, and even puts the other woman on the phone with his wife. In another, a man cheats on his wife with the wife of a former governor, and is left having to clean his bedsheets.

Even though Capote didn’t use their names in the story, the first tale echoed the real life of Johnny Carson and his ex-wife Joanne, who was a friend of Capote’s. The second seemed to reference a story told to Truman about the husband of Capote’s favorite swan, Babe. When reading “La Côte Basque, 1965” before publication, the writer Gerald Clarke—Capote’s friend and biographer—questioned whether including veiled references to real people was a good idea.

“I said to him, ‘You know Truman, they’re not going to be very happy with this,’” Clarke says. “And he said, ‘Nah, they’re too dumb, they won’t know who they are.’”

But they did. Slim Keith recognized herself in Lady Ina Coolbirth and, disliking the portrayal, cut Capote out of her life.

“I read it, and I was absolutely horrified,” she told writer George Plimpton. “The story about the sheets, the story about Ann Woodward…There was no question in anybody’s mind who it was.”

Babe Paley recognized the portrayal of her husband and refused to see Capote again. Babe was battling lung cancer at the time, and she died three years later without reconciling with Capote. Although it’s unclear if Ann Woodward ever read the story or heard about its contents, her suicide shortly before its publication fueled rumors that she knew it referenced her.

Gloria Vanderbilt, one of the socialites who appeared in “La Côte Basque, 1965” under her real name, also had a falling out with Capote. In the story, Capote emphasized how many marriages she and actress Carol Matthau (another friend of his) had had by listing all of their previous last names, and included a scene in which Vanderbilt fails to recognize one of her former husbands.

Not everyone was angry about “La Côte Basque, 1965.” CZ Guest, who wasn’t in the story, maintained her friendship with Capote. So did Jackie Kennedy Onassis’ sister Lee Radziwill, whom Capote mentioned in the story by name (he painted a flattering portrait of Radziwill and a less flattering one of the former first lady). Joanne Carson, who did have an embarrassing story appear in “La Côte Basque, 1965,” continued to be friends with Capote as well.

However, many socialites who didn’t even show up in the story turned against Capote as a way of taking sides. Women’s Wear Daily began referring to the writer as the “Tiny Terror,” and New York magazine published a cover illustration portraying Capote as a dog biting a socialite’s hand under the headline “Capote Bites The Hands That Fed Him.”

“It became kind of a cult in New York among these people to denigrate Truman Capote,” Clarke says. He recalls waiting for Capote at a restaurant one day while socialite Jerry Zipkin, who “was not mentioned in the story at all,” sat at a nearby table. “When Truman came in to sit down with me, Jerry Zipkin ostentatiously got up and asked the head waiter for another table.”

Capote had already been struggling with alcohol and drug use before 1975, and the social fallout from “La Côte Basque, 1965” likely took a toll on him. He died in 1984 without finishing his long-awaited novel Answered Prayers. After his death, publishers released an unfinished version of the novel that included “La Côte Basque, 1965.”