Fidel Castro had previously visited New York in April 1959, just four months after he led his victorious guerilla army into Havana and took charge of Cuba. The trip was part of Castro’s victory lap after toppling the dictatorship of Fulgencio Batista, and he’d made the most of it by hiring a top public relations firm and touring the city with all the swagger of a rock star. News cameras had stalked the young revolutionary as he held babies, ate hotdogs and tossed peanuts to elephants at the Bronx Zoo. At one photo-op, he was pictured next to a group of American schoolchildren wearing fake Castro-style beards. There had been a few whispers about his suspected political leanings—he still hadn’t declared himself a communist—but many reporters were taken in by his fiery speeches and rugged military uniforms.

The mood was not so lighthearted when Fidel returned to New York in 1960 for the United Nations General Assembly. By then, Castro had nationalized U.S. business interests in Cuba, banned land ownership by foreigners and cozied up to the Soviet Union. Fearing Cuba was creeping toward communism, the United States had issued blanket sanctions including cutting sugar imports and restricting petroleum sales. President Dwight D. Eisenhower had even approved a secret plan to overthrow Castro’s regime.

The tensions appeared to weigh on Castro as he stepped onto the tarmac at Idlewild Airport (now JFK) on September 18. He was clad in the same olive fatigues he’d worn during his cheery 1959 trip, but this time he hardly cracked a smile as he greeted journalists and took in the sea of gathered supporters. The cigar-chomping leader seemed particularly annoyed by the large American security detail assigned to him. Unlike his previous visit, when he’d had free reign to wander, a suspicious U.S. State Department had now restricted his travel to the island of Manhattan.

After saying he would save any remarks for the U.N. meeting, Castro headed for the plush Shelburne Hotel in Midtown and disappeared into his rooms. The controversy began just a few hours later, when he stormed into the hotel’s lobby and announced that he was leaving because of unfair treatment. According to Castro, the Shelburne’s management had asked for an “unacceptable” cash advance of $10,000—allegedly to cover any damage his Cuban delegation might cause to their rooms. Convinced he was being harassed on the orders of the U.S. government, Castro immediately drove to the United Nations to complain. He even threatened to pitch camp in Central Park if need be. “We are mountain people,” he told reporters. “We are used to sleeping in the open air.”

Ignoring an offer of free lodgings at the nearby Commodore, Castro and his delegation eventually cruised to Harlem and checked into the Hotel Theresa, a decaying inn that doubled as a meeting place for New York’s African American politicians. Critics labeled the switch a publicity stunt, but Castro claimed to feel more at home among Harlem’s majority black population. He spent his first night giving exclusive interviews to African American newspapers and welcoming activist Malcolm X into his suite. A more controversial caller appeared the following morning, when Soviet leader Nikita Khrushchev strode up to the Theresa for his first ever face-to-face meeting with Castro. Afterwards, Khrushchev assured associates that the Cuban would turn communist, but added that he was “like a young horse that hasn’t been broken. He needs some training.”

A Cuban flag was soon hoisted above the Theresa, and the streets outside filled with thousands of curiosity seekers and journalists. Pro- and anti-Castro protestors came in droves, occasionally tossing eggs at one another or duking it out with fists and baseball bats. Tabloid papers also began printing wild—and most likely fictional—stories about the Cuban delegation’s rowdy behavior. There were rumors of prostitutes plying their trade in the Theresa, and a myth was born that Castro had been kicked out of the Shelburne for keeping live chickens in his hotel room.

Outside of occasional forays to the floor of the U.N., Castro spent the next few days laying low in the Theresa and rubbing elbows with the likes of poet Langston Hughes and Beat writer Allen Ginsberg (who supposedly asked about the Cuban Revolution’s stance on marijuana). When President Eisenhower excluded him from a September 22 luncheon for Latin American leaders, Castro held his own banquet in the Theresa’s ballroom and invited “the poor and humble people of Harlem” to join him.

The Cuban premier’s antics struck a chord with many Harlem locals, but some black commenters considered them nothing more than propaganda. “The Negro (U.S.) gets a kick, a big bang out of any embarrassment his Government suffers because of the race problem,” read an editorial in the New York Amsterdam News, a prominent African American paper. “But Castro’s stomping at the Theresa is not the answer to his problem.” The Baptist Ministers’ Conference of Greater New York, meanwhile, issued a letter denouncing “any attempts by Fidel Castro to make the Harlem community a battleground for his ideologies.”

Castro would find a better ideological battleground at the United Nations General Assembly, where he finally spoke on September 26. After assuring the gathered world leaders that he would “endeavor to be brief,” the 34-year-old launched into a searing monologue that clocked in at 4-and-a-half hours—a U.N. record that still stands today. The speech included a stern condemnation of the United States’ foreign policy toward Cuba and other small nations in Latin America and Africa. Castro then went on to accuse American leaders of plotting his government’s destruction and supporting “imperialist” and “monopolist” forces around the world. To the sound of applause from Khrushchev, he also praised the Soviet Union and warned that Cuba would not stand alone if provoked.

Two days after throwing down the gauntlet at the U.N., Castro checked out of the Theresa and prepared to return to Havana. A final bit of drama unfolded at Idlewild Airport, where his delegation learned that their planes had been seized over non-payment of debts to American creditors. Castro was furious, but he found an alternative ride home from Khrushchev, who was all too happy to lend the Cubans a luxury Soviet airliner. “The Soviets are our friends,” Castro told reporters in broken English. “Here you took our planes—the authorities rob our planes. Soviet gave us plane.” At that, he disappeared inside the hammer-and-sickle stamped aircraft and took off.

Castro wouldn’t officially declare himself a “Marxist-Leninist” until a year later, but Cuban-American relations continued to deteriorate in the wake of his riotous New York trip. The United States instituted a partial trade embargo on Cuba in October 1960, and later severed diplomatic relations entirely in January 1961. The years that followed brought a seemingly endless series of controversies and Cold War close calls including the Bay of Pigs Invasion and the Cuban Missile Crisis. Even stranger was Operation Mongoose, a covert CIA program aimed at assassinating Castro and undermining his Soviet-aligned government. But despite repeated attempts to remove him, Castro remained Cuba’s leader until 2008, when he voluntarily stepped down at the age of 81. He also made several more trips to the United Nations in New York—including two more stopovers in Harlem in 1995 and 2000.