Frank Sinatra was many things: A crooner who could make bobby-soxers faint, an Academy Award-winning actor, the elder statesman of the Rat Pack. At the height of his career, it was rumored that “every woman wants to have him; every man wants to be him.” But his fans and detractors weren’t the only people who wanted a piece of Old Blue Eyes: So did the FBI.

The Federal Bureau of Investigation tracked Sinatra for over 40 years, amassing a dossier of thousand of pages about his movements, words, and friendships. The files, which were made public after Sinatra’s death in 1998, cover Sinatra throughout his tempestuous career—and read like a thrilling account of a life he led “his way.”

Sinatra rose to fame during the 1940s, and soon attracted the attention of the FBI for claims that he’d paid a doctor $40,000 to declare him medically unfit for World War II service. Though the FBI dismissed the allegations, calling his exemption for a punctured eardrum and psychological issues legitimate, rumors that he’d dodged the draft persisted throughout his lifetime and even hurt his career in the late 1940s.

His excuse for not serving may have been watertight, but Sinatra’s ties to known Mafia members and a revolving cast of characters connected to the underworld weren’t as squeaky clean. Sinatra’s FBI file reads like a guide to the era’s organized crime figures. Though Sinatra always denied he was connected to the mob, he did interact with famous Mafia figures like Chicago mob boss Sam Giancana, with whom he was close friends.

Sinatra supposedly introduced Giancana to John F. Kennedy‘s campaign in 1960 in an attempt to deliver union votes to the future president. According to Sinatra’s daughter, Tina, he played a gig at Giancana’s Chicago club to repay the favor. Sinatra also introduced Kennedy to Judith Campbell Exner, Giancana’s girlfriend. During the years-long affair that followed, Exner allegedly acted as a liaison between Kennedy and Giancana, helping in a plot for the Mob to assassinate Fidel Castro. (It never did, but Exner later testified before a Senate committee investigating JFK’s potential Mob ties.)

Sinatra had other mafioso friends, many of whom can be found in his FBI files, which contain headings like “associations with criminals and hoodlums” and “accusations of being a dope racketeer.” There are accounts of gifts from Joseph and Charles Fischetti, brothers with the Chicago outfit who ran illegal gambling operations. There’s a Godfather-style appearance at an Atlantic City club as a favor for attendees of the wedding of the daughter of Philadelphia mobster Angelo Bruno. There’s even documentation of the Mob exerting pressure on Sinatra’s behalf to release him from a 1951 contract.

FBI agents also watched Sinatra carouse with Detroit mobsters Anthony and Vito Giacalone. “It was like clockwork,” retired FBI agent Sam Ruffino tells author Scott M. Burnstein. “A few times a year, we’d trail the Giacalones to the airport to pick up Sinatra. They’d spend the weekend together socializing before and after his shows….Almost every night, they’d shut the place down. And he didn’t make any apologies for it. Those were his friends. The fact that they were known hoodlums and murderers didn’t matter to him. He didn’t care, he was going to hang around with who he wanted to hang around with.”

Sinatra flaunted his friendships with people connected to organized crime, and took plenty of public photographs with known mobsters. His gangster friends shared his passions—gambling, women, money—and they often met in casinos and nightclubs. Sinatra, however, offered to snitch on some of the criminals he hung around: In 1950, he sent an associate to J. Edgar Hoover to offer Sinatra’s services as an informant, perhaps in an attempt to protect himself from swirling rumors that he was involved with the Mob. The FBI declined. One of Hoover’s aides wrote “We want nothing to do with him” on the report of the meeting.

Sinatra was never prosecuted for criminal behavior in connection to his many Mob ties. He was never brought before the House Un-American Activities Commission, either—but the FBI had him in its sights for what they considered to be suspicious activity with possible ties to Communism. The FBI file is filled with accounts of Sinatra’s supposedly suspicious activities, from his support of anti-racist initiatives to his defense of people accused of being Communists. In addition to appearing on behalf of many liberal causes, Sinatra was one of the founding members of the Committee for the First Amendment, a group that supported the so-called Hollywood Ten, screenwriters and directors who were blacklisted after refusing to divulge whether they were members of the Communist Party.

“Sinatra’s FBI dossier reveals a dismaying situation,” writes historian Gerald Meyer. “At no time does it contain anything that even hints at an activity disallowed by the Bill of Rights.” Meyer, who documented Sinatra’s support of progressive causes and his public confrontation of things like racism and the McCarthy-era Red Scare, sees the FBI files as evidence of a government that perceived Sinatra as a threat.

The FBI didn’t always focus on the singer himself: Since Sinatra was such a high-profile star, he was regularly targeted by people who wanted to extort or blackmail him. In 1963, those seemingly random attacks became all too personal when three men kidnapped Sinatra’s son, Frank Sinatra, Jr. The FBI told Sinatra to wait for a ransom demand, then pay it so that the bureau could track down the money and the kidnappers. Frank Sinatra, Jr. was freed when the kidnappers became cagey about their crime, and the FBI soon caught the abductors. All three were convicted.

Sinatra knew the government was tracking his activities—in 1979 and 1980, he requested and received his FBI file through the Freedom of Information Act. His FBI file never ended up getting him in trouble, but it reflects the power and influence of the charismatic singer. Though it’s full of references to his shady dealings and thuggish friends, it also shows him speaking out against racism and on behalf of democracy.

Even Sinatra’s professional triumphs were enough to get him reported to the FBI: The file begins not with an account of his Mob ties, but with a letter that complains about the “shrill whistling sound” produced by Sinatra’s fans. “How easy it would be for certain-minded manufacturers to create another Hitler here in America through the influence of mass-hysteria!” wrote the anonymous informer. “They intend to get a Hitler in by first planting in the minds of the people that men like Frank Sinatra are O.K.”

Sinatra, it seemed, could attract attention of conspiracy theorists and bobby-soxers with the same crooning voice—and through his career, FBI agents listened as intently as his closest fans.