The United States’ first secretary of the treasury was always tight-lipped about his upbringing, which he called “the subject of the most humiliating criticism,” but there’s little doubt that it was a struggle. Born in 1755 (some sources say 1757) on the Caribbean island of Nevis, Hamilton was the illegitimate son of a Scottish father and a French Huguenot mother who was still married to another man. His father abandoned the family when Alexander was 10, and his mother died from fever just a few years later, leaving Hamilton and his brother orphans. A cousin tapped to serve as the boys’ guardian later committed suicide, but by then the teenaged Hamilton had secured a job as a clerk at an import-export firm on St. Croix. His intellect impressed his managers, and in 1773 a group of local businessmen put up the money to send him to New York, where he studied at what would eventually become Columbia University.

Once on the mainland, Hamilton delved into colonial politics and served as an aide-de-camp to General George Washington during the American Revolution. The man that John Adams once called “the bastard brat of a Scottish peddler” would go on to play a pivotal role in shaping the American political system. Before his death in 1804, he served as a delegate to the Constitutional Convention, authored many of the Federalist Papers and helped establish the United States’ first national bank.

Born March 15, 1767, the 7th president was 69 years and 354 days when he completed his second term in March 1837. Though “Old Hickory” had a reputation for being a rugged ex-soldier and outdoorsman, by the time he reached the White House, he already had spent years coping with a variety of ailments.

According to biographer H.W. Brands, samples of his hair reveal that he had lead poisoning from old bullet wounds. Jackson also struggled with chronic diarrhea from diseases he’d contracted while fighting the Indians in the 1810s. His habits of smoking and chewing tobacco didn’t help his health either, and according to biographer Sean Wilentz, Jackson became so sick at times during his two terms that it appeared he might not survive.

Jackson did make it to the end of his term but when returned to the Hermitage, his plantation in Tennessee, the white-haired ex-president was physically spent and suffered from blinding headaches, insomnia, severe pains in his side and a chronic cough.

Simón Bolívar was born in 1783 into one of the most prosperous families in Spanish Venezuela, yet his childhood was anything but idyllic. The future revolutionary never knew his father, who died before his third birthday, and he later lost his mother to illness when he was 9. The young orphan briefly lived with his grandfather before being passed off to his uncles, but according to Bolívar, his true guardian was a black slave nurse named Hipólita, who he regarded as an adoptive parent. “Her milk has nourished my life and she is the only father I have known,” he once wrote. Thanks to his family’s considerable financial means, Bolívar received a top-notch education in Europe.

After returning to Venezuela in 1807, he became a leading voice in the territory’s fight for independence from Spain. He served as a militia commander and politician during the Venezuelan War of Independence, and later took part in military campaigns that led to the formation of a half-dozen South American states including Colombia, Ecuador and Peru. By the time of his death in 1830, the orphaned aristocrat was known across the continent as “El Libertador,” or “The Liberator.”

The sense of dread and despair that permeates much of Edgar Allan Poe’s writing may have its roots in his unhappy childhood. The author of “The Raven” and “The Tell-Tale Heart” was born in 1809 into a penniless family of traveling actors, and by his third birthday his father had left and his mother had died of tuberculosis. Poe was raised by a Richmond, Virginia, tobacco merchant named John Allan—whose name he eventually took—but the two had a rocky relationship. Allan was not supportive of Poe’s literary aspirations, and young Edgar enraged his guardian by racking up considerable gambling debts during a stint at the University of Virginia.

Poe was later disinherited after the two fell out in the early 1830s, at which point he embarked on a roving writing career that took him to Richmond, Philadelphia, New York and Baltimore. His life was cut short after he died under still-mysterious circumstances in 1849, but his many poems and short stories are now credited with influencing everything from detective fiction to the horror and science fiction genres.

Ella Fitzgerald traveled a difficult path on her way to becoming America’s “First Lady of Song.” Her parents split shortly after her birth in 1917, and her mother died unexpectedly when Ella was just 15. The aspiring entertainer was sent to live with an aunt in Harlem, but she soon drifted to the streets and worked as a lookout for a brothel and a numbers runner for an illegal lottery. Fitzgerald’s frequent absences from school eventually saw her placed in New York’s Colored Orphan Asylum, where she remained for more than a year before running away.

She lived for a time on the streets of Harlem, dancing for spare change and sleeping in friends’ homes, but she finally caught a break in 1934, when she won an amateur singing contest at the Apollo Theater. Fitzgerald’s tuneful, versatile voice soon earned her a gig with bandleader Chick Webb and his orchestra. By 1938—just six years after the death of her mother—she had scored her first hit with the song “A-Tisket, A-Tasket.”

Long before she became Hollywood’s most iconic blond bombshell, Marilyn Monroe was Norma Jeane Baker, the daughter of a single mother in Los Angeles. Monroe would later describe her childhood as being almost completely devoid of happiness. She never knew her father, and her mother suffered from recurring psychiatric problems that eventually saw her committed to an institution. With no parental support, young Norma Jeane spent the majority of her childhood in a string of orphanages and foster homes, including some in which she was sexually abused.

She finally left the foster system at age 16, when she married a neighbor named James Dougherty, who soon shipped out for service in World War II. Norma Jeane began working in a wartime factory, and it was there that an army photographer spotted her and suggested that she try her hand at modeling. She proved to be a natural in front of the camera, and by 1946 she had dyed her hair blond, changed her name to Marilyn Monroe and started a new career as an actress. Her big break followed in 1950, when she nabbed memorable roles in the films The Asphalt Jungle and All About Eve.

Before he became famous for urging African Americans to win their rights “by any means necessary,” Malcolm X suffered through a tumultuous and often violent childhood. Born Malcolm Little in 1925, the future activist was visited by tragedy at age 6, when his father was killed in a Michigan streetcar accident that may have been engineered by local white supremacists. The Little family spent the next several years in dire poverty, and Malcolm bounced between foster care and juvenile homes after his mother suffered a psychological breakdown that saw her committed to a state mental hospital.

In 1941, 15-year-old Malcolm quit school and moved east to live with a half-sister. He later dabbled in drug dealing and petty crime, and in 1946 he was arrested for burglary and sentenced to prison. It was during his seven-year stint behind bars that he joined the Nation of Islam and adopted the moniker Malcolm X. Following his release in 1952, he embarked on the career that would see him become one of the United States’ most influential and controversial activists.