In the mid-19th-century, the two most powerful political parties in the United States were the Democrats and the Whigs. In two presidential elections, 1840 and 1848, Americans voted a Whig into the White House. And some of the most prominent political voices of the contentious pre-Civil War era were Whigs, including Henry Clay, Daniel Webster and a one-term Illinois congressman named Abraham Lincoln.

But for all of their prominence and power, the Whigs couldn’t keep it together. The all-consuming issue of slavery was the Whigs’ ultimate undoing, pitting Northern and Southern Whigs against each other, and scattering Whig leadership to upstart third parties like the Know Nothings and the Republicans.

Over the course of a little more than 20 years, the Whig party experienced a meteoric political rise that was rivaled only by its abrupt and total collapse.

WATCH: US Presidents on HISTORY Vault

The Whigs were a loose coalition of diverse political interests—Anti-Masons, National Republicans, disillusioned Democrats—united by a shared hatred of President Andrew Jackson. To the Whigs, Jackson was “King Andrew the First,” a despot who usurped power from Congress to serve his own populist ideals.

The Whigs formed in 1834 in response to Jackson’s refusal to fund the second National Bank. They took their name from a British anti-monarchist party that was revived in Colonial America as “American Whigs.” Clay, known as “the great compromiser,” was the Whigs’ most influential and vocal early leader.

The Jacksonian Democrats painted the Whigs as a party of wealthy Northern elites who wanted to sidestep the will of the people, but the Whigs actually defied a singular identity. There were Protestant moral reformers who wanted to pass prohibition laws aimed at Catholic immigrants. There were defenders of Native Americans angered at Jackson’s relocation orders that led to the infamous Trail of Tears. And while there was a strong anti-slavery sentiment among some Whigs, it wasn’t an abolitionist party.

Like the Democratic party before the Civil War, the Whigs were a “bisectional” party that drew voters from both the North and South, explains Philip Wallach, a resident scholar at the American Enterprise Institute.

“Both parties therefore had an interest in keeping slavery off the national agenda as much as possible,” says Wallach. “But in the case of the Whig party, it just couldn’t find any way of dealing with the slavery issue that would satisfy both its Northern and Southern wings.”

READ MORE: How Andrew Jackson Rode a Populist Wave Into the White House

Even before slavery tore apart the Whig party, the Whigs faced a string of bad luck.

After four separate Whig-affiliated candidates lost the 1836 election to Jackson’s Democratic successor, Martin Van Buren, the Whigs finally won the presidency in 1840 with William Henry Harrison. But Harrison famously died from pneumonia after only 32 days in office, handing the White House to his vice president, John Tyler, a former Democrat who wasn’t a Whig party loyalist.

“Tyler served almost four full years as president, and for nearly all those years he was a man without a party,” says Wallach. “Tyler’s presidency turned out to be a major detriment to the Whig party’s ability to put down solid roots.”

Tyler, known to detractors as “His Accidency,” was such a disappointment to the Whigs—he vetoed Whig-sponsored national banking and tariff bills—that the Whigs took the extraordinary step of expelling him from the party while Tyler was still in office.

READ MORE: Why John Tyler Was a Reviled President

In the 1844 election, Clay was nominated again as the Whig candidate and lost to James K. Polk. So in 1848, the Whigs chose Zachary Taylor, a hero of the Mexican-American War and a owner of enslaved people.

Taylor won the election, but also died two years into his presidency, leaving it in the hands of Millard Fillmore, an anti-slavery Northerner. Taylor and Fillmore never saw eye to eye politically and Fillmore’s new policies did little to solidify the Whig party after Taylor’s sudden demise.

Death continued to haunt the Whig party into the 1850s. Clay, the stalwart Whig leader who inspired Lincoln and other prominent politicians to join the party, died in 1852, as did Daniel Webster.

“These men are considered two of the most important legislators who never became president,” says Wallach. “Their deaths didn’t help the forward momentum of the Whig party.”

In 1849, California petitioned to join the Union as a free state, which threatened to upset the delicate power balance between free and slaveholding states. In one of his last major political maneuvers, Henry Clay brokered the Compromise of 1850, a series of five bills which welcomed California as a free state, but also strengthened the Fugitive Slave Act that legally required Northern states to prosecute and return runaway slaves.

The Compromise of 1850, signed into law by Fillmore, was immediately and wildly unpopular with both Northern and Southern Whigs, who each had their own grievances.

“Because Fillmore hitched his wagon to the unpopular Compromise of 1850, he found himself thrown out as the Whig nominee in the 1852 party convention,” says Wallach. It took 53 separate votes before the convention delegates finally agreed on a candidate, General Winfield Scott.

Going into the 1852 election, Whigs still considered themselves the party to beat, but “Old Fuss and Feathers,” as Scott was derisively known, was shellacked in the general election by the Democrats (he only won 42 electoral votes), dealing the Whigs a bruising blow from which they never recovered.

The divisive slavery issue came to a head again in 1854 with the passing of the Kansas-Nebraska Act, which authorized new territories and states to decide for themselves if they wanted to allow slavery.

Anti-slavery Whigs, deciding that their party wasn’t sufficiently committed to halting the spread of slavery, splintered off and formed the Republican party along with anti-slavery Democrats. Among the former prominent Whigs who turned Republican were Thaddeus Stevens, William Seward and Abraham Lincoln.

Meanwhile, other Whigs were getting swept up in anti-immigrant, nativist movements like the Know Nothings, a secret society that grew to become a political force in the 1850s. Fillmore, who had been dumped by the Whigs in 1852, ran in 1856 as the nominee of the American Party, the political wing of the Know Nothings. Many conservative Whigs followed him.

1856 was the last election in which the Whigs fielded a candidate, but former Whig William Seward, who went on to serve as Lincoln’s secretary of state, pronounced the party’s eulogy in 1855: “Let, then, the Whig party pass. It committed a grievous fault, and grievously hath it answered it. Let it march out of the field, therefore, with all the honors.”

“It’s remarkable how fast it all fell apart for the Whigs,” says Wallach. “From right before the 1852 election thinking they were in good shape, to 1854 being clearly obsolete and in 1855 literally going out of business.

“It’s pretty striking.”