Secret societies have flourished throughout history and count Founding Fathers and royals among their ranks. Members (most often men) have been tapped to join The Knights Templar, the Freemasons, the Bavarian Illuminati, Skull and Bones and Bilderberg. The allure of secret societies is part mystery, part legend.

Conspiracy theories have surrounded them for centuries, with rumors of groups like the Illuminati being linked to everything from the French Revolution to the assassination of JFK. But it’s important to separate fact from fiction.

Here are the real stories behind history’s most exclusive secret societies.

The Knights Templar were warriors dedicated to protecting Christian pilgrims to the Holy Land during the Crusades. The military order was founded around 1118 when Hugues de Payens, a French knight, created the Poor Fellow-Soldiers of Christ and the Temple of Solomon—or The Knights Templar for short. Headquartered at Temple Mount in Jerusalem, members pledged to live a life of chastity, obedience and poverty, abstaining from gambling, alcohol and even swearing.

The Knights Templar were known for more than their military prowess and moral lifestyle. They became one of the most wealthy and powerful forces in Europe after setting up a bank that allowed pilgrims to deposit money in their home countries and withdraw it in the Holy Land.

Their influence swelled to a new high in 1139 when Pope Innocent II issued a Papal Bull exempting them from paying taxes… and decreeing that the only authority they had to answer to was the Pope. At the apex of their power, the Knights Templar owned the island of Cyprus, a fleet of ships and lent money to kings. But not all kings were happy customers.

When the Crusades came to an end after the fall of Acre, the Knights Templar withdrew to Paris, where they focused on their banking endeavors. On October 13, 1307, King Philip IV of France, whom the Knights Templar had denied additional loans, had a group of knights arrested and tortured until they made false confessions of depravity. In 1309, as the city of Paris watched, dozens of Knights Templar were burned at the stake for their alleged crimes.

Under pressure from the French crown, Pope Clement V formally dissolved the order in 1312 and redistributed their wealth. Rumors that the Knights Templar guarded artifacts like the Holy Grail and Shroud of Turin began bubbling up among conspiracy theorists. Popular books and films like The Da Vinci Code continue to inspire curiosity about the Knights Templar today.

The Cross of Lorraine (Croix de Lorraine in French) is a double-barred cross that is featured prominently in the coat of arms of the Dukes of Lorraine. After Lorraine Nobleman Godfrey de Bouillon became the king of Jerusalem during the First Crusade, the symbol became known as the “Jerusalem Cross.” When the Knights Templar arrived in the Holy Land, they adopted it as the symbol of their order.

During World War II, the Cross of Lorraine was a symbol of the French resistance to Nazi rule. Some eagle-eyed observers have claimed to spot the Cross of Lorraine in the Exxon and Nabisco logos and even stamped on Oreo cookies.

The freemasons loom large in American history—after all, 13 of the 39 men who signed the U.S. Constitution were Masons. Founding Fathers like George Washington, James Monroe, Benjamin Franklin, John Hancock and Paul Revere all counted themselves as members of the fraternal order. But who are the Freemasons?

The Freemasons can trace their routes to the Middle Ages in Europe when most craftsmen were organized into local guilds. Cathedral builders, by nature of their profession, had to travel from city to city. They identified one another via signs of their trade, like the builder’s square and compass in Freemasonry’s now-iconic symbol.

The earliest reference to masons is in the Regius Poem, or Halliwell Manuscript, which was published in 1390, but Freemasonry, as we know it today, was founded in 1717, when four London lodges merged to form England’s first Grand Lodge. Freemasonry quickly spread across Europe and to the American colonies.

Freemasonry is not a religion, though members are encouraged to believe in a Supreme Being, or "Grand Architect of the Universe.” Masonic temples and secret rituals have brought them into conflict with the Catholic Church. The Church first condemned the Freemasons in 1738 and has gone on to issue around 20 decrees against them. In 1985, Roman Catholic Bishops restated over 200 years’ worth of these strictures in the face of an increased number of Catholics joining the order.

The Church wasn’t their only enemy; the secrecy of the masons garnered such distrust in early America that it inspired America’s first “third party”: The Anti-Masonic Party.

Freemasons exist today, and their public image has been greatly influenced by the high-profile charity work of the Shriners, a subset of Freemasons also known as “the Ancient Arabic Order Nobles of the Mystic Shrine.” The Shriners were founded by Freemasons in 1870 at New York City’s Knickerbocker College and continue their volunteer work today.

The rituals around becoming a freemason are shrouded in secrecy, but have entered the public imagination in film and TV and were even parodied on an episode of “The Simpsons.” Membership is open to all males over the age of 21, and women can join an associated group known as “The Order of the Eastern Star.” According to the New York Times, aspiring members must ask to join and cannot be otherwise approached, as summed up in a recruiting slogan: “All you have to do is ask.”

If you do get in, you’ll be in good company: Famous freemasons include Mozart, Winston Churchill, Davy Crockett, Franklin D. Roosevelt and John Wayne.

The most recognizable symbol of the Freemasons is “The Square and Compasses.” The right angle of the builder’s square is joined by a compass, a central tool in geometry–which, according to some experts at MIT, is represented by the “G” at the heart of the symbol. Others have interpreted the “G” as representative of God, the “Grand Architect of the Universe.”

The view of the All-Seeing Eye as a masonic symbol has been sharply debated. Long before the Freemasons, Egyptians used the “Eye of Horus,” and the all-seeing eye appears repeatedly in Renaissance art as a symbol of Christianity and God’s watchfulness. But organizations like the Philadelphia Federal Reserve claim Freemasons Henry Wallace and Franklin D. Roosevelt purposefully chose it when they redesigned the dollar bill in 1934.

According to the George Washington Masonic National Memorial, the “All-Seeing Eye” is a masonic symbol of the “watchful care of the Supreme Architect” that began appearing in printed Masonic literature in the mid-1700s.

The Illuminati was founded by professor Adam Weishaupt in Bavaria on May 1, 1776. Weishaupt, chafing at the power of the conservative Catholic Church and the Bavarian monarchy, sought to cast aside organized religion in favor of a new form of “illumination” through reason. Inspired by the spread of the Enlightenment across Europe, he also drew upon ideas expressed by the Jesuits (he was a former member), the Mysteries of the Seven Sages of Memphis, the Kabbalah and Freemasons. He recruited heavily from the latter group, infiltrating masonic lodges in his quest to recruit some of the wealthiest and most influential men in Europe.

Members of the Bavarian Illuminati, referred to as “Perfectibilists,” were broken into three tiers of increasing power and drawn from societal elites including noblemen like former freemason Baron von Knigge and writer Johann Wolfgang von Goethe. All communication was in cipher and members were given classical nicknames (Weishaupt’s, for example, was Spartacus).

The organization flourished before being stamped out by Karl Theodor of Bavaria, who issued an edict making membership in the Illuminati punishable by death in 1787. But the death of the Bavarian Illuminati did not quell gossip about their clandestine activities, and conspiracy theorists have linked the group to everything from the French Revolution to the assassination of JFK. The Illuminati served as inspiration for Dan Brown’s Angels & Demons and Foucault’s Pendulum by Umberto Eco.

The Order of Skull and Bones is a secret society founded at Yale University in New Haven, Connecticut in 1832. Skull and Bones founder William Huntington Russell was inspired by an occult society he’d visited in Germany. His co-founder was Alphonso Taft, future Secretary of War under President Grant and father of president William Howard Taft… who would also be a member of Skull and Bones. The prominent list of Bonesman includes several presidents and modern-day power brokers.

Each year, 15 seniors at Yale are tapped to join Skull and Bones. Their names are published in Yale Rumpus, though what happens behind the closed doors of The Tomb, the windowless meeting space where Bonesmen gather twice a week, is under wraps: Members take an oath of secrecy. Graduate members are referred to as “patriarchs,” while those undergoing initiation are called “knights.” Outsiders of the group are “barbarians.”

Famous Skull and Bones members include Presidents William Howard Taft, George H.W. Bush and his son, George W. Bush; founder of Time magazine Henry Luce; former secretary of state and presidential hopeful John Kerry; Fortune 500 elites and members of the CIA.

The controversial 1986 exposé America’s Secret Establishment by Anthony Sutton claimed that Skull and Bones was out to create a “new world order” run by Bonesmen, prompting a myriad of conspiracy theories.

The symbol of Skull and Bones is, appropriately, a skull with two crossbones. What’s less clear is the meaning of the number “322” beneath them. Yale Alumni Magazine points to a popular theory that it represents the year 322 B.C. when Alexander the Great died.

The first Bilderberg Meeting was in 1954 and was held at the Hotel de Bilderberg in the Netherlands, from which the organization gets its name. Convened by Prince Bernhard of the Netherlands, it was a gathering of powerful politicians from North America and Europe designed to foster warmer relations between the two continents among fears of growing anti-Americanism in Europe.

While not strictly a secret society like the Illuminati or Freemasons, Bilderberg’s high-profile attendees—previous guests have included Bill Clinton, Margaret Thatcher, Angela Merkel, Tony Blair and Henry Kissinger—and its use of the Chatham House Rule blocked attendees from sharing what actually happens in meetings gives the group an air of mystery. Journalists are barred from reporting on it. Meeting minutes are not released.

Bilderberg attendees are selected by a dedicated international committee. Every year, about 120-140 people are invited, with about two-thirds coming from Europe and one-third from North America. The Washington Post reports that while backgrounds in government and politics are the most common, attendees from fields like academia, finance and media have also been included.

The level of secrecy surrounding the Bilderberg Meeting has given rise to many rumors, including unproven theories that Bilderberg attendees are behind the creation of the European Union, the invasion of Iraq and the bombing of Serbia, reports the New York Times. Conspiracy theorists have painted the group as plotting a new world order. Their official website maintains, “Thanks to the private nature of the Meeting, the participants take part as individuals rather than in any official capacity, and hence are not bound by the conventions of their office or by pre-agreed positions.”

Publicly known topics of conversation for this year’s Bilderberg Meeting are Brexit, cyber security and climate change.

What happens behind the closed doors of these secret societies has caused debate for centuries. What’s clear is that they continue to spark the imagination and curiosity of the public.