Although 10 Downing Street appears to be a modest-looking terrace house from the outside, there’s more than meets the eye behind its gleaming black front door. The original residence, built on the site of a medieval brewery, is a portal to 100 rooms in several larger houses connected by a warren of hallways and staircases.

For over 400 years, the land now occupied by Downing Street has been at the heart of British power. In 1682, King Charles II granted the property located on the edge of the magnificent royal Whitehall Palace to Sir George Downing. Born to Puritan parents in Ireland, Downing grew up in the Massachusetts Bay Colony—where his uncle, John Winthrop, served as governor—and was a member of the first graduating class of Harvard College before moving to England. Described by Samuel Pepys as a “perfidious rogue,” Downing proved savvy—and duplicitous—enough to serve as both anti-monarchist Oliver Cromwell’s intelligence chief and an aide to King Charles II.

After securing the royal lease, Downing built between 15 and 20 terrace houses on the street that now bears his name. Although the haphazardly numbered homes—10 Downing Street was actually 5 Downing Street until 1779—were designed by Sir Christopher Wren, they were built on the cheap. In addition, the marshy property wasn’t the most solid ground for building, and 10 Downing Street’s shallow foundations and shoddy construction would plague future British prime ministers, including Winston Churchill who called the residence “shaky and lightly built by the profiteering contractor whose name they bear.”

Lords, earls and even a man named Mr. Chicken lived in 10 Downing Street until the early 1730s when King George II presented the terrace house and the large mansion behind it as a personal gift to First Lord of the Treasury Robert Walpole, considered to be the first British prime minister. (Today the title “First Lord of the Treasury” is still inscribed on the brass letter box on 10 Downing Street’s front door.) Walpole accepted the royal gift with the stipulation that it be available to all future prime ministers. After the terrace house and the adjacent mansion were joined together in 1735, Walpole moved into his new residence and office.

Many of Walpole’s successors, however, were not impressed with what William Pitt the Younger called a “vast, awkward house” and chose to continue to reside in their own personal homes. By the early 1800s, the neighborhood around Downing Street also grew seedy with brothels and gin joints. Calling 10 Downing Street “dingy and decaying,” Prime Minister Benjamin Disraeli paid for the renovation of the home’s personal quarters in 1874 while the government financed the improvement of other parts of the residence. Since 1902, every British prime minister has called Downing Street home.

During the blitz of London in World War II, German bombs damaged 10 Downing Street’s kitchen and state rooms on October 14, 1940, while Churchill dined yards away in the Garden Room. The near-miss forced Churchill to move his living quarters to an underground bunker in the nearby war rooms, but he continued to work and eat in the sparsely furnished rooms of 10 Downing Street on occasion.

After the considerable damage sustained by the residence during World War II, complete demolition was considered but ultimately rejected in favor of a massive renovation during the 1950s. The restoration brought with it a surprise—the color of 10 Downing Street’s trademark black facade was thanks to two centuries of London pollution. Finding the original bright yellow color too jarring, workers ultimately replicated the dark exterior with a coat of black paint.

Outgoing prime minister David Cameron and his family didn’t actually reside in 10 Downing Street but next door in the more spacious living quarters of 11 Downing Street, the official residence of the chancellor of the exchequer. According to London’s Evening Standard newspaper, new British Prime Minister Theresa May might follow suit in living next door to 10 Downing Street.

As May assumes power, she will not actually be given the keys to 10 Downing Street. There is no lock on the outside of the residence’s iconic front door, which can only be opened from the inside by a guard on duty. The new prime minister, however, will be assuming stewardship of Larry, 10 Downing Street’s resident cat. Recruited into service in 2011, the feline with the official title of “Chief Mouser” is now the most famous member of the residence’s