When six U.S. Marines raised a flag over Iwo Jima in February 1945, they were laying claim to the slopes of a mountain that was part of a strategically important chain of volcanic islands south of Tokyo. The Ogasawara Islands, also known as the Bonin Islands by Americans, were largely uninhabited. But during World War II, they offered a place where the invasion of Japan could be staged.

The islands themselves weren’t completely empty—they were home to thousands of Japanese people, many of them with British and American ancestry. And the American victory turned most of them into refugees when the United States occupied the islands for the next 23 years. The story of the Bonin Islands is one of a group of seemingly obscure islands caught in the crosshairs of international conflict—and the effects of war and occupation still reverberate decades after they were finally turned back over to Japan.

The group of islands came by their Japanese name, which means “empty of men,” honestly. Though they seem to have been at least briefly occupied by Stone Age humans at some point, they were uninhabited by the 1670s, when they were mapped and named by Japanese explorers. However, they were only developed 150 years later, when American and English settlers established a colony after being shipwrecked there. The small community soon became racially diverse when sailors from whaling ships, some of them black, settled on the islands.

In 1860, Japan finally decided to lay claim to the islands and went to investigate. They found a small but diverse group of residents that hailed from everywhere from the Netherlands to Hawaii. Once the Ogasawaras became Japanese territory in earnest in 1875, Japanese settlers joined their ranks.

This diverse group of settlers lived quietly in small villages on the islands in thatched-roof homes, supporting themselves by selling goods to passing ships, hunting and weaving baskets and living off of fish, seals and sea turtles. Though the islands were technically Japanese, they were off the radar for an empire that had other, more pressing territorial and political questions on its hands.

During World War II, the obscure islands were suddenly of strategic importance, and islanders were faced with the realization that their homes were about to become battlegrounds. On Iwo To, also known as Iwo Jima, the island’s 1,000 were evacuated to the mainland in July 1944 as the Japanese military began a military buildup on the islands.

Residents may have suspected that the island would be home to some of the war’s fiercest battles, but they could never have guessed that they’d never go home. In just 36 days in 1945, the Battle of Iwo Jima resulted in 26,000 American casualties and the deaths of nearly 7,000 U.S. Marines. Twenty thousand Japanese soldiers fought in the battle; only 1,000 or so survived.

Then, after the war, Japan surrendered possession of the Ogasawara Islands—called the Bonin Islands by Americans—to the United States. The U.S. Navy occupied the islands until 1952 and administered them until 1968. Rather than let the historical residents of Ogasawara return home, the Navy only allowed people who were descended from or married to descendants of the islands’ original settlers to stay. Europeans and Americans were welcomed back, but only a small number of Japanese people were allowed back on the islands.

Though the people on the island were mostly Japanese citizens, the United States only considered those of American and European descent to be worthy of staying there. Meanwhile, thousands of former residents lived as refugees in Tokyo and elsewhere.

“To the Japanese government,” wrote Samuel Jameson for The Chicago Tribune in 1964, “this represents an attempt by the Navy to set up its own little empire under the pretext of maintaining a base vital to free world security in the East.” According to historian David Chapman, the attempt to paint the Bonin Islands as “American” was designed to weaken Japan’s ability to reclaim the islands.

“The notion that all islanders wanted to become U.S. citizens or wanted their islands to be part of the United States is not without controversy,” he writes.

During the years of American occupation, people were banned from writing to Japan and communicating with the refugees.

“I missed my old island friends so much,” Kyoko Ohira, a descendant of Western settlers, told the Japan Times’ Takumi Toguchi. Only a handful of people were allowed to return, and some found the islands a haven compared to the bombed-out, starving Japanese mainland. English was taught in schools, and the islands were dominated by a Navy submarine base. The “Navy generation,” as they called themselves, created a new language called Ogasawara Mixed Language (OML), a kind of creole that mixed Japanese and English. But they lived under the direct influence of the United States.

Then, in 1962, the United States abruptly gave the islands back to Japan. Many islanders weren’t prepared—they were linked to the United States, but now were considered Japanese citizens. As the islands once again fell under Japanese control, islanders reconnected with their long-lost friends and family members and refugees returned. But for those who had lived on the island during the occupation, there was longstanding confusion as to their cultural identity.

Even years after the handover, some Ogasawara residents are ambivalent about the change. “There are people who are very sad about the handover,” Yoko Tahashi, who lives in Chichijima, told the Japan Times’ David McNeill. “They don’t think of themselves as either Japanese or American, and feel that they have been cast aside. I feel sympathy for both sides.”