On the morning of April 24, New Orleans residents woke to news that the “Liberty Place” monument, an obelisk, had been removed from the cityscape. The obelisk, dedicated in 1891, honored a group of white supremacists who, 20 years earlier, had initiated a clash with black police and state militia in an effort to overthrow the government of Louisiana during Reconstruction. Local reporters covering the scene of the obelisk’s removal last month described something out of an action film: snipers on nearby rooftops, the identity of the work crew concealed with face masks, tape placed over the contractor’s name on the work trucks, and a ring of police officers.

Removing a monument under the cover of night might seem a bit over the top, especially considering that it was the most objectionable of four monuments to Confederate leaders and their defenders slated for removal by the city. In addition to the “Liberty Place” monument, the other three then in question were statues to Robert E. Lee, Jefferson Davis, and P.G.T. Beauregard. Just this week, again in the wee hours, the city removed the Davis statue.

The opposition to the monuments’ removal in New Orleans has been heated and sometimes violent from the earliest days of this debate, beginning with efforts to intimidate potential contractors who were bidding on removal. After the city removed the “Liberty Place” monument, pro-monument protesters held vigils, armed at times, to protect the remaining statues.

The debate over the monuments in New Orleans has been most active since 2015, in the aftermath of the Charleston massacre at the Emmanuel Baptist Church committed by Confederate sympathizer Dylann Roof. It was then that Mayor Mitch Landrieu proclaimed that the Confederate monuments in New Orleans should come down because they do not reflect “who we are” as a city.

Since then, other southern cities and towns have had similar debates about what to do with Confederate monuments. College campuses, such as the University of Texas at Austin and the University of Louisville, appointed commissions to study the issue and make recommendations. Other cities such as Memphis and Charlottesville, are still divided, with the fate of the monuments often becoming a tug-of-war between city councils and preservation groups. Baltimore, which also appointed a commission, is still weighing the fate of its monuments.

New Orleans’s three Confederate statues were the work of prominent white citizens who, in the aftermath of Reconstruction, wanted to honor their Civil War leaders. The committees who raised money to build them were comprised of white veterans, the wives of white veterans, and their descendants. The first was the Robert E. Lee monument, erected in 1884 and one of the earliest such tributes erected to Lee in the South. The Jefferson Davis and P.G. T. Beauregard monuments followed, all part of a trend throughout the South at the turn of the twentieth century to put up monuments and memorials dedicated to Confederate leaders.

Such memorials both celebrated the defenders of the pro-slavery South during the Civil War and reasserted white political power and a return to “home rule” after Reconstruction. Unlike Lee and Davis, Beauregard was a Louisiana native who after the war sought collaboration between Republicans and Democrats in city and state politics, much to the dismay of some of his peers.

Because debates over removal of these statues are part of a regional movement, what has been lost in most media coverage of these disputes is that they are all, at heart, local struggles. There may be groups across the country that want to honor Confederate leaders, but there is no longer a Confederacy. Indeed, only at the local level can this standoff be resolved.

Confederate monuments throughout the South were placed in the public sphere at a time (the early decades of the 20th century) when African Americans were denied voting rights, access to public office, equal access to public space, and the right to free speech without intimidation. The statues of Lee, Davis and Beauregard were the work of the city’s white elite. In this light, replacing them with something that reflects the city’s current population, as a whole, is an act of democracy long overdue.

Those in favor of the removal of Lee, Davis, and Beauregard see these artifacts as oppressive symbols of white supremacy that do not represent the culturally rich and majority black city. Some members of Take ‘em Down NOLA, a group supporting removal, have been fighting against racism and inequality since the Civil Rights movement in the 1960s. Those opposed cite arguments about the monument’s educational and historical value and the dangers of whitewashing history. The monument supporters are getting late reinforcements from state legislators who are trying to pass the Louisiana Military Memorial Conservation Act that, in future, will compel local governments to hold a city-wide vote before they can remove any military monument from their property. The Act, if passed will come too late affect the imminent removal of the New Orleans monuments.

The recent vigils and protests, which happened during New Orleans’s popular Jazz and Heritage Festival, when thousands of visitors join locals in celebration of the cultural diversity of New Orleans and south Louisiana. The concurrence of the protests and Jazzfest exposed the crux of the problem with the Confederate monuments in New Orleans: there are two versions of this city that co-exist in the minds of residents and in the brochures minted by the tourist industry on which we all depend to keep the economy going. One version is a circum-Caribbean port set culturally apart from the rest of the southern states because of its African, French, and Spanish colonial heritage and its close ties to Latin America and the Caribbean. The other is an old-line Southern city with its attendant class and racial divides, the legacies of slavery and segregation. These narratives about New Orleans have co-existed, you might argue, since Thomas Jefferson signed the Louisiana Purchase, bestowing an American fate on a multi-ethnic city of French-speaking Creoles. Only sometimes do these two imagined cities directly confront each other, until they do.

The monuments in New Orleans inhabit major public spaces, and their visibility and centrality to daily commutes has helped fuel the debates. The Lee monument, in particular, was considered a major landmark since it was erected in 1884, and was often featured on engravings and postcards. So, they are monuments, memorials, and symbols but even more to the point in this local battle they are prominent ones.

Further still, New Orleans’s existence depends upon the tourist industry, to an extraordinary degree. And in 2018, the city will celebrate its Tricentennial, aimed at drawing even more visitors than usual. The notion that in 2018 New Orleans will be in the spotlight brings urgency to the decision to remove artifacts that celebrate the history of the Confederacy.

Mayor Landrieu, in a recent interview, maintains that despite accusations to the contrary, he did not create the current standoff—that he “didn’t tear the community apart”—but instead “raised the issue.” But it is clear in hindsight that the city’s path towards removal produced opportunities for acrimonious exchange—from raucous city council hearings, to threats against public-bid contractors, to court challenges that delayed the city’s removal of the monuments by over a year.

It’s unclear what will happen to the monuments when they are removed. For now, they will be housed in a city warehouse until a permanent home can be found. There are some models for relocation of Confederate monuments already. Among southern states, university campuses seem to have had the most success, so far, in developing plans for moving Confederate monuments to more appropriate sites, such as a center for historical study for their statue of Jefferson Davis, in the case of U.T. (The fate of the remaining Confederate statues on the U.T. campus, however, is still the subject of debate.) The monument from the University of Louisville has been relocated to a nearby town that hosts Civil War re-enactments.

Even with the statues removed, however, we are left with the question of what to do with these public spaces. There are already venues that celebrate the city’s cultural achievements, for instance, but as yet there is no monument to the long history of civil rights struggles in New Orleans, struggles that began in the 19th century. And historians and cultural institutions have only recently begun to engage the public about the city’s history of slavery and its role in the domestic slave trade, without which the city would not have thrived before the Civil War. If the city is committed to addressing symbols of white supremacy, it should also support—in material and public ways—these long neglected parts of its history.

How do we engage these and other topics in a way that is inclusive, honest, and far reaching? And what other parts of the city’s history need to be recognized? Perhaps hardest of all: how will we remember the Civil War itself and its significance for this city whose booming economy was once built on slavery and cotton? These are questions that only New Orleans residents can answer.

In the early days of this debate, I hoped the city could find a creative and instructive way to publicly confront the white supremacy symbolized by these monuments—perhaps countering at least one of the statues with a contemporary of Lee: Andre Cailloux, a black New Orleanian who died fighting for freedom during the Civil War. The idea was to put Cailloux on equal footing with Lee, not unlike the way the recent “Fearless Girl” statue counters the bull of Wall Street. After two years of discussion and protest, it is clear that the hatred of Confederate statues and what they represent, in the eyes of many residents, is too powerful for such a plan.

The mayor has indicated that he will form a committee to develop a design for Lee Circle—a plan that represents “a gift to the future that represents the past.” Hopefully, this can be a truly collaborative process. A fierce attachment to this low-lying port city still characterizes both sides of the monument debate. The physical city is, after all, what makes it both part of the South and part of the Caribbean world. Let us hope that this shared love of place, publicly acknowledged, will inspire a new chapter in New Orleans’s public history.

Mary Niall Mitchell is Ethel & Herman L. Midlo Chair in New Orleans Studies at the University of New Orleans.