Since the United States became a nation, people have come together to count their blessings, feast on bountiful foods and give thanks with family and friends. These days, Thanksgiving celebrations usually involve turkey, pie and a food coma; in the past, they involved fasting, prayer gatherings and solemn religious ceremonies.

But there’s one president who refused to endorse the tradition: Thomas Jefferson.

Ever since Jefferson first declined to mark the day in 1801, rumors have swirled that the third president despised the event. But it was more complicated than that. For Jefferson, supporting Thanksgiving meant supporting state-sponsored religion, and it was his aversion to mixing church and state that earned him a reputation as America’s only anti-Thanksgiving president.

In Jefferson’s time, Thanksgiving as a national holiday didn’t exist at all. The formal observance of Thanksgiving Day only began in 1863, when Lincoln proclaimed the holiday in response to the horrors of the Civil War. By then, the tradition of giving thanks as a nation had been in place since 1777 when Congress declared a national day of thanksgiving after America’s victory at the Battle of Saratoga. Afterward, presidents would proclaim periodic days of fasting, prayer and expressing gratitude.

But not Jefferson. When he became president, he stopped declaring the holidays that George Washington and John Adams had so enthusiastically supported—and in 1802, he flirted with telling the nation why.

Shortly after his inauguration, a Baptist group in Connecticut wrote a letter to Jefferson congratulating him on his election and expressing concern about the state’s constitution, which didn’t specifically provide for religious liberty. Baptists had long been persecuted in the colonies due to their emotional religious ceremonies, their decision to baptize adults instead of children, and their belief in the separation of church and state. The Baptist Association of Danbury wanted to be sure that they’d be protected under Jefferson’s presidency.

Jefferson saw this as an opportunity to explain his views on state-sponsored religion. “I have long wished to find [an occasion to say] why I do not proclaim fastings & thanksgivings, as my predecessors did,” Jefferson wrote to his attorney general and friend, Levi Lincoln.

At the time, Jefferson’s political enemies, the Federalists, loved to use his stance on the separation of church and state as a political cudgel, convincing Americans that he was an atheist who was making America less godly. Perhaps his response to the Baptists, which would be widely read, could make his views clearer and protect him against those slurs.

In an early draft of the letter, Jefferson faced the Federalist accusations head-on, explaining that he considered declaring fasts or days of thanksgiving to be expressions of religion and that he opposed them because they were remnants of Britain’s reign over the American colonies.

But Levi Lincoln warned him that his words might be construed as a criticism of New England, where the feast of thanksgiving had become a beloved tradition. After careful consideration, Jefferson decided to drop the reference from his letter. His public reply to the Danbury Baptists didn’t include a comment on public celebrations of Thanksgiving. Rather, Jefferson told them he believed in “a wall of separation between Church and State.”

Jefferson paid the political price for that edit. “Withholding from the public the rationale for his policy on thanksgivings and fasts did not solve Jefferson’s problem,” writes historian James Hutson. Since the public didn’t know the reasoning behind his lack of thanksgiving proclamations, says Hutson, he remained vulnerable to Federalist attacks that accused him of godlessness.

In fact, Jefferson had once declared a Thanksgiving of his own: In 1779, while serving as governor of Virginia, he declared a day of Thanksgiving and Prayer. In 1808, he explained why he had been willing to do so as governor, but not president. Jefferson believed he could not endorse such a holiday without running afoul of the First Amendment—and furthermore, he considered days of thanksgiving the responsibility of the states, not the federal government.

For Jefferson, it was more important to maintain a strict separation of church and state than to cave into the public’s love of giving thanks. But since he never explained himself in public, American citizens never got the chance to appreciate his principled stance on the holiday. Jefferson’s public silence on Thanksgiving spun out into a centuries-long rumor that he was a Turkey Day grinch—especially when his successor, James Madison, resuscitated the tradition in 1815.