The 2020 census won’t ask you about how many people in your family are “idiots” or “insane,” but in 1840 that was a question census workers had to answer for every household. The Census Bureau added the question at a time when reformers were interested in creating institutions to help people with mental disabilities. Yet near the turn of the century, scientists and doctors became less interested in helping these people and more interested in preventing them from reproducing. It was around that time that the census stopped asking about mental health.

From the beginning, experts complained that the census enumerators who went from house to house weren’t accurately identifying the number of people with mental disabilities. In particular, the 1840 census was shown to have severely over-counted the number of free black people who were “insane” or “idiots”—data that supporters of slavery used as propaganda to argue black Americans were unable to handle freedom.

LISTEN: A Century of Stigma for Black America and Mental Health on HISTORY This Week podcast

Loosely, “insane” could refer to a periodic condition of unusual behavior, and “idiotic” could refer to a permanent learning disability, but there were no precise definitions for these terms. Of the more than 17 million people counted in the 1840 census, 17,456 were listed as “insane and idiots.” Yet after experts debunked the census’ findings relating to black people, one doctor asserted in a letter to the Journal of Insanity that “the fifth census of 1840 is absolutely worthless as regards any correct enumeration of cases of insanity and idiocy.”

Why might medical authorities want to accurately count these people? “This is a period of massive asylum building,” says Sarah F. Rose, author of No Right to Be Idle: The Invention of Disability, 1840s–1930s. “Some of them really did have a very much enlightenment-based idea of: these are all human beings, we want to educate them, we want to show that they’re capable.” One great way to secure funding for an institution was to demonstrate that there was a need for one, and asylums may have seen the census as a way to do that.

READ MORE: The Most Controversial Census Changes in American History

The census continued to label people as “insane” or “idiotic” through 1880, and in 1890 dropped those words for “defective of mind.” During this time, the census was also developing new racial categories. A “racial scientist” named Josiah Nott requested that the 1850 census record mixed-race people using the term “mulatto.” Nott believed black and white people may have been different species, and he wanted to track mixed-race people to see, for example, if they had shortened lifespans. In 1890, the census also added “quadroon” and “octoroon” to indicate one-fourth and one-eighth African ancestry, respectively (it dropped the terms the next year).

At the same time that white scientists were developing ideas about “racial purity,” medical professionals were increasingly describing people with mental disabilities as a menace to society in need of restraint. Eugenicists theorized that behaviors like criminality and prostitution were products of mental instability, and therefore inheritable traits that “feeble-minded” parents would pass on to their children. Immigrants and poor people were especially stereotyped as “feeble-minded,” and nativists feared that these demographic groups were reproducing too quickly for institutions to handle them all.

“Eventually by the end of the century, they will be seen as a menace,” says James Trent, author of Inventing the Feeble Mind: A History of Intellectual Disability in the United States, and “associated with all kinds of social problems.” The belief was “it’s necessary to institutionalize them and keep them away from the rest of us, because they tend to be engaged in petty crimes.”

Many nativists felt that, because they perceived a large number of social problems in their communities, there must be many “feeble-minded” people causing them—far more than the census was counting. Even the U.S. Census Bureau seemed to think this, writing in an 1880 report that its tally of “insane” and “idiotic” people “was certainly less than half the number actually present.”

These concerns about the census’ accuracy may have been the reason the U.S. Census Bureau stopped counting people with mental disabilities on its national census in 1900. However, the concern with “feeble-minded” people didn’t go away. The Census Bureau performed a couple of mini-censuses after 1900 focusing only on people in asylums, hospitals or other institutional facilities.

By that time, these institutions were no longer focusing solely on caring for mentally disabled people and teaching them work skills. Increasingly, they wanted to keep feeble-minded people locked up indefinitely so they couldn’t reproduce. “The founder of the New York State Asylum for Idiots…create[d] what becomes a eugenic institution, in many ways, for feeble-minded women,” Rose says. “Women were released after menopause, and they were often then just dumped in the poor house.”

American eugenics was most popular in the early 20th century, during the same period when Nazi Germany was obsessed with creating a “master race” (the Nazis actually took inspiration from discriminatory U.S. laws). But eugenic practices like forced sterilization continued all the way up until the 1970s and ‘80s in the U.S., targeting especially people who were poor, Indigenous, non-white or immigrants.

READ MORE: 10 Things You May Not Know About the U.S. Census