“What are you doing in bed with this woman?” Sheriff R Garnett Brooks asked as he shone his flashlight on a couple in bed. It was 2 a.m. on July 11, 1958, and the couple in question, Richard Loving and Mildred Jeter, had been married for five weeks. “I’m his wife,” Mildred responded. The sheriff, who was acting on an anonymous tip, didn’t relent with his questioning. Richard was of Irish and English descent, and Mildred of African American and Native American descent, and according to state law, it was a crime for them to be married. They were arrested for violating Virginia’s Racial Integrity Act.

Richard spent a night in jail before being released on a $1,000 bond his sister procured. Mildred, however, was not allowed a bond. She spent three nights alone in the small woman’s cell that only fit one. When she was finally released, it was to her father’s care. After the couple pled guilty, the presiding judge, Leon M. Bazile, gave them a choice, leave Virginia for 25 years or go to prison. They left and would spend the next nine years in exile.

The Lovings first met when Mildred was 11 and Richard was 17. He was a family friend, but their dating courtship didn’t begin until years later. Growing up about three or four miles apart, they were raised in a relatively mixed community that saw themselves as a family, regardless of race. Often coming together over music and drag racing, it was not uncommon for people of different races to intermingle, work together and sometimes date. Mildred’s mother was part Rappahannock Indian, and her father was part Cherokee. She later identified herself as Indian.

Richard and Mildred dated on and off for a couple of years before they decided to get married after Mildred became pregnant. (Mildred already had a first child from another relationship.) The Lovings traveled to Washington, D.C. to marry, where interracial marriage was legal, and it was the nation’s capital that they would later return to when they were forced to leave their home.

Leaving behind their family and friends, the Lovings attempted to make a life in Washington, D.C., but they never felt at home. Mildred didn’t adapt to city life; she was a country girl who was used to a rural area where there was room for kids to play. Wanting to see family, the Lovings would defy the court order to periodically return to Virginia. As they were not allowed to return together, they would take precautions not to be seen together in Virginia, Richard often never venturing outside the house.

In the backdrop of the Lovings’ struggle, the civil rights movement was taking root. While the Lovings were too preoccupied with their own hardships to be involved, they were inspired by the activism they saw. In 1964, Mildred wrote to Attorney General Robert F. Kennedy for help. Kennedy told her to contact the American Civil Liberties Union. ACLU lawyers Bernard S. Cohen and Philip J. Hirschkop eagerly took the case.

Their first attempt at justice was to have the case vacated and the ruling reversed by the original judge. After waiting almost a year for a response, they brought a class action suit to the U.S. District Court of the Eastern District of Virginia, which finally elicited a response from Judge Bazile. He stated, “Almighty God created the races white, black, yellow, malay and red, and he placed them on separate continents. And but for the interference with his arrangement there would be no cause for such marriages. The fact that he separated the races shows that he did not intend for the races to mix.” This prejudice-filled response provided the grounds for an appeal to the Virginia Supreme Court of Appeal, but that court upheld the original ruling.

By this time, the Lovings were living secretly together in Virginia. They considered staying separately with their own families, but on the advice of their lawyers they remained together only after being assured that even if arrested, they would only be held for a couple of hours (with the ACLU on call to assist with a release).

LIFE photographer Grey Villet met the Lovings in 1965, before the landmark case went to trial when he was sent on assignment to document the day-to-day world of the couple. He captured a simple story, a love story. He took photos of the Lovings watching TV together, playing with their kids and kissing. The photos ran in a 1966 issue, providing a rare look into the private lives of a couple that would have such a lasting impact on the laws of the United States.

The case made its way to the United States Supreme Court, where oral arguments began on April 10, 1967. Philip Hirschkop wasn’t qualified to try a case in front of the Court, since he was only out of law school a little over two years (a year shy of the requirement). This meant anything Hirschkop wrote had to be signed off by Bernard Cohen, who had been out of law school over three years but had no experience in federal court. These two novice lawyers understood they were arguing one of the most important constitutional law cases ever to come before the Court.

When asked her thoughts on the case before the oral arguments began, Mildred said, “It’s the principle, it’s the law. I don’t think it’s right. If we do win, we will be helping a lot of people. I know we have some enemies, but we have some friends too, so it really don’t make any difference about my enemies.” Neither of the Lovings appeared in court, but Richard did send a letter to his lawyers that read, “Tell the Court I love my wife and it is just not fair that I cannot live with her in Virginia.” The judges agreed. In a unanimous decision handed down on June 12, 1967, laws banning interracial marriage were deemed unconstitutional, overturning them in 16 states (although Alabama would only repeal its anti-miscegenation laws in 2000). Basing its decision on the due process and equal protection clauses of the 14th Amendment, the ruling read, “Under our Constitution, the freedom to marry, or not marry, a person of another race resides with the individual and cannot be infringed by the state. These convictions must be reversed. It is so ordered.”

It took nine years, but the Lovings were finally—legally—home. They built a house together on an acre of land Richard’s father had given them. Eight years later, the Lovings were hit by a drunk driver while driving home on a Saturday night. Richard was killed. Mildred never remarried, but she stayed in the home Richard built surrounded by family and friends.

Mildred lived a quiet, private life declining interviews and staying clear of the spotlight. She did, however, make a rare exception in June of 2007. On the 40th anniversary of the Loving v. Virginia ruling, three people working on behalf of the gay rights group Faith in America came to Mildred for her thoughts on same-sex marriage. After careful reflection and discussions with neighbors and her children the devoutly religious Mildred issued a statement that read, in part, “I believe all Americans, no matter their race, no matter their sex, no matter their sexual orientation, should have that same freedom to marry. Government has no business imposing some people’s religious beliefs over others. Especially if it denies people’s civil rights.”

There is little doubt about Mildred and Richard’s legacy. There’s an unofficial celebration on June 12, called “Loving Day,” honoring the anniversary of the Supreme Court decision and multiculturalism. Loving v. Virginia declared anti-miscegenation laws to be illegal across the United States, but perhaps, even more importantly, it’s the legacy of an ever-lasting love—a love that triumphed even in the face of persistent hate.

Their story hit the silver screen on November 4, 2016, in the award-winning film “Loving.”