Prisons all over America are seeing a new era of rapid, radical change. But things were different just a decade ago.

Back in 2005, the US Supreme Court ordered California to end its long-held policy of racially segregating inmates. Ever since, California’s prison system has been under pressure to comply with that ruling. A new documentary, called "In An Ideal World,” shows how difficult it can be to change America’s segregated prison system.

Noel Schwerin, the producer and director of the film, spent seven years filming prisoners and corrections officials inside California’s Soledad Correctional Training Facility, including John Picirillo, a white separatist and murderer; Sam Lewis, a black ex-gang member; and warden Ben Curry.

“Ben is a really interesting figure, and like the other two main characters, I wanted to choose men who had been inside prison for the last 35 years, during this moment of this huge experiment in mass incarceration,” says Schwerin.

Though Ben Curry eventually became the warden of Soledad Correctional Training Facility, which is commonly referred to as Soledad State Prison, he started out as a correctional officer. Having spent his entire adult life in the prison system, Curry has seen the radical jump in incarceration rates that have come as a result of America’s prison industrial complex.

Between 1980 and 2006, the California prison population went from about23,000 inmates to more than 170,000. During the same time period, the US prison population jumped from about 300,000 to 1.5 million, according to the US Bureau of Justice Statistics. The jump in incarceration rates, which disproportionately affect black men, have been linked to the “tough on crime” era of the 1980s.

For Curry, racially segregating Soledad State Prison was just the norm.

“We traditionally have housed black inmates with black inmates; white inmates with white inmates; southern Hispanics inmates with southern Hispanics; northern Hispanics with northern Hispanics, others with others,” he says. “We’ve done that as a means to control the violence, and we were fairly successful with it.”

At Soledad State Prison, racial segregation has been “explicit,” says Schwerin, and it’s a rule that the inmates feel they need to live by in order to survive.

“We live by a certain convict code, and when the rules are broken, it’s expected to be dealt with in a violent manner,” says Picirillo. “If we all stand strong and we all stood united, no one else would be able to prey upon us.”

Another individual profiled in the documentary, Sam Lewis, became a model inmate during his tenure at the prison. He was released from the facility while Schwerin was filming, and he told her that he was conflicted about the racial game he had to play at Soledad.

“The rules are there to keep order, but the wild card is you never know what the next man is going to do,” he said. “The wild card is like the guy who bumps into you in the chow hall, and you don’t say excuse me, and then you get hit in the back of the head with a tray and then a riot kicks off.”

Lewis told Schwerin that even if you didn’t arrive in a gang at Soledad, you had no choice with whom to associate with. Schwerin says that the inmates and correctional staff adopted the rules of segregation due to the absence of security and safety.

“Individuals on both sides of the bars turns toward the system for order and the maintenance of order,” she says.

Since the Supreme Court decided that fear of racial violence is not a valid argument for maintaining segregation, Soledad has introduced mixed race workshops to help inmates challenge their views about each other as integration moves forward. But Schwerin says that real change must come from the outside.

“[Prison] is our institution,” she says. “It is endorsed by, sustained by, subsidized by our monies, and as much as we like it or not, our values.”

This story first aired as an interview on PRI's The Takeaway, a public radio program that invites you to be part of the American conversation. "In An Ideal World" is part of the America Reframed series, which is co-produced by our partner, WGBH.

From The Takeaway ©2016 PRI and WNYC