Three days after CBS CEO Les Moonves was accused of sexual harassment by six women, Stephen Colbert has demanded accountability for his boss. In a thoughtful, measured segment on The Late Show Monday night, Colbert argued that if we aren’t willing to demand that everyone answer for the terrible things they’ve done, even the people we like, then we aren’t really aiming for justice.

“Make no mistake, Les Moonves is my guy,” Colbert said. “He hired me to sit in this chair. He stood behind the show when we were struggling to find our voice. He gave us the time and the resources to succeed, and he has stood by us when people were mad at me. I liked working for him. But accountability is meaningless unless it’s for everybody.”

It’s a careful argument that mirrors the debate that many on the left have been having for the past couple of years about those on “their team” who have been credibly accused of predatory behavior, like Bill Clinton and Al Franken. But what Colbert carefully sidesteps is the question of what accountability should look like for Moonves.

That’s because in the months since the accusations against Harvey Weinstein broke last October and the #MeToo movement exploded into the public consciousness, we haven’t really come up with a good model for what accountability does look like for predatory men, beyond making them all go away somewhere where we don’t have to look at them, at least for a little while.

Some of the accused men have lost their jobs and gone off to live in the “exile” of their lavish homes. Many are plotting comebacks. Very, very few, like Harvey Weinstein and Bill Cosby, who we have culturally agreed should be considered beyond the pale, are facing criminal charges or have been convicted of crimes.

In his segment, Colbert suggests that firing the really bad men is justified, but it’s also a pretty dramatic move, possibly as dramatic as the #MeToo movement should aim for. “Those who make peaceful revolution impossible make violent revolution inevitable,” he says, quoting JFK. “So we shouldn’t be surprised that when the change comes, it comes radically. This roar [of men getting fired] is just a natural backlash to all that silence.”

But is firing individual bad actors an act of violent revolution? Is it really the most radical change to which the #MeToo and Time’s Up movements can aspire? What about changing the system that allows those with power to prey on other people with impunity? What about working against a culture that teaches men to see women as sexual trophies, placing more women in positions of power, and, if you want a more concrete action item, maybe changing the way we use nondisclosure agreements?

Accountability is meaningless unless it’s for everybody, and that’s why it can’t just be for individual bad actors. It has to involve changing the entire system that allows those bad actors to flourish.