Marlon Wayans reveals why his family left “In Living Color”

Marlon Wayans reveals why his family left him In vibrant color at the height of his success.

The sketch comedy show, created by his brother Keenen Ivory Wayans, premiered on Fox in 1990 and ran for five seasons until 1994.

While the show served as a launching pad for various A-list celebrities such as Jennifer Lopez, Jim Carrey and Jamie Foxx, it ultimately proved to be a family affair, as Marlon (52) and Keenen (66) were joined by their siblings Damon Wayans, Kim Wayans and Shawn Wayans.

Marlon shared during an appearance on Wednesday’s edition of Club Shay Shay Podcast that his family left the show after the fourth season because “it hurt Keenen's wallet.”

“I remember when we went In vibrant color because Fox was syndicating the show and it was hurting Keenen's wallet. Keenen said, “Nah, you're not going to do that to me,” Marlon recalled. “My family said, 'Screw that money.' And we all went In vibrant colors.”

The comedian recalled the 1992 Christmas episode of “Driving Miss Schott,” in which he and his siblings decided to protest the show by wearing black sunglasses while Foxx, 56, sang “This Christmas.”

“You will not f— my brother. We are family. If you touch one, you touch them all,” Marlon said of her decision.

The PEOPLE Puzzler crossword puzzle is here! How fast can you solve it? Play now!

While the actor states that they were offered “big checks” to stay on the show, he says his family decided to “support” their brother and exit.

“That's why season five got weird. There was Biz Markie. God bless Chris Rock, [but] He came at the wrong time. As brilliant as he is, he couldn't save it. Because the eye of the show is gone,” Marlon said.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

After leaving the series, Marlon said he was “broke” and had “only $700 in the bank and $900,” which motivated him to The Wayans Bros., which aired for five seasons from 1995 to 1999.