Alves, 35, had been given a formal warning after he utilized racist language on the Channel 5 reality program.
"After a further incident, Rodrigo has been removed from the Big Brother House and will not be returning," the broadcaster said.
Alves told The Sun that he had quit the show because he "could not cope without his mobile phone".
"I'm no longer interested in the show or anything to do with it," he said. "Simply because it wasn't for me."
He added that he was also "not coping well" without his family and friends.
During his time on the show, Alves was called into the diary room for using a racist slur during an alcohol-fuelled party with individual contestants.
He was also reprimanded after telling former The Only Way Is Essex star Dan Osborne that he was not attracted to him because he was "too white".
TV watchdog Ofcom received more than 1,000 complaints over his behavior.
The Brazilian-born television personality gained worldwide attention after undergoing dozens of plastic surgeries in an attempt to perfect his appearance.
One of the surgeries included removing four ribs from his rib cage to fit into his clothing better.
As of January, Alves had spent more than £500,000 on 103 cosmetic procedures, 51 of which were plastic surgeries.