No. 4: Danny Ferry
Danny Ferry was a longtime Cavaliers player, but the animosity between he and the organization -- and the fan base -- is due more to his time as general manager of the team. Despite having the unquestioned best player in the world in LeBron James, as archietect of the roster Ferry was unable to put a true contending team around him.
He leaned into size and defense over any sense of scoring and shooting, then tried to violently pivot by adding old, injury-prone scorers at the end of LeBron's first tenure. Those teams won a lot of games because of James, but they were unable to support him in the playoffs.
Ferry and the Cavaliers parted ways in 2010 just weeks before "The Decision" and it would be unlikely to see him return.
No. 3: Isaiah Thomas
The Cavaliers and their fans likely do not bear Isaiah Thomas any ill will as a person. It would be hard to blame him personally for what happened once he was traded to Cleveland ahead of the 2017-18 NBA season. Even so, the bitter taste his name leaves would likely to be too much to overcome.
Thomas was the primary win-now return in the Kyrie Irving trade. The Cavaliers were losing the second-best player on a championship team, and Thomas was coming off of an All-NBA season where he received a number of MVP votes. It seemed like it might work out.
Then Thomas failed his physical, but the Cavs traded for him anyway. He was battling a debilitating hip injury that he could not shake, and he was a shell of his former self. The Cavaliers were forced to trade him for pennies on the dollar just to make a change at midseason and limp back to the NBA Finals. It's not personal, but while Thomas hangs around the fringes of the NBA hoping for a comeback, Cleveland is not the place.