You also have to factor in that Cleveland isn't like Toronto which is surrounded by other cities like Mississauga, Brampton, Vaughn, Oshawa, Barrie, Burlington and Hamilton etc. that have a decent size population on top of the population already in Toronto & the suburbs. Also, Toronto is more of a business city, so you'll have a lot of foreigners/people here on business and that have money. Same with tourists who visit Canada, Toronto is a main spot since its the biggest city and with Niagara Falls close by. Tourists who visit the U.S. from overseas tend to visit NYC, Washington DC, Florida, California, Las Vegas etc. They have no incentive to visit Cleveland lol.
With 81 home games, it's tough to sell out every game in Cleveland especially when the weather is usually s*** in April/early May. Even in Toronto, it's not like April/May games are all sold out except maybe on weekends, and still its easy to get a ticket. Tickets are cheap in Cleveland, though still $15 USD per game, plus food/beers would add up if the average joe goes to 3-4 games per week. And beer prices aren't that cheap in Cleveland at the ballpark, especially when doing the USD/CAD conversion. Also, have to factor in that that walk-up crowds in Toronto are significantly larger than in Cleveland. Cleveland's downtown is pretty much a ghost town, and it's not like many people live in condos in the downtown Cleveland area as they do in Toronto.
So like you said, I don't think it has to do with the notion that Clevelanders can't afford to attend games. While I'm sure money does play a small factor, there are other numerous factors at play.