They were top 5 in attendance, but they were also top 5 in payroll those years. Ownership stopped spending on the team, which means the fans left. They went from a top 5 payroll to a bottom 5 in 2 years. Hard to put a team on the field that people want to pay to watch when you have a bottom 5 payroll to work with.
It's almost like more spending = more fans. Weird huh