You may find courses you liked in HS you hate in university, and vice versa. Sometimes the teacher / prof makes all the difference too. I agree with kcjaysfan, some of the courses I enjoyed the most were physics, and I wasn't a physics major (I took a number of them for fun, seriously). Also, the boundary between some of the upper level math and physics courses kind of gets muddied. Whether you consider a course math or physics is often determined by whether you focus more on the underlying equations or more on the physical interpretation.
Also, remember that each field has many different branches. Some of those branches you may detest, and some you may absolutely love. In HS you don't really get to see them.