I started out trying to get my BS as a EE but switched to another degree last year. I have taken the first ~2-3 years of the EE classes which aren't a cakewalk by any means. The freshmen courses consist of C programming, digital logic, and DC circuit analysis. After that you get bombarded with EE specific math courses that show you how to deal with the differential equations you get when you start incorporating inductors and capacitors into your circuits then it is off to device electronics i.e. op-amps, BJTs, MOSFETs, diodes, etc. I never liked the math so I changed my degree but I enjoyed understanding how transistors worked and fully grasping how filters worked.
If you get past all the classes with 100+ students and into the small classes where you actually interact with your professors, the classes become more fun albeit still hard. So, if you can make it past all the foundation classes to where you can actually start figuring out things instead of just memorizing things and you find you like it, you might want to try for a Ph. D.