I would have to say that it's only hard driving without power steering if the vehicle in question is supposed to have it and it isn't operating correctly for some reason.
A vehicle with no pwer steering which never had it to begin with, IMO, no bid deal.
Now bear in mind I am 6'2" and go about 205-210 so possibly were I a smaller fella it might be different...