That's kind of my take as well, MM, but it's lots cheaper to swap out the fluid than pay for a rack inspection. If it doesn't improve it, you're only out the cost of the fluid and a few minutes' time. And going back to the question of scrupulous mechanics, unless you
already have a trusted indie, you roll into a shop and say "Hey, can you inspect the rack? My steering feels heavy" there are all too many cretins ready to take a quick look and come tell you "yep, we looked at it, you need a new rack" knowing that when they change it they'll swap the fluid, etc. and most of the time they'll be okay. If it turns out the pump was really the culprit they can say "well, we were concerned about that because that can kill the rack; we hoped it wouldn't and were trying to save you money, but that's life". And most people would probably fall for that nonsense.
To our OP, if you haven't changed the fluid, personally I'd start there. If you have already had that serviced, then go to your trusted mechanic and have the steering and suspension inspected and evaluated. Also, you didn't mention if this was worse when the car/climate was cold and improved when warm, if so that is a bit stronger pointer at the rack.
Take care and enjoy the ride,
Greg