But I really don't see a big inanimate lump like a stearing box behaving like a spindly arm with a pivot under the same vibration. But I guess we'll see.
Exactly, what I am saying is that the big inanimate lump that is the steering box is currently transmitting every vibration going from the drivers side through the chassis and setup and hence using an alternative material on the idler will make bog all difference. 
....If there's no play. With play in a bearing in the idler, or something solid used instead of the rubber....
....it'll give an audible warning.

The dimples shown allow some play to gradually creep in. The steering box, when loose, creaks. As we know, thanks to the pikey pensioner. The creak is totally insulted from the driver thanks to the riveted damper at the join of the column to box. Forget the name of it.
Anyway, I don't agree with you.

but let's see what transpires.
I need to get the old idler off and fit the new one first. Not this week anyway.