Neither tubes nor transistors, on their own, are free of distortion. They both have non-linear characteristics. It is the circuits that they are employed in that either correct this distortion or, as a design compromise, permit it to get to the output. One way of limiting this distortion is by using only a small part of the overall operation region of the device. So a tube amplifier or a transistor amplifier that is only handling a very low level signal can have a lot less distortion than one which is trying to drive the speakers or headsets which requires higher Voltages and/or currents and therefore must use a larger part of that overall operating region of the device.
Another way for controlling distortion is with feedback. The problem with feedback is you need a higher gain amplifier which is throttled down by the feedback signal. And that higher gain must exist over the entire frequency range of the device. This is why one of our posters above mentioned the ease with which op amp circuits can so easily be almost distortion free. But an op amp uses many transistors, not just one or two. This is possible at a reasonable price with transistors but could be quite expensive for a similar circuit made with tubes (not to mention the need for tons of air conditioning to handle the heat).
Other circuit designs have been used. These try to take advantage of the characteristics of the tubes or transistors to produce low distortion. These will be somewhat of a compromise and each will have it's own "sound". Again, since transistors are smaller, less expensive, and consume far less power than tubes, the transistor circuits can be a lot more economical for a given (low) level of distortion. Any level of low distortion that could be achieved with transistors could also be achieved with tubes: it would just cost a heck of a lot more. Just imagine a tube amplifier with 50 or 100 tubes.
But there is no magic in either device, just a practical matter of where the price vs. distortion compromise falls. And the types of distortion that will come along with that practical design compromise.
As for headset impedance, the most important thing would be to match the headphone impedance with the amplifier that is driving them. That will minimize the distortion in that, output circuit. As a practical matter modern, low output impedance amplifiers, which usually employ a lot of feedback, will drive any impedance headphones with little discernible difference. The audio nuts will argue about this, but they will not ever accept the results of a double blind test. I have tried and they always find some supposed flaw when the less expensive alternative proves to be as good as or even better than what they KNOW is better.