Old rule of thumb: if you can keep your hand on it, it is probably OK. If you can't, then definitely investigate why.
Reality Check: I am 76 years old. I worked in electronics for all my 45+ year career and I have never seen a rheostat used for a light dimmer in a home or commercial building. Rheostats are variable resistors and resistors are basically a device that converts electrical energy to HEAT in a controlled manner. That is basic to the operation of any resistor. So, if you really are observing rheostats, then the heat is probably part of their basic operation. And they should be designed to dissipate that heat: heat sinks or whatever; some means of dealing with that heat.
BUT, each and every light dimmer that I have ever seen in use in a building's lighting was NOT a rheostat. They use electronic circuits to do this in a more efficient manner. So less heat is produced. Not no heat, but LESS heat.
Earlier ones, several decades ago, and higher power ones used less efficient circuits and they often did have a heat sink that had external fins. Today the fins are often not needed because much less heat is generated. So the age of the light dimmer can be a factor.
All that being said, excessive heat in any electric circuit can and often IS a sign of a problem. If you are not qualified to inspect electric circuits and you do not appear to be, then when you encounter a light dimmer that is excessively hot you should call in someone who is. It may be a simple problem, like loose connections. Or it could be something more serious.
How do you know if the heat is excessive? See my first sentence, above. If you can't put your hand on the heat sink or face plate and keep it there for a minute or so, then it is probably excessive.