If you are installing these correctly, the hole size is critical to the strength of the assembly.
Depending on the particulars of the part number, it may have a slightly enlarged section of the shank, or just a very carefully sized parallel side to it. But in any case, the hole size is usually reamed to a decent tolerance, and sometimes cold worked.
The key is used to stop it from turning as it begins it's trip down the hole. Once in the hole, if the hole is correctly sized, the fastener should not rotate.
The tolerances for the hole will be specific to the specific type of fastener you are dealing with, but for non-aircraft, non-critical fastenings, a reasonable interference fit is what is required. Not so critical if using a regular nut, but if using the proper collars, too tight a fit will cause the collar to break before the pin is seated. Not so good.
Cheers
Trev