Cyanidekid - Good questions.
100-octane fuel from the USA provided the essential foundation for the performance increase. This New York Times article provides some interesting background:-
Who Helped Win Battle of Britain - The New York Times
This makes a nice succinct newspaper story, but inevitably only hints at the bigger picture. Alec Harvey-Bailey points out that switching from 87 to 100 octane and increasing the supercharged boost raised the power output from 1000 to 1300 HP at 8000-9000 ft, but the advantage was
lost at higher altitudes. The majority of fighting was done at 18,000 ft and higher, where the higher octane fuel gave
no benefit. Considerable work was needed to produce blended fuels and to improve supercharger design and power transmission to the superchargers, etc, etc.
An article by Francis Rodwell (Rod) Banks hints at the efforts on the fuel side:-
http://www.wwiiaircraftperformance.org/100-octane/banks-fuel.pdf
Banks was an IC engine fuel/combustion expert, who developed fuels for various world record attempts before the war. He was appointed an Air Commodore for the duration
Stanley Hooker (mentioned earlier by Plannerpower) led the way in transforming the supercharger design at Rolls-Royce.
Then the story gets complicated. Years ago I read a fascinating book which recounted the progressive development of the Merlin to keep one step ahead of the Luftwaffe, but I’ve failed to track it down.
As the design was progressively altered to increase power, testing to destruction was used to find the weaknesses, as cyanidekid says. Obviously this had its limitations compared with in-flight behaviour. Moreover, application of the engine in different types of aircraft gave different problems, requiring quick solutions.
The best information, warts and all, comes from the Rolls-Royce Heritage books, written mostly by engineers who were directly involved. However, I don't think any of them give a full picture of how the power output was progressively increased. From memory,
The Power to Fly by L J K Setright does give a good account of the developments.
'
The Merlin in Perspective – the combat years' by Alec Harvey-Bailey, Rolls-Royce Heritage Trust, 2003) provides fascinating information about design, production, failures, etc, but only has 90 pages of text, the rest of the 210 pages being photos and tables
'
Rolls-Royce Piston Aero Engines – a designer remembers' by A A Rubbra, RRHT, 1990, covers the development of various R-R piston aero engines, packed with drawings and photos.
'
Rolls-Royce and the Mustang' by David Birch, RRHT 1987/1997 gives a fascinating insight into the challenges of matching engine and airframe.
If I get chance I'll post some more information.