Air-to-water heat pumps are regarded as a fundamental part of many proposed climate action plans in heating dominated climates. However, individual studies have concluded that this technology generally underperforms in practice when compared to its rated performance. This study aims to harmonise these conclusions and propose reasons for this reported underperformance. The European standards used to rate the performance of heat pumps are first reviewed and deconstructed, a review and analysis of all past research that report air-to-water heat pump performance is then conducted. Finally, a novel in-situ monitoring study of an air-to-water heat pump in Ireland is presented. Of the 378 heat pumps reviewed, it was found that the average seasonal performance was 2.59, significantly lower than typical product rated performances. Three reasons for this underperformance include, misrepresentative weather data for certain locations, unaccounted for losses in the distribution pipes and unaccounted for effects of cycling. The measured seasonal performance factor of the case study heat pump monitored in this paper was 2.49 for space heating. This 40% reduction in performance, when compared to the product rated performance of 4.12, was attributed to the three reasons identified. Because these reasons are largely due to inadequate installation practice, environmental policy should focus not just on the quantity of these technologies to be installed but critically also on the quality of the installation itself.