OBJECTIVE: Research into echocardiography (echo) during cardiac arrest has suffered from methodological flaws that limit aggregation of findings. We developed and validated a novel image rating scale for qualitative analysis of echo images obtained during resuscitation. METHODS: A novel 5-point ordinal rating scale was developed and validated using recorded echo images from 145 consecutive cardiac arrest patients. Recorded echo images were reviewed in a blinded fashion by investigators experienced in cardiac arrest echo, and image quality was rated using this scale. Cardiac activity was subsequently classified as no activity, disorganized activity and organized activity. The primary outcome was inter-rater agreement using the image quality rating scale. Secondary outcome was the qualitative evaluation of the type of cardiac activity. RESULTS: A total of 235 ultrasounds were analyzed by study investigators using the image quality rating scale. The overall image quality agreement between reviewers using the scale was good with a weighted kappa of 0.65. Agreement for image quality in subxyphoid images was greater than in parasternal images (0.65-0.52). Echo analysis of cardiac activity showed no activity (33%), disorganized activity (18%), and organized activity (49%). Agreement was great for presence or absence of "cardiac activity" and "organized cardiac activity" with a kappa of 0.84 and 0.78. CONCLUSIONS: A novel image quality rating scale for echo during cardiac arrest demonstrates substantial agreement between reviewers. Agreement regarding the presence or absence, as well as the organization of cardiac activity was substantial.