Crowdsourcing as a Novel Method for the Evaluation of Postoperative Outcomes in Unilateral Cleft Lip Repair
Eugene Oh, MD, PhD1, Raymond Tse, MD2.
1University of Washington, Seattle, WA, USA, 2Seattle Children’s Hospital, Seattle, WA, USA.
PURPOSE: Lack of convenient and reliable methods to grade esthetic outcomes limits the ability to study results and optimize treatments. Crowdsourcing involves soliciting contributions from a large group to achieve a greater task. We hypothesized that Crowdsourcing could be used to reliably grade esthetic outcomes following unilateral cleft lip repair.
METHODS: 50 de-identified photographs of 8-10 year old subjects were used (46 unilateral cleft lip, 4 control). Crowds performed pairwise comparisons of the sample set to produce a rank order for nasal appearance. N=50 comparisons per image was used to generate ELO scores for ranking. Asher-McDade scores were obtained for each subject (n=38). Two distinct surveys of each method were conducted. Additionally, expert surgeons were asked to perform the same tasks on a smaller set.
RESULTS: 2500 anonymous lay-person comparisons were obtained within 2-4 hours for each round. Rank order was reproducible with a correlation coefficient of 0.892. Greater than 1900 Asher-McDade scores were generated within 2 hours and showed excellent inter-run consistency (correlation coefficient=0.988).
Comparing ELO rank and Asher-McDade score showed strong correlation=-0.920. Furthermore, Crowdsourced data correlated well to expert assessments with correlation coefficients of 0.931 (ELO) and 0.950 (Asher-McDade). Expert analysis required months and ultimately was not fully completed.
CONCLUSION: Large numbers of lay-person assessments generated aggregate rankings and ratings of esthetic outcomes that were highly reliable and correlated well to expert assessments. Despite smaller sample set, data collection took much longer for our expert cohort. Crowdsourcing is a convenient, rapid, and reliable means to assess postoperative outcomes.
Back to 2016 Annual Meeting Posters