The PhysioNet/CinC Challenge

Read about the 15th annual PhysioNet/CinC Challenge, now in progress:
Robust Detection of Heart Beats in Multimodal Data

Background

While planning CinC 2000, local hosts Roger Mark and George Moody proposed to organize an activity that would make effective use of their newly-established PhysioNet web site to stimulate rapid progress on an unsolved problem of practical clinical significance. A timely contribution of data made it possible to create the first PhysioNet/CinC Challenge, which attracted the attention of more than a dozen teams to the subject of detecting sleep apnea from the ECG. Their efforts were broadly successful, they discussed their findings at CinC 2000, and an annual tradition was born.

PhysioNet offers free access via the web to large and growing collections of recorded physiologic signals and related open-source software. Originally established under the auspices of the NIH's National Center for Research Resources, PhysioNet has been funded since September 2007 under a cooperative agreement with the NIH's National Institute of Biomedical Imaging and Bioengineering (NIBIB), and with the NIH's National Institute of General Medical Sciences (NIGMS).

In complementary ways, PhysioNet and Computing in Cardiology catalyze and support scientific communication and collaboration between basic and clinical scientists. The annual meetings of CinC are gatherings of researchers from many nations and disciplines, bridging the geographic and specialty chasms that separate understanding from practice, while PhysioNet provides on-line data and software resources that support collaborations of basic and clinical researchers throughout the year. The annual PhysioNet/CinC Challenges seek to provide stimulating yet friendly competitions, while at the same time offering both specialists and non-specialists alike opportunities to make progress on significant open problems whose solutions may be of profound clinical value. The use of shared data provided via PhysioNet makes it possible for participants to work independently toward a common objective. At CinC, participants can make meaningful results-based comparisons of their methods; lively and well-informed discussions are the norm at scientific sessions dedicated to these challenges. Discovery of the complementary strengths of diverse approaches to a problem when coupled with deep understanding of that problem frequently sparks new collaborations and opportunities for further study.

A new challenge topic is announced each year on the Challenge page at PhysioNet. The PhysioNet team assembles and posts the raw materials needed to begin work. Usually, these raw materials consist of a collection of data to be analyzed; the analyses are provided for a subset of the data (the "learning set") in each case, and the challenge is to analyze the remaining data (the "test set").

What will be the topic of the next challenge? It might be image analysis, or simulation, or forecasting.... An ideal challenge problem is interesting, clinically important, and possible to study using available materials that have not been widely circulated previously. Moreover, there must be an objective way to evaluate the quality of a challenge entry (for an analysis problem, this usually means there must be a known set of correct analyses of the data, i.e., a "gold standard" against which entries can be compared). You are invited to submit your ideas for future challenge topics to the Challenge organizer, George Moody; suggestions accompanying relevant data are particularly welcome.

PhysioNet/CinC Challenge Awards

Challenges are open to all. An important milestone for participants is the deadline for submitting abstracts for CinC, which is 15 April each year (this date is new in 2014, and is earlier than in previous years). Those wishing to qualify as official entrants, with eligibility for awards, must submit an abstract describing their work as well as an entry for scoring by about one week before the abstract deadline. A limited number of revised entries may be submitted between the abstract deadline and the final challenge deadline in early September. Eligibility for awards also requires participants to present their work in a scientific session of CinC. See the Challenge pages on PhysioNet for deadlines and rules for this year's competition.

Most Challenges are presented as two events (often a narrowly-defined question and a more general one), or sometimes more. In most events, an award is offered for the best solution obtained by any eligible participant, and another award for the best open-source solution (i.e., the best solution for which the software written to obtain the solution is contributed by its author in open-source form to support further study). If an open-source solution is best overall, its author wins both awards for the event. Some events involving hidden data sets are restricted to open-source entries only. The winners receive their awards during the final plenary session on Wednesday afternoon. Follow the links below for details about previous challenges.

PhysioNet/CinC Challenge Award Winners

2013: Noninvasive Fetal ECG

Events 1 and 2:
Maurizio Varanini, Gennaro Tartarisco, Lucia Billeci, Alberto Macerata, Giovanni Pioggia, and Rita Balocchi, CNR ICP, Pisa, Italy (first place; also third place in events 4 and 5)
Rui Rodrigues, Universidade Nova de Lisboa (third place, event 1)
Luigi Yuri Di Marco, Alberto Marzo, and Alejandro Frangi, University of Sheffield (third place, event 2)
Event 3:
Piotr Podziemski and Jan Gierałtowski, Warsaw University of Technology, Warsaw, Poland (first place; also second place in events 1 and 2)
Events 4 and 5:
Fernando Andreotti, Maik Riedl, Tilo Himmelsbach, Daniel Wedekind, Sebastian Zaunseder, Niels Wessel, and Hagen Malberg, Dresden University of Technology, Dresden, Germany (first place)
Jukka A Lipponen and Mika P Tarvainen, University of Eastern Finland (second place)
Joachim Behar, Julien Oster, and Gari Clifford, University of Oxford, UK, obtained the top scores in events 1 and 2 while participating unofficially.

2012: Predicting Mortality of ICU Patients

Event 1:
Alistair Johnson, Nic Dunkley, Louis Mayaud, Athanasios Tsanas, Andrew Kramer and Gari Clifford, University of Oxford, UK
Event 2:
Luca Citi and Riccardo Barbieri, MIT, Cambridge, Massachusetts, USA, and Massachusetts General Hospital, Boston, Massachusetts, USA

2011: Improving the Quality of ECGs Collected using Mobile Phones

Event 1:
Henian Xia, Gabriel Garcia, Joseph McBride, Adam Sullivan, Thibaut De Bock, Jujhar Bains, Dale Wortham, and Xiaopeng Zhao, University of Tennessee, Knoxville, Tennessee, USA (first place)
Gari Clifford, Daniel Lopez, Qiao Li, and Iead Rezek, University of Oxford, UK (second place)
Event 2:
Henian Xia, Gabriel Garcia, Joseph McBride, Adam Sullivan, Thibaut De Bock, Jujhar Bains, Dale Wortham, and Xiaopeng Zhao, University of Tennessee, Knoxville, Tennessee, USA (first place)
Benjamin Moody, MIT, Cambridge, Massachusetts (second place)
Event 3:
Dieter Hayn, Bernhard Jammerbund, Günter Schreier, AIT, Graz, Austria (first place)
Václav Chudáček, Lukáš Zach, Jakub Kužílek, Jiří Spilka and Lenka Lhotská, Czech Technical University, Prague, Czech Republic (second place)

2010: Mind the Gap!

Event 1 (general and open-source):
Rui Rodrigues, Universidade Nova de Lisboa, Portugal
Event 2 (general and open-source):
The team of Adam Sullivan, Henian Xia, Joseph McBride, and Xiaopeng Zhao, University of Tennessee, Knoxville, Tennessee, USA

2009: Predicting Acute Hypotensive Episodes

Event 1 (general and open-source):
Xiaoxiao Chen, Michigan State University, Lansing, Michigan, USA
Event 2 (general and open-source):
Jorge Henriques and Teresa Rocha, University of Coimbra, Coimbra, Portugal

2008: Detecting and Quantifying T-Wave Alternans

General:
Giovanni Bortolan, ISIB-CNR, Padova, Italy, and
Ivaylo Christov, Bulgarian Academy of Sciences, Sofia, Bulgaria
Open-source:
Alexander Khaustov, Incart, St. Petersburg, Russia
Jubair Saieed, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh, obtained the top score overall while participating unofficially in the open-source division.

2007: Electrocardiographic Imaging of Myocardial Infarction

Event 1:
Hamid SadAbadi, KN Toosi University, Teheran, Iran
Event 2:
Mohamed Mneimneh, Marquette University, Milwaukee, Wisconsin, USA
Event 3 (tie):
Masood Ghasemi, KN Toosi University, Teheran, Iran
Mohamed Mneimneh, Marquette University, Milwaukee, Wisconsin, USA
Hamid SadAbadi, KN Toosi University, Teheran, Iran
Event 4 (tie):
Mohamed Mneimneh, Marquette University, Milwaukee, Wisconsin, USA
Hamid SadAbadi, KN Toosi University, Teheran, Iran

2006: QT Interval Measurement

Division 1 (manual/semi-automated):
Mariano Llamedo Soria, Universidad Tecnologica Nacional FRBA, Buenos Aires, Argentina
Division 2 (automated, general):
Dieter Hayn, ARC Seibersdorf Research GmbH, Graz, Austria
Division 3 (automated, open-source):
Yuri Chesnokov, Cambridge University, Cambridge, UK

2005: The First Five Challenges Revisited

Richard Povinelli, Marquette University, Milwaukee, Wisconsin, USA
C Raab, University of Potsdam, Potsdam, Germany

2004: Spontaneous Termination of Atrial Fibrillation

Event A (general):
Dieter Hayn, ARC Seibersdorf Research GmbH, Graz, Austria
Event A (open-source):
Maurizio Varanini, CNR-ICP, Pisa, Italy
Event B (general and open-source):
Federico Cantini, CNR-ICP, Pisa, Italy
Simona Petrutiu, a member of the research group at Northwestern University that contributed the data, obtained the top scores in both events while participating unofficially.

2003: Distinguishing Ischemic from Non-Ischemic ST Changes

Philip Langley, University of Newcastle upon Tyne, Newcastle, UK

2002: RR Interval Time Series Modeling

Event 1:
DC Lin, Ryerson University, Toronto, Canada
Event 2:
Albert C-C Yang, National Yang-Ming University, Taipei, Taiwan

2001: Predicting Paroxysmal Atrial Fibrillation

Event 1:
Günther Schreier, ARC Seibersdorf, Graz, Austria
Event 2:
Günther Schreier, ARC Seibersdorf, Graz, Austria
Wei Zong, a member of the PhysioNet team at MIT, obtained the top score in event 2 while participating unofficially.

2000: Detecting Sleep Apnea from the ECG

Event 1:
Murray Jarvis, Caltech, Pasadena, California, USA
Event 2:
James McNames, Portland State University, Portland, Oregon, USA