Announcement
There is a planned maintenance for storage servers connected to Codalab.lisn.fr and codabench.org in Paris-Saclay.
Service will be interrupted from March, 28th 8:00 CET to approximately 15:00 CET.
Portal will be accessible read-only: NO submission can be processed. We apologize for inconvenience.
your Codalab Team
Find benchmarks that pique your interest! A benchmark allows you to test new algorithms against reference datasets OR (inverted benchmark) submit challenging data to reference algorithms.
Organize a benchmark on Codabench. Start with our tutorial.
Interested in joining the development team? Join us on Github or contact us directly.
Codabench is a platform allowing you to flexibly specify a benchmark. First you define tasks, e.g. datasets and metrics of success, then you specify the API for submissions of code (algorithms), add some documentation pages, and [CLICK] your benchmark is created, ready to accept submissions of new algorithms. Participant results get appended to an ever-growing leaderboard.
You may also create inverted benchmarks in which the role of datasets and algorithms are swapped. You specify reference algorithms and your participants submit datasets.
CodaLab Competitions is a powerful open source framework for running competitions that involve result or code submission. You can either participate in an existing competition or host a new competition.
Most competitions hosted on Codalab are machine learning (data science) competitions, but Codalab is NOT limited to this application domain. It can accommodate any problem for which a solution can be provided in the form of a zip archive containing a number of files to be evaluated quantitatively by a scoring program (provided by the organizers). The scoring program must return a numeric score, which is displayed on a leaderboard where the performances of participants are compared.
Codalab was created in 2013 as a joint venture between Microsoft and Stanford University. Originally the vision was to create an ecosystem for conducting computational research in a more efficient, reproducible, and collaborative manner, combining worksheets and competitions. Worksheets capture complex research pipelines in a reproducible way and create "executable papers". Currently, we are developing the V2 of Codalab, which will be able to organize benchmarks.
Some competitions have been organized using worksheets, but the competition platform and the worksheet platform have both a large user base and can be used independently. In 2014, ChaLearn joined to co-develop Codalab competitions. Since 2015, University Paris-Saclay is community lead of Codalab competitions, under the direction of Isabelle Guyon, professor of big data. Codalab is administered by CKCollab and the LRI staff.
Codalab is used actively in research. In 2019/2020, 400 new challenges were launched. Recent popular challenges organized with Codalab include the COVID-19 retweet prediction challenge, the ECCV 2020 ChaLearn LAP Fair face recognition challenge, the 2020 DriveML Huawei Autonomous Vehicle Challenge, and high profile challenges include the 2 million Euro prize of the EU, organized by the See.4C consortium, the CIKM AnalytiCup 2017, which attracted 493 participants, MSCOCO (633 participants) and the ChaLearn AutoML challenge 2017 (687 participants).
Since 2016, Codalab offers the possibility of organizing machine learning challenges with code submission. The simplest machine learning challenges require only the submission of results, which are compared to a solution (or key) by a scoring program. Result submission challenges are less computationally expensive than code submission challenges. However, they offer less possibilities. In particular, code submission allows conducting fair benchmarks by executing submitted code in the same condition for all participants.
Codalab has been providing free resources for challenge organizers who want to run high impact events, within a pre-approved agreed upon budget. New since version 1.5: organizers can hook up their own compute workers to the backend of Codalab to redirect the code submissions, enabling growth to big data competitions running at the expense of the organizers. For very special dedicated projects, Codalab can be customized since it is an open source project.
August 2020: Codalab exceeds 50,000 users, 1000 competitions (over 400 in the last year), and ~600 submissions per day!
July 2020: We launched a new Learning to Run a Power Network competition, in collaboration with ChaLearn and RTE. We have a robustness and an adaptability track. This is an NeurIPS 2020 competition.
May 2020: We released a new application to help instructors use challenges in the classroom and grade them called Chagrade.
April 2020: The NeurIPS AutoDL challenge ended. But the series of challenges on Automated Deep Learning, in collaboration with ChaLearn, Google Zurich, and 4Paradigm continues with AutoSeries and AutoGraph.
June 2019: We organized a data science bootcamp at Data Science Africa 2019 in the form of a challenge to detect Malaria parasites in microscope images.
September 2018: The LAL and CERN are organizing a challenge to reconstruct particle trajectories in high energy physics detectors. After the success of the first phase with result submission only, a second phase with code submission will be run on Codalab. TrackML is an officially selected challenge of the NIPS 2018 conference.
August 2018: Codalab is proud to host the third challenge on Automatic Machine Learning: Lifelong Machine Learning with drift. AutoML3 is an officially selected challenge of the NIPS 2018 conference.
February 2018: 2 million Euro Big Data EU prize powered by Codalab.
February 2018: Isabelle Guyon presents Codalab at the newly formed Institute of Convergence DataIA
January 2018: Paris-Saclay master students create challenges for L2 students.
January 2018: Paris-Saclay instructors create reinforcement learning homework.
December 2017: Codalab exceeds 10000 users with 480 competitions (145 public)
December 2017: Codalab presented at the Challenges in Machine Learning workshop [slides].
November 2017: Explore the new features: scale up your code submission competition with your own compute workers (full privacy, dockers); organize RL challenges and hook up simulators providing data on demand (with your own "ingestion program"); use the ChaLab wizard to create competitions in minutes. Thanks!