Contact
Save the world, one flop at a time.
An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing
1. How to make a submission
Please find the given CodaLab Competition Link CodaLab Competition Link and submit the corresponding file as follows:i). A README.txt file including your name, your affiliation, your advisor's name(if applicable) and contact information.
ii). A one page Intro.pdf briefly talking about your method for model design / pre-training. You code should be open sourced and link attached in the pdf.
iii). For pre-training phase, please submit a CSV file result.csv with 4 columns, the time and 3 dev-set result column for CoNLL 2003, SST-2 and MNLI datasets after certain time. For fine-tuning phase, please submit a CSV file result.csv with 6 columns, 3 time and dev-set result column for CoNLL 2003, SST-2 and MNLI datasets. For inference phase, please just submit a CSV file result.csv with 3 columns with different inference time on the CoNLL 2003, SST-2 and MNLI datasets.
2. How are the data estimated for table 1?
The data are estimated using the number given in the papers, for example, the RoBERTa model is pre-trained with 1024 Tesla V100 GPUs within 1 day for 100k iteration. The cost is estimated using the price at that time, for example, $8 for 4 TPU units an hour.
3. Can I have a sample submission?
To appear here.
@misc{zhou2020hulk,
title={HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing},
author={Xiyou Zhou and Zhiyu Chen and Xiaoyong Jin and William Yang Wang},
year={2020},
eprint={2002.05829},
archivePrefix={arXiv},
primaryClass={cs.CL}
}