Low-Power Image Recognition Challenge (LPIRC) June 5 2016 in Austin Texas (with Design Automation Conference)
LPIRC 2016 Winners
- Chinese Academy of Science: first prize in Track 1, second prize in Track 1,first prize in Track 3, highest accuracy, lowest energy in Track 1
- Tsinghua University: third prize in Track 1
- Peking University: Special Award "Never Give Up"
Sponsors: IEEE Rebooting Computing, IEEE Council on superconductivity, IEEE Council on Electronic Design Automation, Nvidia, and Xilinx.
News:
- Xilinx will support participants by providing Xilinx's products as the computing platforms. People interested in this opportunity may contact Aaron Behman (aaron.behman@xilinx.com) for more details.
- Registration at DAC is open. Each participant needs to register at two different sites: DAC workshop (and pay the registration fee, http://www2.dac.com/events/eventdetails.aspx?id=200-254) and downloading training images (http://vision.cs.unc.edu/LPIRC/login.php)
- LPIRC 2016 will add a third track. The images will be displayed on a computer screen (Dell UltraSharp U2414H 24-inch) and the teams need to use cameras to obtain the images. The other two tracks use a computer network to obtain the images (same as 2015).
- LPIRC 2016 will adopt the "tournament style" in Tracks 1 and 2: two teams compete. The winner receives two points. If they tie, both teams receive one point. Please see the "Prizes" section below about the details.
- Participants in track 3 compete by only the absolute scores.
- The US Government announced the Nanotechnology-Inspired Grand Challenge for Future Computing.
- LPIRC 2016 organizing committee: Co-Chairs: Yung-Hsiang Lu (Purdue) and Alex Berg (UNC); Committee: Daniel Hammerstrom (DARPA), Greg P Leeming (Intel), David Atienza Alonso (EPFL), Massimo Alioto (NUS), David Pan (University of Taxes). Students (doing the "real work"): Wei Liu (UNC) and Rohit Rangan (Purdue)
- A special session in ICCAD (International Conference on Computer Aided Design) for LPIRC on 2015/11/02 in Austin Texas.
- Questions? Please send email to lpirc@ecn.purdue.edu
Motivation
Many mobile systems (smartphones, electronic glass, autonomous robots) can capture images. These systems use batteries and energy conservation is essential. This challenge aims to discover the best technology in both image recognition and energy conservation. Winners will be evaluated based on both high recognition accuracy and low power usage.
Image Recognition
Image recognition involves many tasks. This challenge focuses on object detection, a basic routine in many recognition approaches. The following two examples illustrate the task. In the first example, there are two objects: a bird and a frog. In the second example, there are several objects: cars, persons, motorcycle, and a helmet. The training and validation data for LPIRC comes from the ImageNet Large Scale Visual Recognition Challenge detection competition. The test data will be specific to LPIRC.
Many mobile systems (smartphones, electronic glass, autonomous robots) can capture images. These systems use batteries and energy conservation is essential. This challenge aims to discover the best technology in both image recognition and energy conservation. Winners will be evaluated based on both high recognition accuracy and low power usage.
Image Recognition
Image recognition involves many tasks. This challenge focuses on object detection, a basic routine in many recognition approaches. The following two examples illustrate the task. In the first example, there are two objects: a bird and a frog. In the second example, there are several objects: cars, persons, motorcycle, and a helmet. The training and validation data for LPIRC comes from the ImageNet Large Scale Visual Recognition Challenge detection competition. The test data will be specific to LPIRC.
Competition Tracks
- Two participants will compete at the same time in Track 1 and 2.
- For Track 1, computation offloading to another device or server is not allowed.
- For Track 2, computation offloading is allowed.
- For Track 3, only one team participates each time. The team can choose offloading or not.
Important Dates
- 2016/06/06 Announcement of the winners.
- 2016/06/05 Competition in Austin as a one-day workshop in Design Automation Conference
- 2016/04/30 Reference code for interface with the referee program. To obtain the source code (in Python), please go to https://github.com/luyunghsiang/LPIRC.git. The README explains how to use the program. You can retrieve the version used for 2015 for reference. The referee program for 2016 is different for handling the tournament style.
- 2016/04 Registration for Design Automation Conference is open.
- 2015/01 Announcement of the power meter. The power meter is Yokogawa WT 300. The network router is Nighthawk AC1900 Dual Band Wi-Fi.
- 2014/11 Training and validation data available
Organizers:
- Yung-Hsiang Lu (Purdue University)
- Alex Berg (University of North Carolina at Chapel Hill)
- Wei Liu (UNC)
- Rohit Rangan (Purdue)
Track 3
The display and the camera are enclosed by a black box as shown below. The photographs show the top of the box removed. The camera in this photographs is for demonstration purpose and participants must bring their own cameras.
|
|
|
The network router. The wireless SSID will be hidden and the password will be given to you before June 5. You may use the wired network. Please be aware that this system connects to only the referee program and do not connect to the Internet.
|
AC connection. If your system takes 110V AC, simply plug into the AC socket.
This is an example of measuring AC. The power adapter is the power load and the power adapter is connected to the system directly. The voltage shown in the power meter is 121.18V.
This is a different configuration. The power adapter is plugged into a wall socket. The DC from the adapter is the input to the power meter (11.816V).
If your system uses DC (the output from the power adapter in this case), the DC output is plugged in to the input of the power meter.
The other end (the output of the power meter) becomes the power source of the system. This Nvidia board is for demonstration purpose only. It does not mean you need to use an Nvidia board.
The left is a universal connector.
FAQ (Frequently Asked Questions):
Q: How will be energy be measured?
A: Please see the photographs above.
If your system is connected to 110V AC (standard wall socket in USA), the system will be plugged into a specially made socket.
We have universal connectors taking two wires from your power source (such as a battery) and providing two wires out to your system.
Q: When will the travel grants be given?
A: At the beginning of the challenge (around 850-910AM) on June 5, 2016.
Q: On June 5, do I have a chance testing whether my system can communicate with the referee?
A: Yes. Before the challenge starts, all teams can test whether their systems can communicate with the referee.
Q: May I test my system during the challenge, while another team is running?
A: No. During the challenge, only the competing teams' systems can communicate with the referee. This is to prevent interference.
Q: Do I have another chance testing the communication before the ten minutes start?
A: Yes. Your system can send a request to retrieve the document (/help in the URL).
Q: When does power measurement start?
A: When your system logs in. It is advised that your system starts requesting images as soon as the system logs in.
Q: Will all teams receive the same images?
A: Yes, but the orders may be different. The images are divided into small groups. The order of images inside each group may be different.
Q: If the image orders are different, would that be fair? Would some images be harder than the others?
A: The group sizes are small. If a team processes many groups, the differences should be negligible.
Q: Why are the image orders different for different teams?
A: Some teams have multiple solutions. If the image orders are the same, one of these teams might use the images from the first solution, perform off-line processing, and obtain advantage in the second time. This would be unfair. If the orders are different, it would be more difficult to perform off-line processing and this will improve fairness.
Q: What is the group size?
A: The group sizes will not be announced. In fact, the groups may have different sizes.
Q: How much time do I have for setup?
A: We expect that 10 minutes should be sufficient. Thus, each team (or each solution of a team that has multiple solutions) should take approximately 20 minutes total.
Q: How is the team order determined?
A: The team order will be determined by drawing lottery before the challenge starts. There will be two rounds of lottery. The first round determines the order of the second round. The second round determines the order of the challenge.
Q: Do I have to stay when the other teams are running?
A: You are free to leave anytime. If it is your turn and you are absent, you lose that round.
Q: When will the entire challenge finish?
A: It is hard to predict since the setup time may vary.
Q: Will we know which team is running?
A: Yes. A sign will be posted showing the currently running team.
Q: When will power measurement stop?
A: Ten minutes after your system logs in.
Q: Should my system send all answers to the referee at once, or one at a time?
A: You decide.
It is advised that your system sends answers to the referee often. The referee program will refuse to communicate with your system ten minutes after your system logs in. If your system sends all answers at once near the end of the ten minutes, it is possible that your system does not finish sending all results within the ten minutes.
Q: Is the ten-minute rule enforced by hand (using a stopwatch)?
A: This rule is enforced by the referee program.
Q: My team has several solutions. Can we try all of them and use the highest score?
A: For fairness, each registration allows only one solution. If your team has multiple solutions, please have one registration for each solution.
Q: Are there only 5,000 images? What happens if we finish 5,000 images within 10 minutes?
A: You can shut down the system to reduce the energy consumption. Alternatively, your system may log out and this will stop the power meter. We encourage you to do both (shutdown and log out).
Q: Do I have to disassemble the system so that you can measure power?
A: You need to provide a way to measure the power.
Q: I do not want to break my system.
A: We can measure the power consumed by a power adapter that charges your system. Please be aware that the power adapter consumes additional power.
Q: What is the maximum current my system can draw?
A: The power meter's upper limit is 20A. For DC (up to 32V), a fuse of 15A is used. For AC (up to 250V), a fuse of 2A is used.
Q: Does the power meter supply power?
A: No.
Q: Does my system have to communicate with the referee system through HTTP?
A: Yes.
Q: Can my system communicate with the referee program when another team is running?
A: No. The network router allows only the running teams.
Q: Do you provide a power supply?
A: No.
Q: How will the travel grants be determined?
A: It depends on the number of participants. If there is enough fund, every participant will receive a travel grant. If the fund is insufficient, earlier registrations will have higher priority.
Q: Can an observer (without paying the registration fee) receive a travel grant?
A: Due to the limited fund, only the people that register and pay the fee are eligible for the travel grants.
Q: If I register and pay the fee but cannot attend on June 5, can I get refund?
A: Please contact the conference.
Q: Can I bring special hardware?
A: Yes.
Q: Can I use commodity hardware (such as mobile phone, laptop, or desktop)?
A: Yes.
Q: Can I combine special hardware and commodity hardware?
A: Yes.
Q: Are companies allow to compete?
A: Yes, of course.
Q: Are participants outside USA allowed?
A: Yes, of course.
Q: The interface program is written in Python. Does my program have to use Python?
A: You can use any language, as long as it communicates with the referee program using HTTP.
Q: I want to spend a lot of money to build the best system and win. Is there any restriction on how much I can be spend?
A: You can spend as much money as you prefer.
Q: Must the system use DC power from batteries?
A: There is no such restriction. Your system may draw AC power from a wall socket.
Q: Is it possible that an image contain multiple objects?
A: Yes, that is possible
Q: Can I observe without competing?
A: Yes.
Q: Is there any financial support?
A: Yes, travel grants are available. Please visit the conference web site (www.dac.com) for application.
Q: How does my program communicate with the system?
A: A program template will be available for the interface. The interface will be HTTP based. There will be a wireless/wired router.
Q: How should I write my program so that it can communicate with your program?
A: Please structure your program so that it has two components: the recognition engine and the interface. The interface will be released before the competition.
Q: How can my computer connect to yours, get data, and send results?
A: An intranet will be set up. Both wired (RJ 45) and wireless (Wifi) will be available. For track 3, the images are provided through a display (thus, you need to bring a camera). For all tracks, the answers are sent to the referee through the network.
Q: How do I know which Wifi network to connect to?
A: It will be announced before the competition. To prevent interference, the Wifi for this competition will make the SSID invisible and a password is needed.
Q: Can my computer connect to the Internet?
A: The referee system forms an intranet and it does not connect to the Internet. If your system also needs access to the Internet, you will need a way (possibly two network interfaces) to connect to it.
Q: May I bring a "server" and offload computation to that server using the intranet?
A: Yes. Your server can connect to router provided by the organizer. Please be aware that this is an intranet for connecting to the referee computer and your own server.
Q: How can my team be qualified for Track 2 (offloading)?
A: You must have a front end and a back end. The front end communicates with the referee system. The back end does not communicate with the referee system. If your system does not have a back end, it is considered Track 1.
Q: How can my front end communicate with the back end?
A: You can use the network router (wifi or RJ45) provided by the organizer.
Q: Do I need to register the whole conference.
A: It is not necessary to register the entire conference.
Q: Do I have to register this challenge in advance?
A: Yes. On-site registration is not allowed.
Q: How many test images are there?
A: There will be around 5000 test images. It is not expected that entries doing on-device processing will complete most of the images.
Q: Does the accuracy criteria reward processing more images? Would I do better by processing just one easy image and stopping to save energy?
A: The average precision measure is computed by averaging precision over the amount of recall, so the more correct detections a system produces, the higher the accuracy score will be.
Q: Will there be any presentation on June 5, 2016?
A: There will be opening remarks by the organizers. There is no presentation during the competition.
Q: Do I have to stay for the entire duration of the competition?
A: You can come and go as your schedule permits.
Q: My team has several solutions with different strengths in energy efficiency and recognition accuracy. Can one team have multiple entries?
A: Yes. Each registration allows one solution. If your team has K registrations, K solutions can be tested.
Q: In Track 1 and Track 2, how many rounds will each time compete?
A: This depends on the number of teams. It is expected that there is enough time for each team to compete at least twice against two different teams.
Q: Do I have time fixing problems?
A: You can do anything before your turn. If it is your turn and you are not ready, you lose the round.
The following Q/A is for Track 3.
Q: What is the size of the display?
A: Dell U2414H is 24 inch and the resolution is 1920 x 1080.
Q: What are the sizes of the images?
A: The display will have black background with white square markers on the four corners. The size of the markers will be the same for all images.
Q: Do all images have the same size?
A: No. The images will be resized to fit in the rectangle between the markers. Resizing will keep the images' aspect ratios.
Q: How do I know the ID of an image?
A: A 20-bit binary code will be used to show an image's ID. Each bit can be black or white. Black = 0. White = 1.
Q: How is the binary code displayed on the screen?
A: The binary code will be at the left side (bits 10-19) and right side (bits 0-9) of the image between the markers. The most significant bit (MSB) is at the upper left corner (below the upper left marker). The least significant bit is at the lower right corner (above the lower right marker).
Q: Do I have to decide the binary code to determine the images' IDs?
A: Yes.
Q: What is the format of the answers to be sent to the referee system?
A: The same format as Tracks 1 and 2.
Q: Is offloading allowed in Track 3?
A: Yes. Track 3 does not distinguish offloading or not. If you use offloading, only the front end's energy is measured.
Future LPIRC
Q: Will LPIRC be held in 2017 and beyond?
A: The organizers plan to have LPIRC every year, as long as funds are available for the prizes and the organization costs.
Q: I want to sponsor LPIRC. Who should I talk to?
A: Thank you. Please send email to lpirc@ecn.purdue.edu
Q: What is the long-term goal of LPIRC?
A: To encourage innovative solutions that can perform accurate and fast image recognition with low energy consumption.
Q: Will the rules be changed for LPIRC 2017 and beyond?
A: Possibly.
Q: How will be energy be measured?
A: Please see the photographs above.
If your system is connected to 110V AC (standard wall socket in USA), the system will be plugged into a specially made socket.
We have universal connectors taking two wires from your power source (such as a battery) and providing two wires out to your system.
Q: When will the travel grants be given?
A: At the beginning of the challenge (around 850-910AM) on June 5, 2016.
Q: On June 5, do I have a chance testing whether my system can communicate with the referee?
A: Yes. Before the challenge starts, all teams can test whether their systems can communicate with the referee.
Q: May I test my system during the challenge, while another team is running?
A: No. During the challenge, only the competing teams' systems can communicate with the referee. This is to prevent interference.
Q: Do I have another chance testing the communication before the ten minutes start?
A: Yes. Your system can send a request to retrieve the document (/help in the URL).
Q: When does power measurement start?
A: When your system logs in. It is advised that your system starts requesting images as soon as the system logs in.
Q: Will all teams receive the same images?
A: Yes, but the orders may be different. The images are divided into small groups. The order of images inside each group may be different.
Q: If the image orders are different, would that be fair? Would some images be harder than the others?
A: The group sizes are small. If a team processes many groups, the differences should be negligible.
Q: Why are the image orders different for different teams?
A: Some teams have multiple solutions. If the image orders are the same, one of these teams might use the images from the first solution, perform off-line processing, and obtain advantage in the second time. This would be unfair. If the orders are different, it would be more difficult to perform off-line processing and this will improve fairness.
Q: What is the group size?
A: The group sizes will not be announced. In fact, the groups may have different sizes.
Q: How much time do I have for setup?
A: We expect that 10 minutes should be sufficient. Thus, each team (or each solution of a team that has multiple solutions) should take approximately 20 minutes total.
Q: How is the team order determined?
A: The team order will be determined by drawing lottery before the challenge starts. There will be two rounds of lottery. The first round determines the order of the second round. The second round determines the order of the challenge.
Q: Do I have to stay when the other teams are running?
A: You are free to leave anytime. If it is your turn and you are absent, you lose that round.
Q: When will the entire challenge finish?
A: It is hard to predict since the setup time may vary.
Q: Will we know which team is running?
A: Yes. A sign will be posted showing the currently running team.
Q: When will power measurement stop?
A: Ten minutes after your system logs in.
Q: Should my system send all answers to the referee at once, or one at a time?
A: You decide.
It is advised that your system sends answers to the referee often. The referee program will refuse to communicate with your system ten minutes after your system logs in. If your system sends all answers at once near the end of the ten minutes, it is possible that your system does not finish sending all results within the ten minutes.
Q: Is the ten-minute rule enforced by hand (using a stopwatch)?
A: This rule is enforced by the referee program.
Q: My team has several solutions. Can we try all of them and use the highest score?
A: For fairness, each registration allows only one solution. If your team has multiple solutions, please have one registration for each solution.
Q: Are there only 5,000 images? What happens if we finish 5,000 images within 10 minutes?
A: You can shut down the system to reduce the energy consumption. Alternatively, your system may log out and this will stop the power meter. We encourage you to do both (shutdown and log out).
Q: Do I have to disassemble the system so that you can measure power?
A: You need to provide a way to measure the power.
Q: I do not want to break my system.
A: We can measure the power consumed by a power adapter that charges your system. Please be aware that the power adapter consumes additional power.
Q: What is the maximum current my system can draw?
A: The power meter's upper limit is 20A. For DC (up to 32V), a fuse of 15A is used. For AC (up to 250V), a fuse of 2A is used.
Q: Does the power meter supply power?
A: No.
Q: Does my system have to communicate with the referee system through HTTP?
A: Yes.
Q: Can my system communicate with the referee program when another team is running?
A: No. The network router allows only the running teams.
Q: Do you provide a power supply?
A: No.
Q: How will the travel grants be determined?
A: It depends on the number of participants. If there is enough fund, every participant will receive a travel grant. If the fund is insufficient, earlier registrations will have higher priority.
Q: Can an observer (without paying the registration fee) receive a travel grant?
A: Due to the limited fund, only the people that register and pay the fee are eligible for the travel grants.
Q: If I register and pay the fee but cannot attend on June 5, can I get refund?
A: Please contact the conference.
Q: Can I bring special hardware?
A: Yes.
Q: Can I use commodity hardware (such as mobile phone, laptop, or desktop)?
A: Yes.
Q: Can I combine special hardware and commodity hardware?
A: Yes.
Q: Are companies allow to compete?
A: Yes, of course.
Q: Are participants outside USA allowed?
A: Yes, of course.
Q: The interface program is written in Python. Does my program have to use Python?
A: You can use any language, as long as it communicates with the referee program using HTTP.
Q: I want to spend a lot of money to build the best system and win. Is there any restriction on how much I can be spend?
A: You can spend as much money as you prefer.
Q: Must the system use DC power from batteries?
A: There is no such restriction. Your system may draw AC power from a wall socket.
Q: Is it possible that an image contain multiple objects?
A: Yes, that is possible
Q: Can I observe without competing?
A: Yes.
Q: Is there any financial support?
A: Yes, travel grants are available. Please visit the conference web site (www.dac.com) for application.
Q: How does my program communicate with the system?
A: A program template will be available for the interface. The interface will be HTTP based. There will be a wireless/wired router.
Q: How should I write my program so that it can communicate with your program?
A: Please structure your program so that it has two components: the recognition engine and the interface. The interface will be released before the competition.
Q: How can my computer connect to yours, get data, and send results?
A: An intranet will be set up. Both wired (RJ 45) and wireless (Wifi) will be available. For track 3, the images are provided through a display (thus, you need to bring a camera). For all tracks, the answers are sent to the referee through the network.
Q: How do I know which Wifi network to connect to?
A: It will be announced before the competition. To prevent interference, the Wifi for this competition will make the SSID invisible and a password is needed.
Q: Can my computer connect to the Internet?
A: The referee system forms an intranet and it does not connect to the Internet. If your system also needs access to the Internet, you will need a way (possibly two network interfaces) to connect to it.
Q: May I bring a "server" and offload computation to that server using the intranet?
A: Yes. Your server can connect to router provided by the organizer. Please be aware that this is an intranet for connecting to the referee computer and your own server.
Q: How can my team be qualified for Track 2 (offloading)?
A: You must have a front end and a back end. The front end communicates with the referee system. The back end does not communicate with the referee system. If your system does not have a back end, it is considered Track 1.
Q: How can my front end communicate with the back end?
A: You can use the network router (wifi or RJ45) provided by the organizer.
Q: Do I need to register the whole conference.
A: It is not necessary to register the entire conference.
Q: Do I have to register this challenge in advance?
A: Yes. On-site registration is not allowed.
Q: How many test images are there?
A: There will be around 5000 test images. It is not expected that entries doing on-device processing will complete most of the images.
Q: Does the accuracy criteria reward processing more images? Would I do better by processing just one easy image and stopping to save energy?
A: The average precision measure is computed by averaging precision over the amount of recall, so the more correct detections a system produces, the higher the accuracy score will be.
Q: Will there be any presentation on June 5, 2016?
A: There will be opening remarks by the organizers. There is no presentation during the competition.
Q: Do I have to stay for the entire duration of the competition?
A: You can come and go as your schedule permits.
Q: My team has several solutions with different strengths in energy efficiency and recognition accuracy. Can one team have multiple entries?
A: Yes. Each registration allows one solution. If your team has K registrations, K solutions can be tested.
Q: In Track 1 and Track 2, how many rounds will each time compete?
A: This depends on the number of teams. It is expected that there is enough time for each team to compete at least twice against two different teams.
Q: Do I have time fixing problems?
A: You can do anything before your turn. If it is your turn and you are not ready, you lose the round.
The following Q/A is for Track 3.
Q: What is the size of the display?
A: Dell U2414H is 24 inch and the resolution is 1920 x 1080.
Q: What are the sizes of the images?
A: The display will have black background with white square markers on the four corners. The size of the markers will be the same for all images.
Q: Do all images have the same size?
A: No. The images will be resized to fit in the rectangle between the markers. Resizing will keep the images' aspect ratios.
Q: How do I know the ID of an image?
A: A 20-bit binary code will be used to show an image's ID. Each bit can be black or white. Black = 0. White = 1.
Q: How is the binary code displayed on the screen?
A: The binary code will be at the left side (bits 10-19) and right side (bits 0-9) of the image between the markers. The most significant bit (MSB) is at the upper left corner (below the upper left marker). The least significant bit is at the lower right corner (above the lower right marker).
Q: Do I have to decide the binary code to determine the images' IDs?
A: Yes.
Q: What is the format of the answers to be sent to the referee system?
A: The same format as Tracks 1 and 2.
Q: Is offloading allowed in Track 3?
A: Yes. Track 3 does not distinguish offloading or not. If you use offloading, only the front end's energy is measured.
Future LPIRC
Q: Will LPIRC be held in 2017 and beyond?
A: The organizers plan to have LPIRC every year, as long as funds are available for the prizes and the organization costs.
Q: I want to sponsor LPIRC. Who should I talk to?
A: Thank you. Please send email to lpirc@ecn.purdue.edu
Q: What is the long-term goal of LPIRC?
A: To encourage innovative solutions that can perform accurate and fast image recognition with low energy consumption.
Q: Will the rules be changed for LPIRC 2017 and beyond?
A: Possibly.
Procedure
Before June 5, 2016
Before June 5, 2016
- Please download the development toolkit.
- A skeleton program for the communication between your program and the referee program will be release in April 2016. This program is HTTP based. The program has the same interface as the one used in 2015. The main change is to handle the tournament style.
- Participants need to register at the DAC workshop. This is necessary because of the administrative cost. If you need financial support, please apply for the travel grants. We will do our best to reimburse your travel expenses. The reimbursement will occur at the workshop.
- Each system needs to enter an assigned web site to retrieve the test images. Each test image may contain objects in the 200 categories. When a contestant system starts retrieving the test images, power measurement starts.
- When the system finishes recognition, it sends the results (bounding boxes, object labels, and detection scores) to the assigned web site. Power measurement stops after the system logs out, or after the time limit is finished.
- The score is the ratio mAP/E, where mAP is mean across object categories of the average precision for detection of that category, and E is the total energy consumption. The participant that has the highest score wins the competition.
- In addition to the winner by the highest score, prizes will be given to the participants whose systems achieve (1) the least amount of energy with mAP at least one standard deviation above of the average for all participants and (2) the highest mAP with the energy is at least one standard deviation below the average of all participants.
- If a tie occurs, there will be multiple winners.
- After a time limit (10 minutes) recording will stop and only results already received will be counted.
- The winners will be announced on the next day (June 6) in the conference's Pavilion session. The winners will be given chances to briefly explain their approaches.
- Each winner is required to submit a report within three months explaining the method. The reports are not reviewed.
- If a winner fails to submit a report within three months, the prize will be recalled and given to the participant of the next highest score.
- If a participant does not wish to publish the method, the participant may choose so during registration. In this case, the participant's score will be announced and ranked but cannot receive any prize.
Prizes:
- The champion will receive $1,000 (ranked by mAP/energy)
- The second prize is $500 (ranked by mAP/energy).
- The third prize is $200 (ranked by mAP/energy).
- The prize for the least energy with high accuracy (at least one standard above the average) is $200.
- The prize for the highest accuracy with low energy (at least one standard below the average) is $200.
- The prize for the highest points (in Track 1 or Track 2) is $200.
How are scores calculated?
The score has two parts: the accuracy in image recognition and the energy consumption.
Accuracy: To compute the accuracy, the bounding boxes returned by the system are sorted by putative category of object and by detection score. For each category of object, the bounding boxes are evaluated in order from highest detection score to lowest. If a bounding box has an intersection over union of greater than 0.5 with a ground truth bounding box that has not already been used to give a correct score to another bounding box, then it is marked correct with score 1, otherwise it is marked incorrect with score 0. After all boxes with detection score >= t have been evaluated the precision is the sum of the scores for each box divided by the number of boxes considered. Similarly the recall is the sum of the scores for the boxes returned divided by the total number of boxes of that category in the ground truth (for the whole test set). The threshold t is varied to compute a precision vs recall curve. Average Precision (AP) is the average of precision over all recalls [0,1]. For more details on how mAP is computed, see this technical report ( http://arxiv.org/abs/1409.0575 ) on the ImageNet Challenge that includes details for how mAP is computed including how multiple detections for the same object are handled as well as a modification to the criteria for small bounding boxes (Section 4.3).
Energy: The energy will be measured by a power meter. The meter can measure both DC power and AC power. These are several examples:
As these examples describe, there is great flexibility in measuring energy consumption. Participants can decide which option is the most advantageous for the specific situation. Please be aware that the energy loss due to voltage conversion by the charger (or the power supply) is added to the total energy consumption and may reduce the overall score.
The score has two parts: the accuracy in image recognition and the energy consumption.
Accuracy: To compute the accuracy, the bounding boxes returned by the system are sorted by putative category of object and by detection score. For each category of object, the bounding boxes are evaluated in order from highest detection score to lowest. If a bounding box has an intersection over union of greater than 0.5 with a ground truth bounding box that has not already been used to give a correct score to another bounding box, then it is marked correct with score 1, otherwise it is marked incorrect with score 0. After all boxes with detection score >= t have been evaluated the precision is the sum of the scores for each box divided by the number of boxes considered. Similarly the recall is the sum of the scores for the boxes returned divided by the total number of boxes of that category in the ground truth (for the whole test set). The threshold t is varied to compute a precision vs recall curve. Average Precision (AP) is the average of precision over all recalls [0,1]. For more details on how mAP is computed, see this technical report ( http://arxiv.org/abs/1409.0575 ) on the ImageNet Challenge that includes details for how mAP is computed including how multiple detections for the same object are handled as well as a modification to the criteria for small bounding boxes (Section 4.3).
Energy: The energy will be measured by a power meter. The meter can measure both DC power and AC power. These are several examples:
- A participant may use a tablet and disconnect a tablet's battery and connect the tablet with the battery with external wires. The power meter can measure the voltage and the current from the battery to calculate the power consumption.
- A participant may use a smartphone without removing the battery. In this case, the power consumed by the (AC to DC) charger is measured. This requires no change to the smartphone.
- A participant may use a desktop that draws AC power. The power from the socket is measured. This requires no change to the desktop.
As these examples describe, there is great flexibility in measuring energy consumption. Participants can decide which option is the most advantageous for the specific situation. Please be aware that the energy loss due to voltage conversion by the charger (or the power supply) is added to the total energy consumption and may reduce the overall score.