Low-Power Image Recognition Challenge (LPIRC) June 7 2015 in Austin Texas (with Design Automation Conference)
News:
- LPIRC 2016 will add a third track. The images will be displayed on a computer screen and the teams need to use cameras to obtain the the images. The other two tracks use a computer network to obtain the images (same as 2015).
- Sponsors: IEEE, Nvidia, Xilinx.
- The US Government announced the Nanotechnology-Inspired Grand Challenge for Future Computing.
- LPIRC 2016 organizing committee: Co-Chairs: Yung-Hsiang Lu (Purdue) and Alex Berg (UNC); Committee: Daniel Hammerstrom (DARPA), Greg P Leeming (Intel), David Atienza Alonso (EPFL), Massimo Alioto (NUS), David Pan (University of Taxes)
- A special session in ICCAD (International Conference on Computer Aided Design) for LPIRC on 2015/11/02 in Austin Texas.
Participants before the first LPIRC started. 34 participants from 10 teams in 4 countries.
Congratulations to the winners of the first LPIRC. CHAMPION: Tsinghua/Huawei Team; SECOND PRIZE: CASIA/Huawei Team; THIRD PRIZE: Tsinghua/Huawei Team; HIGHEST ACCURACY WITH LOW ENERGY: Tsinghua/Huawei Team; LEAST ENERGY WITH HIGH ACCURACY: CASIA/Huawei Team. Two special prizes were also awarded. READY TO GO: Carnegie Mellon Team; STANDING ALONE: Rice University Team. All Winning Teams received certificates and awards from Nvidia and IEEE Rebooting Computing
Please sign up to receive updates and to download the training data.
Questions? Please send email to lpirc@ecn.purdue.edu
The first LPIRC was held 9AM on June 7 in Room 310 in the San Francisco Convention Center, California, USA.
Please visit IEEE Rebooting Computing for the photographs.
News:
Motivation
Many mobile systems (smartphones, electronic glass, autonomous robots) can capture images. These systems use batteries and energy conservation is essential. This challenge aims to discover the best technology in both image recognition and energy conservation. Winners will be evaluated based on both high recognition accuracy and low power usage.
Image Recognition
Image recognition involves many tasks. This challenge focuses on object detection, a basic routine in many recognition approaches. The following two examples illustrate the task. In the first example, there are two objects: a bird and a frog. In the second example, there are several objects: cars, persons, motorcycle, and a helmet. The training and validation data for LPIRC comes from the ImageNet Large Scale Visual Recognition Challenge detection competition. The test data will be specific to LPIRC.
Questions? Please send email to lpirc@ecn.purdue.edu
The first LPIRC was held 9AM on June 7 in Room 310 in the San Francisco Convention Center, California, USA.
Please visit IEEE Rebooting Computing for the photographs.
News:
- Free for observers. If you want to see, without competing and winning, you do not have to pay. Please register so that the conference can count the number of people. The registration remains open.
- If you want to compete, please visit this page, with the code DAC15W1. The registration fee is $180. Registrations for the competition will be closed at noon June 5 US Eastern Time.
Motivation
Many mobile systems (smartphones, electronic glass, autonomous robots) can capture images. These systems use batteries and energy conservation is essential. This challenge aims to discover the best technology in both image recognition and energy conservation. Winners will be evaluated based on both high recognition accuracy and low power usage.
Image Recognition
Image recognition involves many tasks. This challenge focuses on object detection, a basic routine in many recognition approaches. The following two examples illustrate the task. In the first example, there are two objects: a bird and a frog. In the second example, there are several objects: cars, persons, motorcycle, and a helmet. The training and validation data for LPIRC comes from the ImageNet Large Scale Visual Recognition Challenge detection competition. The test data will be specific to LPIRC.
Competition Tracks
- For Track 1, computation offloading to another device or server is not allowed.
- For Track 2, computation offloading is allowed.
Important Dates
- 2015/06/08 Announcement of the winners at 4:30-5:30PM in DAC Pavilion Booth 311.
- 2015/06/07 Competition in San Francisco as a one-day workshop in Design Automation Conference
- 2015/04/10 Registration is open.
- 2015/03/31 Reference code for interface with the referee program. To obtain the source code (in Python), please go to https://github.com/luyunghsiang/LPIRC.git. The README explains how to use the program
- 2015/01 Announcement of the power meter. The power meter is Yokogawa WT 300. The network router is Nighthawk AC1900 Dual Band Wi-Fi.
- 2014/11 Training and validation data available
Organizers:
Students:
Sponsors:
IEEE Council on Superconductivity
IEEE Council on Electronic Design Automation,
Nvidia
- Yung-Hsiang Lu (Purdue University)
- Alex Berg (University of North Carolina at Chapel Hill)
- Massimo Alioto (NUS)
- David Atienza Alonso (EPFL)
- Patrick Groeneveld (Synposys)
- David Kirk (Nvidia)
- Gi-Joon Nam (IBM)
- Vivek Tiwari (Intel)
Students:
- Wei Liu (UNC)
- Ganesh Gingade (Purdue)
- Rachit Garg (Purdue)
Sponsors:
IEEE Council on Superconductivity
IEEE Council on Electronic Design Automation,
Nvidia
The network router. The wireless SSID will be hidden and the password will be given to you before June 7. You may use the wired network. Please be aware that this system connects to only the referee program and do not connect to the Internet.
AC connection. If your system takes 110V AC, simply plug into the AC socket.
This is an example of measuring AC. The power adapter is the power load and the power adapter is connected to the system directly. The voltage shown in the power meter is 121.18V.
This is a different configuration. The power adapter is plugged into a wall socket. The DC from the adapter is the input to the power meter (11.816V).
If your system uses DC (the output from the power adapter in this case), the DC output is plugged in to the input of the power meter.
The other end (the output of the power meter) becomes the power source of the system. This Nvidia board is for demonstration purpose only. It does not mean you need to use an Nvidia board.
The left is a universal connector.
FAQ (Frequently Asked Questions):
Q: How will be energy be measured?
A: Please see the photographs above.
If your system is connected to 110V AC (standard wall socket in USA), the system will be plugged into a specially made socket.
We have universal connectors taking two wires in from your power source (such as a battery) and providing two wires out to your system.
Q: When will the travel grants be given?
A: At the beginning of the challenge (around 850-910AM) on June 7.
Q: On June 7, do I have a chance testing whether my system can communicate with the referee?
A: Yes. Before the challenge starts, all teams can test whether their systems can communicate with the referee. Approximately 9:05AM on June 7.
Q: May I test my system during the challenge, while another team is running?
A: No. During the challenge, only one team's system can communicate with the referee. This is to prevent interference.
Q: Is the challenge sequential, one team at a time?
A: Yes.
Q: Do I have another chance testing the communication before the ten minutes start?
A: Yes. Your system can send a request to retrieve the document (/help in the URL).
Q: When does power measurement start?
A: When your system logs in. It is advised that your system starts requesting images as soon as the system logs in.
Q: Will all teams receive the same images?
A: Yes, but the orders may be slightly different. The images are divided into small groups. The order of images inside each group may be different.
Q: If the image orders are different, would that be fair? Would some images be harder than the others?
A: The group sizes are small. If a team process many groups, the differences should be negligible.
Q: Why are the image orders different for different teams?
A: Some teams have multiple solutions. If the image orders are the same, one of these teams might use the images from the first solution, perform off-line processing, and obtain advantage in the second time. This would be unfair. If the orders are different, it would be more difficult to perform off-line processing and this will improve fairness.
Q: What is the group size?
A: The group sizes will not be announced. In fact, the groups may have different sizes.
Q: How much time do I have for setup?
A: We expect that 10 minutes should be sufficient. Thus, each team (or each solution of a team that has multiple solutions) should take approximately 20 minutes total.
Q: How is the team order determined?
A: The team order will be determined by drawing lottery before the challenge starts. There will be two rounds of lottery. The first round determines the order of the second round. The second round determines the order of the challenge.
Q: Do I have to stay when the other teams are running?
A: You are free to leave anytime. If it is your turn and you are absent, the next team will start. When you come back, you will be the one after the current team.
Q: When will the entire challenge finish?
A: It is hard to predict since the setup time may vary.
Q: If I am late, what happens?
A: If you do not show up when all the other teams have finished, you are disqualified.
Q: Will we know which team is running?
A: Yes. A sign will be posted showing the currently running team.
Q: When will power measurement stop?
A: Ten minutes after your system logs in.
Q: Should my system send all answers to the referee at once, or one at a time?
A: You decide.
It is advised that your system sends answers to the referee often. The referee program will refuse to communicate with your system ten minutes after your system logs in. If your system sends all answers at once near the end of the ten minutes, it is possible that your system does not finish sending all results within the ten minutes.
Q: Is the ten-minute rule enforced by hand (using a stopwatch)?
A: This rule is enforced by the referee program.
Q: My team has several solutions. Can we try all of them and use the highest score?
A: For fairness, each registration allows only one solution. If your team has multiple solutions, please have one registration for each solution.
Q: Are there only 5,000 images? What happens if we finish 5,000 images within 10 minutes?
A: You can shut down the system to reduce the energy consumption. Alternatively, your system may log out and this will stop the power meter. We encourage you to do both (shutdown and log out).
Q: Do I have to disassemble the system?
A: You need to provide a way to measure the power.
Q: I do not want to break my system.
A: We can measure the power consumed by a power adapter that charges your system. Please be aware that the power adapter consumes additional power.
Q: What is the maximum current my system can draw?
A: The power meter's upper limit is 20A. For DC (up to 32V), a fuse of 15A is used. For AC (up to 250V), a fuse of 2A is used.
Q: Does the power meter supply power?
A: No.
Q: When will we get the username and password?
A: At least 12 hours before the challenge starts.
Q: When will the network router's username and password be given?
A: At least 12 hours before the challenge starts.
Q: Can my system communicate with the referee program when another team is running?
A: No. The network router allows only the running team.
Q: Do you provide a power supply?
A: No.
Q: How will the travel grants be determined?
A: It depends on the number of participants. If there is enough fund, every participant will receive a travel grant. If the fund is insufficient, earlier registrations will have higher priority.
Q: Can an observer (without paying the $180 registration fee) receive a travel grant?
A: Due to the limited fund, only the people that register and pay $180 are eligible for the travel grants.
Q: If I register and pay $180 but cannot attend on June 7, can I get refund?
A: Please contact the conference.
Q: Can I bring special hardware?
A: Yes.
Q: Can I use commodity hardware (such as mobile phone, laptop, or desktop)?
A: Yes.
Q: Can I combine special hardware and commodity hardware?
A: Yes.
Q: Are companies allow to compete?
A: Yes, of course.
Q: Are participants outside USA allowed?
A: Yes, of course.
Q: The interface program is written in Python. Does my program have to use Python?
A: You can use any language, as long as it communicates with the referee program using HTTP.
Q: I want to spend a lot of money to build the best system and win. Is there any restriction on how much I can be spend?
A: Yes, you can spend as much money as you prefer.
Q: Must the system use DC power from batteries?
A: There is no such restriction. Your system may draw AC power from a wall socket.
Q: Is it possible that an image contain multiple objects?
A: Yes, that is possible
Q: Can I see without competing?
A: Yes.
Q: Is there any financial support?
A: Yes, travel grants are available. Please visit the conference web site (www.dac.com) for application.
Q: How does my program communicate with the system?
A: A program template will be available for the interface. The interface will be HTTP based. There will be a wireless/wired router.
Q: How should I write my program so that it can communicate with your program?
A: Please structure your program so that it has two components: the recognition engine and the interface. The interface will be released before the competition.
Q: How can my computer connect to yours, get data, and send results?
A: An intranet will be set up. Both wired (RJ 45) and wireless (Wifi) will be available.
Q: How do I know which Wifi network to connect to?
A: It will be announced before the competition. To prevent interference, the Wifi for this competition will make the SSID invisible and a password is needed.
Q: Can my computer connect to the Internet?
A: Yes. The San Francisco Convention Center has wireless network. Please be aware that your system needs to connect to an intranet to get the images and to send the results. If your system also needs access to the Internet, you will need a way (possibly two network interfaces) to connect to it.
Q: May I bring a "server" and offload computation to that server using the intranet?
A: Yes. Your server can connect to router provided by the organizer. Please be aware that this is an intranet for connecting to the referee computer and your own server. This router is not connected to the Internet.
Q: Will there be an opportunity to publish my method?
A: Yes. You are invited to submit your paper to a special issue in IEEE Transactions on Emerging Topics in Computing. The deadline is September 1, 2015.
Q: Do I have to compete in June if I want to submit a paper to IEEE TETC?
A: No. These are two different events. You may compete in June without submitting a paper to TETC. You may submit a paper to TETC without competing in June.
Q: Is this the first competition of its kind?
A: We believe so, even though there have been related competitions. For example, Large Scale Visual Recognition Challenge at Image-Net concerns about only accuracy, not energy. The design contest at ISLPED does not focus on image recognition.
Q: My team has several people. Does everyone need to register?
A: Yes. Please.
Q: Do I need to register the whole conference.
A: It is not necessary to register the entire conference.
Q: When will registration start?
A: March 31, 2015.
Q: Do I have to register in advance?
A: "Walk-in" would not be ideal since we need to distribute user names and passwords. Please register early.
Q: How many test images are there?
A: There will be around 5000 test images. It is not expected that entries doing on-device processing will complete most of the images.
Q: Does the accuracy criteria reward processing more images? Would I do better by processing just one easy image and stopping to save energy?
A: The average precision measure is computed by averaging precision over the amount of recall, so the more correct detections a system produces, the higher the accuracy score will be.
Q: Will there be any presentation on June 7, 2015?
A: There will be opening remarks by the organizers. There is no presentation during the competition.
Q: Do I have to stay for the entire duration of the competition?
A: You can come and go as your schedule permits.
Q: Will you tell me the time of my turn?
A: Yes. We will inform the teams of the assigned time based on the registrations.
Q: My team has several solutions with different strengths in energy efficiency and recognition accuracy. Can one team have multiple entries?
A: Yes. Each registration allows one solution. If your team has two registrations, both solutions can be tested.
Q: How will be energy be measured?
A: Please see the photographs above.
If your system is connected to 110V AC (standard wall socket in USA), the system will be plugged into a specially made socket.
We have universal connectors taking two wires in from your power source (such as a battery) and providing two wires out to your system.
Q: When will the travel grants be given?
A: At the beginning of the challenge (around 850-910AM) on June 7.
Q: On June 7, do I have a chance testing whether my system can communicate with the referee?
A: Yes. Before the challenge starts, all teams can test whether their systems can communicate with the referee. Approximately 9:05AM on June 7.
Q: May I test my system during the challenge, while another team is running?
A: No. During the challenge, only one team's system can communicate with the referee. This is to prevent interference.
Q: Is the challenge sequential, one team at a time?
A: Yes.
Q: Do I have another chance testing the communication before the ten minutes start?
A: Yes. Your system can send a request to retrieve the document (/help in the URL).
Q: When does power measurement start?
A: When your system logs in. It is advised that your system starts requesting images as soon as the system logs in.
Q: Will all teams receive the same images?
A: Yes, but the orders may be slightly different. The images are divided into small groups. The order of images inside each group may be different.
Q: If the image orders are different, would that be fair? Would some images be harder than the others?
A: The group sizes are small. If a team process many groups, the differences should be negligible.
Q: Why are the image orders different for different teams?
A: Some teams have multiple solutions. If the image orders are the same, one of these teams might use the images from the first solution, perform off-line processing, and obtain advantage in the second time. This would be unfair. If the orders are different, it would be more difficult to perform off-line processing and this will improve fairness.
Q: What is the group size?
A: The group sizes will not be announced. In fact, the groups may have different sizes.
Q: How much time do I have for setup?
A: We expect that 10 minutes should be sufficient. Thus, each team (or each solution of a team that has multiple solutions) should take approximately 20 minutes total.
Q: How is the team order determined?
A: The team order will be determined by drawing lottery before the challenge starts. There will be two rounds of lottery. The first round determines the order of the second round. The second round determines the order of the challenge.
Q: Do I have to stay when the other teams are running?
A: You are free to leave anytime. If it is your turn and you are absent, the next team will start. When you come back, you will be the one after the current team.
Q: When will the entire challenge finish?
A: It is hard to predict since the setup time may vary.
Q: If I am late, what happens?
A: If you do not show up when all the other teams have finished, you are disqualified.
Q: Will we know which team is running?
A: Yes. A sign will be posted showing the currently running team.
Q: When will power measurement stop?
A: Ten minutes after your system logs in.
Q: Should my system send all answers to the referee at once, or one at a time?
A: You decide.
It is advised that your system sends answers to the referee often. The referee program will refuse to communicate with your system ten minutes after your system logs in. If your system sends all answers at once near the end of the ten minutes, it is possible that your system does not finish sending all results within the ten minutes.
Q: Is the ten-minute rule enforced by hand (using a stopwatch)?
A: This rule is enforced by the referee program.
Q: My team has several solutions. Can we try all of them and use the highest score?
A: For fairness, each registration allows only one solution. If your team has multiple solutions, please have one registration for each solution.
Q: Are there only 5,000 images? What happens if we finish 5,000 images within 10 minutes?
A: You can shut down the system to reduce the energy consumption. Alternatively, your system may log out and this will stop the power meter. We encourage you to do both (shutdown and log out).
Q: Do I have to disassemble the system?
A: You need to provide a way to measure the power.
Q: I do not want to break my system.
A: We can measure the power consumed by a power adapter that charges your system. Please be aware that the power adapter consumes additional power.
Q: What is the maximum current my system can draw?
A: The power meter's upper limit is 20A. For DC (up to 32V), a fuse of 15A is used. For AC (up to 250V), a fuse of 2A is used.
Q: Does the power meter supply power?
A: No.
Q: When will we get the username and password?
A: At least 12 hours before the challenge starts.
Q: When will the network router's username and password be given?
A: At least 12 hours before the challenge starts.
Q: Can my system communicate with the referee program when another team is running?
A: No. The network router allows only the running team.
Q: Do you provide a power supply?
A: No.
Q: How will the travel grants be determined?
A: It depends on the number of participants. If there is enough fund, every participant will receive a travel grant. If the fund is insufficient, earlier registrations will have higher priority.
Q: Can an observer (without paying the $180 registration fee) receive a travel grant?
A: Due to the limited fund, only the people that register and pay $180 are eligible for the travel grants.
Q: If I register and pay $180 but cannot attend on June 7, can I get refund?
A: Please contact the conference.
Q: Can I bring special hardware?
A: Yes.
Q: Can I use commodity hardware (such as mobile phone, laptop, or desktop)?
A: Yes.
Q: Can I combine special hardware and commodity hardware?
A: Yes.
Q: Are companies allow to compete?
A: Yes, of course.
Q: Are participants outside USA allowed?
A: Yes, of course.
Q: The interface program is written in Python. Does my program have to use Python?
A: You can use any language, as long as it communicates with the referee program using HTTP.
Q: I want to spend a lot of money to build the best system and win. Is there any restriction on how much I can be spend?
A: Yes, you can spend as much money as you prefer.
Q: Must the system use DC power from batteries?
A: There is no such restriction. Your system may draw AC power from a wall socket.
Q: Is it possible that an image contain multiple objects?
A: Yes, that is possible
Q: Can I see without competing?
A: Yes.
Q: Is there any financial support?
A: Yes, travel grants are available. Please visit the conference web site (www.dac.com) for application.
Q: How does my program communicate with the system?
A: A program template will be available for the interface. The interface will be HTTP based. There will be a wireless/wired router.
Q: How should I write my program so that it can communicate with your program?
A: Please structure your program so that it has two components: the recognition engine and the interface. The interface will be released before the competition.
Q: How can my computer connect to yours, get data, and send results?
A: An intranet will be set up. Both wired (RJ 45) and wireless (Wifi) will be available.
Q: How do I know which Wifi network to connect to?
A: It will be announced before the competition. To prevent interference, the Wifi for this competition will make the SSID invisible and a password is needed.
Q: Can my computer connect to the Internet?
A: Yes. The San Francisco Convention Center has wireless network. Please be aware that your system needs to connect to an intranet to get the images and to send the results. If your system also needs access to the Internet, you will need a way (possibly two network interfaces) to connect to it.
Q: May I bring a "server" and offload computation to that server using the intranet?
A: Yes. Your server can connect to router provided by the organizer. Please be aware that this is an intranet for connecting to the referee computer and your own server. This router is not connected to the Internet.
Q: Will there be an opportunity to publish my method?
A: Yes. You are invited to submit your paper to a special issue in IEEE Transactions on Emerging Topics in Computing. The deadline is September 1, 2015.
Q: Do I have to compete in June if I want to submit a paper to IEEE TETC?
A: No. These are two different events. You may compete in June without submitting a paper to TETC. You may submit a paper to TETC without competing in June.
Q: Is this the first competition of its kind?
A: We believe so, even though there have been related competitions. For example, Large Scale Visual Recognition Challenge at Image-Net concerns about only accuracy, not energy. The design contest at ISLPED does not focus on image recognition.
Q: My team has several people. Does everyone need to register?
A: Yes. Please.
Q: Do I need to register the whole conference.
A: It is not necessary to register the entire conference.
Q: When will registration start?
A: March 31, 2015.
Q: Do I have to register in advance?
A: "Walk-in" would not be ideal since we need to distribute user names and passwords. Please register early.
Q: How many test images are there?
A: There will be around 5000 test images. It is not expected that entries doing on-device processing will complete most of the images.
Q: Does the accuracy criteria reward processing more images? Would I do better by processing just one easy image and stopping to save energy?
A: The average precision measure is computed by averaging precision over the amount of recall, so the more correct detections a system produces, the higher the accuracy score will be.
Q: Will there be any presentation on June 7, 2015?
A: There will be opening remarks by the organizers. There is no presentation during the competition.
Q: Do I have to stay for the entire duration of the competition?
A: You can come and go as your schedule permits.
Q: Will you tell me the time of my turn?
A: Yes. We will inform the teams of the assigned time based on the registrations.
Q: My team has several solutions with different strengths in energy efficiency and recognition accuracy. Can one team have multiple entries?
A: Yes. Each registration allows one solution. If your team has two registrations, both solutions can be tested.
Procedure
Before June 7, 2015
Before June 7, 2015
- Please download the development toolkit.
- A skeleton program for the communication between your program and the referee program will be release in spring 2015. This program is HTTP based.
- Participants need to register at the DAC workshop. This is necessary because of the administrative cost. It is unnecessary to register the conference itself. If you need financial support, please apply for the travel grants. We will do our best to reimburse your travel expenses. The reimbursement will occur after the workshop.
- Each system needs to enter an assigned web site to retrieve the test images. Each test image may contain objects in the 200 categories. When a contestant system starts retrieving the test images, power measurement starts.
- When the system finishes recognition, it sends the results (bounding boxes, object labels, and detection scores) to the assigned web site. Power measurement stops after all the results are received, or after the time limit is finished.
- The score is the ratio mAP/E, where mAP is mean across object categories of the average precision for detection of that category, and E is the total energy consumption. The participant that has the highest score wins the competition.
- In addition to the winner by the highest score, prizes will be given to the participants whose systems achieve (1) the least amount of energy with mAP at least one standard deviation above of the average for all participants and (2) the highest mAP with the energy is at least one standard deviation below the average of all participants.
- If a tie occurs, there will be multiple winners.
- If a tie occurs, additional test images will be given until the tie is broken. This is no longer applicable since the winners will be announced the next day.
- After a time limit (currently set to 10 minutes) recording will stop and only results already transmitted will be counted.
- The winners will be announced on the next day (June 8) in the conference's Pavilion session. The winners will be given chances to briefly explain their approaches.
- All participants, as well as those that do not participate in this competition, are invited to submit papers to a special issue in IEEE Transactions on Emerging Topics in Computing. The submission deadline is September 1, 2015 and the papers will be published in September 2016.The submissions will be reviewed and it is not guaranteed that competition winners’ papers are accepted.
- Each winner is required to submit a report within three months explaining the method. The reports are not reviewed. If a winner submits a paper to the special issue mentioned above, the winner does not need to submit a separate report and the report will not be published immediately. If a winner does not submit a paper to the special issue, the report will be published immediately.
- If a winner fails to submit a report within three months, the prize will be recalled and given to the participant of the next highest score.
- If a participant does not wish to publish the method, the participant may choose so during registration. In this case, the participant's score will be announced and ranked but cannot receive any prize.
Prizes:
- The champion will receive $1,000.
- The second prize is $500.
- The third prize is $200.
- The prize for the least energy with high accuracy (at least one standard above the average) is $200.
- The prize for the highest accuracy with low energy (at least one standard below the average) is $200.
- Each winner will also receive one GPU card donated by Nvidia.
How are scores calculated?
The score has two parts: the accuracy in image recognition and the energy consumption.
Accuracy: To compute the accuracy, the bounding boxes returned by the system are sorted by putative category of object and by detection score. For each category of object, the bounding boxes are evaluated in order from highest detection score to lowest. If a bounding box has an intersection over union of greater than 0.5 with a ground truth bounding box that has not already been used to give a correct score to another bounding box, then it is marked correct with score 1, otherwise it is marked incorrect with score 0. After all boxes with detection score >= t have been evaluated the precision is the sum of the scores for each box divided by the number of boxes considered. Similarly the recall is the sum of the scores for the boxes returned divided by the total number of boxes of that category in the ground truth (for the whole test set). The threshold t is varied to compute a precision vs recall curve. Average Precision (AP) is the average of precision over all recalls [0,1]. For more details on how mAP is computed, see this technical report ( http://arxiv.org/abs/1409.0575 ) on the ImageNet Challenge that includes details for how mAP is computed including how multiple detections for the same object are handled as well as a modification to the criteria for small bounding boxes (Section 4.3).
Energy: The energy will be measured by a power meter. The meter can measure both DC power and AC power. There are several options:
As these examples describe, there is great flexibility in measuring energy consumption. Participants can decide which option is the most advantageous for the specific situation. Please be aware that the energy loss due to voltage conversion by the charger (or the power supply) is added to the total energy consumption and may reduce the overall score.
The score has two parts: the accuracy in image recognition and the energy consumption.
Accuracy: To compute the accuracy, the bounding boxes returned by the system are sorted by putative category of object and by detection score. For each category of object, the bounding boxes are evaluated in order from highest detection score to lowest. If a bounding box has an intersection over union of greater than 0.5 with a ground truth bounding box that has not already been used to give a correct score to another bounding box, then it is marked correct with score 1, otherwise it is marked incorrect with score 0. After all boxes with detection score >= t have been evaluated the precision is the sum of the scores for each box divided by the number of boxes considered. Similarly the recall is the sum of the scores for the boxes returned divided by the total number of boxes of that category in the ground truth (for the whole test set). The threshold t is varied to compute a precision vs recall curve. Average Precision (AP) is the average of precision over all recalls [0,1]. For more details on how mAP is computed, see this technical report ( http://arxiv.org/abs/1409.0575 ) on the ImageNet Challenge that includes details for how mAP is computed including how multiple detections for the same object are handled as well as a modification to the criteria for small bounding boxes (Section 4.3).
Energy: The energy will be measured by a power meter. The meter can measure both DC power and AC power. There are several options:
- A participant may use a tablet and disconnect a tablet's battery and connect the tablet with the battery with external wires. The power meter can measure the voltage and the current from the battery to calculate the power consumption.
- A participant may use a smartphone without removing the battery. In this case, the power consumed by the (AC to DC) charger is measured. This requires no change to the smartphone.
- A participant may use a desktop that draws AC power. The power from the socket is measured. This requires no change to the desktop.
As these examples describe, there is great flexibility in measuring energy consumption. Participants can decide which option is the most advantageous for the specific situation. Please be aware that the energy loss due to voltage conversion by the charger (or the power supply) is added to the total energy consumption and may reduce the overall score.