LPIRC 2019 Workshop Onsite Track
The Onsite Track is a competition for accurate and low power consumption for detecting objects in 20,000 images within 10 minutes. If your system finishes all 20,000 images within 10 minutes, the system can log out and power measurement stops. Solutions will be evaluated based on both accuracy (mean average precision, mAP) and energy (Watt-Hour). This track has no limitations on the platform you use. However, in order to measure the energy consumption, your device cannot contain batteries and must be powered externally (by a DC source or plugging into a wall socket). Please be aware that the organizers do not provide a power supply. If you need a power supply, you need to bring it.
This track encourages innovation in all aspects, including system-level improvements (e.g., better cache performance or voltage scaling), vision algorithms, hardware components (e.g., new intelligence chips), etc. This track has no restriction on the deep learning frameworks; you can use any software framework for example Tensorflow, Pytorch, Caffe2. The test data will be “ImageNet-like” (200 categories) but you can train your model on any dataset.
You are encouraged to inspect the sample code for communication between your system and the referee system (https://github.com/ieeelpirc/). Your vision system communicates with the referee system through a network router as shown below:
The output of your system has the following format. Each detected object has seven numbers:
1 159 0.884664 119.976089 104.969680 446.638915 196.629038
3 134 0.956790 84.127186 80.494650 495.113390 265.696301
4 26 0.196648 140.979062 164.311968 355.728417 266.404466
5 134 0.835819 195.385994 108.760951 322.980318 239.285166
...
The meaning of each row is:
Image-name, Class-id, confidence, xmin, ymin, xmax, ymax.
Your system may report the detected objects out of order (e.g., the objects in image 5 may be reported before the objects in image 2).
Evaluation: Submissions will be evaluated based on the matrix as follows:
Score = mean average precision / energy consumption
Mean average precision (mAP): The vision system can detect multiple objects in an image. We use mean average precision (mAP) as recognition accuracy. Specifically, the system detects objects, draws the bounding box, and reports to the referee system.
Energy consumption is measured by the Yokogawa WT 310 Digital Power Meter. The following photographs show how to connect to the power meter.
The network router (both wired and wireless). The wireless SSID will be hidden and the password will be given to you. You may use the wired network. Please be aware that this system connects to only the referee program and does not connect to the Internet.
AC connection. If your system takes 110V AC, simply plug into the AC socket.
This is an example of measuring AC. The power adapter is the power load and the power adapter is connected to the system directly. The voltage shown in the power meter is 121.18V.
This is a different configuration. The power adapter is plugged into a wall socket. The DC from the adapter is the input to the power meter (11.816V).
If your system uses DC (the output from the power adapter in this case), the DC output is plugged in to the input of the power meter.
The other end (the output of the power meter) becomes the power source of the system. This Nvidia board is for demonstration purpose only. It does not mean you need to use an Nvidia board.
The left is a universal connector.
FAQ (Frequently Asked Questions):
Q: Who can participate?
A: A contestant must be at least 13 years old and not affiliated with the organizers or sponsors.
Q: How will be energy be measured?
A: Please see the photographs above.
If your system is connected to 110V AC (standard wall socket in USA), the system will be plugged into a specially made socket.
We have universal connectors taking two wires in from your power source (such as a battery) and providing two wires out to your system.
Q: May I take an open-source solution by someone outside my team?
A: Participants should submit their own work and develop innovative solutions. Please do not submit solutions released by anyone outside your team. You must improve the open-source solution. The organizers reserve the right to disqualify a team if it is discovered that the team submits a solution that is not created by the team members.
Q: Do I have a chance to test whether my system can communicate with the referee?
A: Yes. Early June 2019, the referee system will be on the Internet for testing. Please register to receive notification.
Q: May I test my system during the challenge, while another team is running?
A: No. During the challenge, only one team's system can communicate with the referee. This is to prevent interference.
Q: Is the challenge sequential, one team at a time?
A: Yes.
Q: When does power measurement start?
A: When your system logs in. It is advised that your system starts requesting images as soon as the system logs in.
Q: Will all teams receive the same images?
A: Yes, but the order may be slightly different. The images are divided into small groups. The order of images inside each group may be different.
Q: If the image orders are different, would that be fair? Would some images be harder than the others?
A: The group sizes are small. If a team processes many groups, the differences should be negligible.
Q: Why are the image orders different for different teams?
A: Some teams have multiple solutions. If the image orders are the same, one of these teams might use the images from the first solution, perform off-line processing, and obtain an advantage the second time. This would be unfair. If the orders are different, it would be more difficult to perform off-line processing and this will improve fairness.
Q: What is the group size?
A: The group sizes will not be announced. In fact, the groups may have different sizes.
Q: How much time do I have for setup?
A: We expect that 10 minutes should be sufficient. Thus, each team (or each solution of a team that has multiple solutions) should take approximately 20 minutes total.
Q: How is the team order determined?
A: The team order will be determined by a lottery before the challenge starts. There will be two rounds of lottery. The first round determines the order of the second round. The second round determines the order of the challenge.
Q: Do I have to stay when the other teams are running?
A: You are free to leave anytime. If it is your turn and you are absent, the next team will start. When you come back, you will be the one after the current team.
Q: When will the entire challenge finish?
A: It is hard to predict since the setup time may vary.
Q: If I am late, what happens?
A: If you do not show up when all the other teams have finished, you are disqualified.
Q: Will we know which team is running?
A: Yes. A sign will be posted showing the currently running team.
Q: When will power measurement stop?
A: Ten minutes after your system logs in. If your system finishes all 20,000 images early, please log out to stop the power meter.
Q: Should my system send all answers to the referee at once, or one at a time?
A: You decide. Please be aware of the network overhead. It is more efficient sending large amounts of data at once.
It is advised that your system sends answers to the referee often. The referee program will refuse to communicate with your system 10 minutes after your system logs in. If your system sends all answers at once near the end of the ten minutes, it is possible that your system may not finish sending all results within the ten minutes.
Q: Is the ten-minute rule enforced by hand (using a stopwatch)?
A: This rule is enforced by a timer in the referee program. There is no stopwatch by hand.
Q: My team has several solutions. Can we try all of them and use the highest score?
A: For fairness, each workshop registration allows only one solution. If your team has multiple solutions, please have one registration for each solution.
Q: Do I have to disassemble the system so that you can measure power?
A: You need to provide a way to measure the power. If your system is fully enclosed (like a mobile phone), it is not possible to measure the power.
Q: I do not want to break my system.
A: We can measure the power consumed by a power adapter that charges your system. Please be aware that the power adapter consumes additional power. You must remove the battery in advance.
Q: What is the maximum current my system can draw?
A: The power meter's upper limit is 20A. For DC (up to 32V), a fuse of 15A is used. For AC (up to 250V), a fuse of 2A is used.
Q: Do you provide a power supply?
A: No.
Q: Can my system communicate with the referee program when another team is running?
A: No. The network router allows only the running team.
Q: Can I bring special hardware?
A: Yes.
Q: Can I use commodity hardware (such as mobile phone, laptop, or desktop)?
A: Yes.
Q: Can I combine special hardware and commodity hardware?
A: Yes.
Q: Are companies allowed to compete?
A: Yes.
Q: Are participants outside USA allowed?
A: Yes. Citizens of US-embargoed countries are excluded.
Q: The interface program is written in Python. Does my program have to use Python?
A: You can use any language, as long as it communicates with the referee program.
Q: I want to spend a lot of money to build the best system and win. Is there any restriction on how much I can spend?
A: You can spend as much money as you prefer.
Q: Must the system use DC power from batteries?
A: Your system may draw AC power from a wall socket. Please be aware that this competition is held in USA. If you use AC, your system must be able to run using a wall socket.
Q: Is it possible that an image contain multiple objects?
A: Yes, that is possible
Q: Can I attend without competing?
A: Yes. Spectators are welcome.
Q: How should I write my program so that it can communicate with your program?
A: Please check (https://github.com/ieeelpirc/).
Q: How can my computer connect to yours, get data, and send results?
A: An intranet will be set up. Both wired (RJ 45) and wireless (Wifi) will be available.
Q: How do I know which Wifi network to connect to?
A: It will be announced before the competition.
Q: Can my computer connect to the Internet?
A: No.
Q: May I bring a "server" and offload computation to that server?
A: No. All computation must occur in your system and its power is measured.
Q: Will there be an opportunity to publish my method?
A: Yes. The organizers actively seek opportunities for winners to present their solutions. Past winners presented their solutions in 2015 ICCAD, 2017 ASPDAC, 2017 DATE, 2018 NIPS, 2019 AICAS, 2019 CVPR.
Q: Do I need to register for the whole conference?
A: You need to register for the workshop. Please be aware that you need to register in two places: CVPR workshop and this competition. CVPR registration allows you to enter the building. It is not necessary to register for the entire conference. The registration for the competition allows the organizers to communicate with you. The organizers do not receive the list of workshop registrations from CVPR.
Q: Does the accuracy criterion reward processing more images? Would I do better by processing just one easy image and stopping to save energy?
A: The more correct detections your system can produce, the higher score you will get. No, you should not process only one image and stop. If your system processes very few images, it is unlikely that you can win.