EuRoC Forum

[jeffdelmerico]

Dear EUROC organizers,

In the "Final Evaluation Hardware" thread, the following was posted, but I think it's more appropriate as a separate question:

According to image header stamps the whole flight took 141.90 sec. If I want to score max 4 points in section "Computation time" sa noted in Scoring system annex, I need to achieve processing time < 0.1tmax which means less than 14.19sec for whole octomap. But... In my case just requesting data from simserv,(no processing, no publising) with only simple console data counting output takes more than 160 sec.

I'm finding the same problem with Task 2...even with no other processing, just the service calls between the two VMs, the minimum total computation time I can achieve is about 60 seconds. I don't see how the maximum score for time (< 0.1 tmax == 14.19 sec) is possible without returning the octomap before the dataset is finished playing. However, several teams seem to have achieved full credit on the scoreboard, so it must be possible. Perhaps this is a local configuration problem? Any assistance would be appreciated.

Thanks,
Jeff Delmerico

[jeffdelmerico]

As a follow-up, I tested this on another host machine, again only running the minimal service calls, and the total time was around 24 seconds. While this is an improvement, and indicates some sort of local issue, the maximum time score still appears to be impossible to achieve.

[sebsch]

Hi,

I am not an organizer. But if you check the section "Evaluation Details" of the technical annex document [1] on page 16 it says:

The time per frame is measured from the time of the service call until the response from the Contestant’s solution. To compensate for transmission delays, we run a calibration run first and subtract the transmission time from the actual measured time.

This is meant for task 1 but I guess it is safe to assume that the organizers will do the same for task 2.

Now for the intermediate evaluation, noone is stopping you from running both the simulation server and your solution client on the same machine, without any virtual machines in between. This is actually much more convenient for development and I guess we are not the only team doing that. The delay in this case is negligible.

Hope this helps,
Sebastian

[1]: http://projects.laas.fr/euroc/euroc_qualif_c3/EuRoC_Call4Challengers_TechnicalAnnex_C3.pdf

[jeffdelmerico]

Hi Sebastian,

Thank you for your thoughts. I must have missed that since it was regarding Task 1, but it makes sense that the policy would be the same. I will attempt to test that with the intermediate results server, but I suspect that you are right.

Thanks,
Jeff

[markusachtelik]

Sebastian, thanks for the explanation!

Jeff, we are aware that replaying the data takes a considerable amount of time, which is unfortunately not in our power. We do the mentioned calibration run to compensate for networking delays. Similar to task1, we measure the time between request/response, which cancels out the time that rosbag needs to load the data. Then, we sum up the single frame times to the total time.

Best, Markus

[jeffdelmerico]

Thank you very much for the explanation, Markus!

[jeffdelmerico]

Hi Markus,

As a follow-up question, is the time for the calibration run factored into the scoring on the intermediate evaluation site?

I set my machine up with the "Alternative Setup" with only 1 virtual machine for the client, in order to recreate the communication environment of the actual evaluation. In this configuration, I ran a trial for Task 2.1 using the unmodified code from the euroc_solution_t2 code (solution.cpp). This node just receives the service calls and returns, doing no additional processing. When I submitted the bag from this trial, I received the full 4.0 points for time, but the total_duration was about 11 seconds. This does not leave much room for actual data processing, unless the time for the calibration run is not deducted from the total_duration that is reported and scored by the evaluation server. I'm just trying to get an accurate picture of our current performance, so any feedback you can provide will be appreciated.

Thanks,
Jeff

[markusachtelik]

Hi Jeff,

we decided not to deduct the transmission time in the intermediate evaluation in the end, since the transmission time at this stage would be more guessing, as it may differ on each machine. Therefore, you may not have top score in the intermediate evaluation, and it's likely that it will be better in the final evaluation. We figured this way would be better than suggesting shorter computation times by deducting a transmission time that we don't really know.

For the true time that your processing took, you can deduct the 11s to get a figure where your results would be. It's actually good to know about the delays you're having, we observed transmission delays in the same order. We will do this dry run in the final evaluation as well, and deduct the time.

Hope that helps!

[jeffdelmerico]

Thank you very much, Markus! This seems totally reasonable, and I will take that into consideration with our intermediate results.

[rpretten]

hi,
how did you get an overall callback time of 11 seconds?
I used the alternative setup too and evaluated the callback times on two different systems:

1.) intel i7 960, 3.2 Ghz (Desktop)

2_1: 15 - 17 seconds
2_2: 9 - 10 seconds
2_3: 12 - 13 second

2.) intel i7 2630QM, 2.0 Ghz (Laptop)
This is the only laptop we have that is similar to the final evaluation system.
Our laptop has a better processor (one generation), therefore the timings should be a bit better than the final ones.

2_1: 28 - 30 seconds
2_2: 16 - 17 seconds
2_3: 21 - 22 seconds

Did you ever test the timings with the final evaluation system?
Since there were some scenario changes, will in the final evaluation the server be installed natively on the system or will it run in a vbox too?
I observed that If the server runs in a vbox too, the callback timings will further increase and have a higher std-deviation.
How will you finally calculate the transmission time to deduct?

regards,
Rudolf