EuRoC C2S2 Benchmarking Rules




EuRoC C2S2 Benchmarking Rules

Postby Peter Lehner » Mon 28. Sep 2015, 10:39

This topic states the questions and answers concerning the benchmarking task rules for EuRoC Challenge 2 Stage 2.

Please post questions concerning the benchmarking task rules here and we will summarize these questions in the top post.

Please read the benchmarking task rules first. You can find the document in the euroc_c2 subversion in the 'rules' folder.

Please subscribe to this topic in order to be notified of updates.

General Questions:
  • lighting conditions: will the blinds be closed during benchmark evaluation? The blinds will be closed.
  • what is meant by "the time is stopped for the task" (p. 13 in the pdf rules)? We will start stopping the time of the task once you signal us you are ready for the first try.
  • which is the maximum size for a marker? can a full table be covered? is there a limit on the number of markers? The size of markers and number are not limited.
  • can the position of the markers change after the beginning of the 2 hour evaluation period? Yes you may change markers in between tasks, but not during the execution of one task. We will always place the dynamic objects after you have added your markers to the environment.
  • markers may have a 3D volume or can they only be sheets of paper? Only flat 2D markers are allowed.
  • how many people are inside the laboratory during the evaluation? Will they be in the robot moving area? At least one team member of DLR will monitor the evaluation. No people are allowed within the robot moving area.
  • will teams have access (during the code camps) to complete setups (positions of SLCs, bolts, nuts and washers) of identical difficulty as the ones used for the start of task 1 and task 2? All teams have access to the same laboratory which is used during the evaluation during the code camps. Setting up the tasks during testing is the task of the individual teams.
  • will teams have access to the other teams 2 hour videos for each task (after all teams have finished their evaluation week, of course)? We will not publish the videos.
  • will the simulator be updated to have the same TF names and orientations as the real hardware system? The TFs in the simulator and the real system have different meaning as the real system contains calibrated transformations which are different from the ideal transformations of the simulator. Therefore we discourage using the same names and orientations.
  • will the simulator be able to grasp objects? We provide the simulator "as is" as a basis for kinematic simulation. We do not support contact and dynamics.

Task1 Questions:
  • Will the SLCs be empty or contain something inside? The SLCs will contain nuts and washers. Extra points are awarded for carrying the SLCs to the goal without dropping parts.
  • can the fixtures for the assembly task be covered (possibly with a marker) during task 1? Yes.
  • can you provide a better specification of the possible positions and orientations of the SLCs in the tables and shelf (eg: minimum distance to the wall, minimum distance between SLCs, can a SLC be stacked on another SLC, admissible orientations in the shelf, minimum distance to the corners of the shelf, shelf levels that will be used, ...)? We updated the specification (see p. 7).
  • will the goal region always be half of the goal table? always the same table half? Yes. Yes.
  • Which is the maximum height and weight of nuts and washers inside the SLCs? We updated the specification (see p. 7).
  • Will the blue pad for the nuts and bolts of the task 2 be fixed to the pickup table during the task 1? Yes, they will remain fixed for task 1. You may cover the fixtures with 2D "markers" for task 1.
  • Could the SLCs be located anywhere in the pickup table? Even on the blue pad area? The SLCs will not stand on the fixtures.

Task2 Questions:
  • dimensions and possible positions of the "fixed area" for bolts, nuts and washers in task 2? We upated the specification (see p. 10).
  • can you provide a better specification of the possible positions and orientations of the bolts, nuts and washers in the table (eg: minimum distance to the wall, minimum distance between parts, can nuts be placed on top of other nuts?...)? We upated the specification (see p. 10).
  • which is the orientation of the washers (leaning to the wall or to the center of the room)? minimum distance between washers? The orientation of the washers will not be specified. The washers are not in contact.
  • How should the assembled bolts be placed after assembly? Anywhere on the assembly table, the orientation can be arbitrary (lying or standing).


Freestyle Questions:
  • does task setup (limited to 2 hours) also includes hardware changes to the robot platform? No, you are allowed to do hardware changes before the setup.
  • the challenger team can only use one camera for recording the freestyle? No, you may record the freestyle with as many cameras as you would like to.
  • can the team use non-fixed cameras for recording freestyle? Yes.
  • can teams also submit an edited video (containing only recordings from the 2 hour freestyle benchmark) to be passed to the independent reviewers? Yes.
Peter Lehner
 
Posts: 53
Joined: Fri 27. Jun 2014, 14:33

by Advertising » Mon 28. Sep 2015, 10:39

Advertising
 

Re: EuRoC C2S2 Benchmarking Rules Questions

Postby nunolau » Thu 29. Oct 2015, 11:19

From: TIMAIRIS

Hi everyone,

Following our first code camp, we had a rule discussion among the team and we came up with an extensive list of questions that we would like to be clarified.
The list is split into general question, and questions regarding tasks 1 and 2 and freestyle.

We would like to have these questions answered by our host at DLR.

#### LIST OF QUESTIONS

## general
- lighting conditions: will the blinds be closed during benchmark evaluation?
- what is meant by "the time is stopped for the task" (p. 13 in the pdf rules)?
- which is the maximum size for a marker? can a full table be covered? is there a limit on the number of markers?
- can the position of the markers change after the beginning of the 2 hour evaluation period?
- markers may have a 3D volume or can they only be sheets of paper?
- how many people are inside the laboratory during the evaluation? Will they be in the robot moving area?
- will teams have access (during the code camps) to complete setups (positions of SLCs, bolts, nuts and washers) of identical difficulty as the ones used for the start of task 1 and task 2?
- will teams have access to the other teams 2 hour videos for each task (after all teams have finished their evaluation week, of course)?
- will the simulator be updated to have the same tf names and orientations as the real hardware system?
- will the simulator be able to grasp objects?

## task 1
- can the fixtures for the assembly task be covered (possibly with a marker) during task 1?
- can you provide a better specification of the possible positions and orientations of the SLCs in the tables and shelf (eg: minimum distance to the wall, minimum distance between SLCs, can a SLC be stacked on another SLC, admissible orientations in the shelf, minimum distance to the corners of the shelf, shelf levels that will be used, ...)?
- will the goal region always be half of the goal table? always the same table half?
- Which is the maximum height and weight of nuts and washers inside the SLCs?

## task 2
- dimensions and possible positions of the "fixed area" for bolts, nuts and washers in task 2?
- can you provide a better specification of the possible positions and orientations of the bolts, nuts and washers in the table (eg: minimum distance to the wall, minimum distance between parts, can nuts be placed on top of other nuts?...)?
- which is the orientation of the washers (leaning to the wall or to the center of the room)? minimum distance between washers?

## freestyle
- does task setup (limited to 2 hours) also includes hardware changes to the robot platform?
- the challenger team can only use one camera for recording the freestyle?
- can the team use non-fixed cameras for recording freestyle?
- can teams also submit an edited video (containing only recordings from the 2 hour freestyle benchmark) to be passed to the independent reviewers?
nunolau
 
Posts: 3
Joined: Thu 29. Oct 2015, 10:54

Re: EuRoC C2S2 Benchmarking Rules

Postby Peter Lehner » Thu 29. Oct 2015, 13:37

Hi TIMAIRIS,

We have answered all the questions which we can answer immediately and added them to the top post. We will specify all unanswered questions which require discussion on our part on the 1. of December once every team has finished their first code camp.

Best,
Peter
Peter Lehner
 
Posts: 53
Joined: Fri 27. Jun 2014, 14:33

Re: EuRoC C2S2 Benchmarking Rules

Postby Peter Lehner » Tue 1. Dec 2015, 14:26

We have updated the questions which required more in depth discussion directly in the task specification. Please refer to page 7 and 10 to for the more precise definitions.
Peter Lehner
 
Posts: 53
Joined: Fri 27. Jun 2014, 14:33

Re: EuRoC C2S2 Benchmarking Rules

Postby Peter Lehner » Fri 15. Jan 2016, 08:50

I have added and answered the following question:

How should the assembled bolts be placed after assembly?

Anywhere on the assembly table, the orientation can be arbitrary (lying or standing).
Peter Lehner
 
Posts: 53
Joined: Fri 27. Jun 2014, 14:33

Re: EuRoC C2S2 Benchmarking Rules

Postby jmartinezot » Mon 25. Jan 2016, 10:46

From RSAII:

Will the blue pad for the nuts and bolts of the task 2 be fixed to the pickup table during the task 1?
Could the SLCs be located anywhere in the pickup table? Even on the blue pad area?
jmartinezot
 
Posts: 1
Joined: Mon 25. Jan 2016, 10:39

Re: EuRoC C2S2 Benchmarking Rules

Postby Peter Lehner » Mon 25. Jan 2016, 10:53

jmartinezot wrote:From RSAII:

Will the blue pad for the nuts and bolts of the task 2 be fixed to the pickup table during the task 1?
Could the SLCs be located anywhere in the pickup table? Even on the blue pad area?


I have added and aswered the questions in the first post above.
Peter Lehner
 
Posts: 53
Joined: Fri 27. Jun 2014, 14:33

Re: EuRoC C2S2 Benchmarking Rules

Postby jmartinezo » Tue 15. Mar 2016, 10:16

From RSAII:

Hi all,

Is it possible to modify the lighting conditions in the room, during the freestyle task, with the use of a screen attached to the ceiling?
jmartinezo
 
Posts: 1
Joined: Tue 15. Mar 2016, 10:12

Re: EuRoC C2S2 Benchmarking Rules

Postby andreasdoemel » Tue 15. Mar 2016, 15:10

During the freestyle you can modify the environment however you want.
During Benchmark I see no problem from our side to let you modify the lightning condition.
However we want to give other teams the opportunity to state their opinion before we modify the rules.
If we do not receive any message in the next week I will add this to the rules in the first post
andreasdoemel
 
Posts: 1
Joined: Tue 15. Mar 2016, 13:30

Re: EuRoC C2S2 Benchmarking Rules

Postby nunolau » Wed 16. Mar 2016, 11:37

We would prefer to have the same lighting condition for every team during the Benchmark, i.e., blinds closed and all lights on.
We think this provides a more standard environment and a more fair Benchmark evaluation.

We would also like to have the complete specification of the Freestyle Evaluation Process.
We were surprised when we were informed that teams can actually have Achievements that are greater than one in a single metric, if they surpass the Target. In fact, we think this is a bad idea.
Now, it is not clear if the evaluation based on the quantifiable objectives/metrics will be saturated at 75 points or not.
Can theoretically a team achieve the 75 points from a single metric by obtaining an Achievement significantly higher than its Target? We were obliged to define at least 3 metrics, hence we find it strange that teams can get the maximum evaluation from a single metric, unless the maximum is unbounded (which is also strange, in our opinion).
The splitting of the 75 difficulty points is, in any case, a very difficult task, but we feel it is much more difficult if the Achievements can be greater than one.
We would also like to know how is the coolness factor evaluation merged in the final result and if there are any additional evaluation factors (interview?, others?).
Could you provide a specific example on how the evaluation will be performed when a team obtains Achievements with a classification higher than 1?

Best,
TIMAIRIS
nunolau
 
Posts: 3
Joined: Thu 29. Oct 2015, 10:54

Next


Similar topics

EuRoC C2S2 Software Versions
Forum: Stage 2 - Benchmarking
Author: Peter Lehner
Replies: 0

Return to Stage 2 - Benchmarking

Who is online

No registered users

cron