EuRoC Forum

[michieletto]

Hi all,
I just logged into the intermediate evaluation service, and I found a strange situation. Only a test has been submitted by XXX performing tasks 1.1 and 1.2.
Points given to task 1.2 was 10 (well done XXX), while point gained in task 1.1 was 6.

Now, my question: how can XXX obtain 6 points on task 1.1 when the expected results are supposed to be 0/2/5/10? Is that really possible?

Thank you for the answer.

Best.
Stefano

[Kapi.m]

Hi,

i guess they had 10 true positives and 4 false positives and because you recieve -1 point for every false positive, 10 - 4 = 6 points.

Kapi

[altella]

Hi all;

I am having problems when evaluating my executed tasks. In all of them the BAG file has the Time Budget to 0, so I receive the Out of Time Error.
Although I finish my task early on time and I send the set_succeded method from my task, the server does not stop execution until the Time Budget value reaches -10.
So, I my BAG file recorded for testing, I always have the time budget error.
Here is the output of the server:

Code: Select all

[INFO] [WallTime: 1412585612.246550] [28.809000] asking client to solve task1.2
Unhandled exception in thread started by
sys.excepthook is missing
lost sys.stderr
collision with workpiece detected
Unhandled exception in thread started by
sys.excepthook is missing
lost sys.stderr
[WARN] [WallTime: 1412585652.279543] [47.152000] After time budget publishing while
[INFO] [WallTime: 1412585652.280209] [47.153000] execution of task finished: 1.2
[ERROR] [WallTime: 1412585652.406697] [47.203000] Received comm state PREEMPTING when in simple state DONE with SimpleActionClient in NS /solve_task
killing /ipa325_record
[ WARN] [1412585652.418017784, 47.205000000]: Shutdown request received.
[ WARN] [1412585652.418057052, 47.205000000]: Reason given for shutdown: [user request]
killed
[INFO] [WallTime: 1412585652.537414] [47.257000] logging stopped
[INFO] [WallTime: 1412585652.537969] [47.257000] Waiting for bagfile to be ready
[INFO] [WallTime: 1412585654.652226] [48.257000] file found
[INFO] [WallTime: 1412585654.652813] [48.257000] finished executing task 1.2


[Kapi.m]

Hi altella,

have you got the latest version of sim server (04-Sep-2014 13:35)? In the erlier version time budget was updated every 10 sec (eg. 10, 0, -10), but in the latest version it is published every second (1, 0, -1), so i think that update to latest version will help you.

Sincerely
Kapi

[Kapi.m]

Hi all,

i 've got a problem with evaluation too. When i'm trying to evaluate task 1.2, i 've got this error message:

Error:
error processing your logfile:
cmd output:


Error :
'NoneType' object has no attribute 'position'

retval: 1

I'm publishing output as a PoseStamped message to the t1_2_workpiece topic, according to Technical Annex. Do you have any idea, why is this happening?

Thx,
Kapi

[michieletto]

Hi,

i guess they had 10 true positives and 4 false positives and because you recieve -1 point for every false positive, 10 - 4 = 6 points.

Kapi

Dear Kapi,
thanks for your answer! You are absolutely right!

Best.
Stefano

[MathieuG]

Hi Kapi.m,

Scoring reads the information you send to the t1_2_worpiece topic. Scoring gives you this error because it doesn't find the information or the information are not PoseStamped messages. Run "rostopic echo /t1_2_workpiece" and check that your results are published.

Regards,
Mathieu

[michieletto]

Hi,

i guess they had 10 true positives and 4 false positives and because you recieve -1 point for every false positive, 10 - 4 = 6 points.

Kapi

Dear Kapi,
thanks for your answer! You are absolutely right!

Best.
Stefano

Hi all,
Any explanation for the 20 points assigned in task 1.1?

Thanks.
Stefano

[MathieuG]

Hi all,

Scoring of task 1.1 doesn't fit the technical annex description, for now it is +1 for correct recognition, -1 for incorrect ones. 20pts is 20 correct recognitions so it should be 10pts. 6pts was 13 correct and 7 incorrect recognitions.
This error has already been reported to the designers of track 1 and I hope it will be changed soon.

Best,
Mathieu

[michieletto]

Dear Mathieu,
thank you for your, it clarifies the situation a lot!

Best.
Stefano