EuRoC Forum

MathieuG wrote:Hi Kapi.m,

Scoring reads the information you send to the t1_2_worpiece topic. Scoring gives you this error because it doesn't find the information or the information are not PoseStamped messages. Run "rostopic echo /t1_2_workpiece" and check that your results are published.

Regards,
Mathieu

Hi Mathieu,

thanks for your hint, today i finally solve our problem. Problem was that we were putting position and orientation to the PoseStamped message as a float instead of double. Strange is, that either rostopic echo and rqt_bag didn't show any problem. Both told us that we are publishing PoseStamped messages corectly. But after explicit conversion to the double everything runs smoothly.

Thanks for your support,
Kapi

Please note that the bug regarding the evaluation of Task 1.1 was fixed. Consequently, all previous results in the ranking of that task were reset.

Hi all, again i'm facing some strange behaviour of intermediate evaluation. Sometimes, when i'm evaluating bag from task 1.2, i've got this error message:

Error:
error processing your logfile:
cmd output:


Error :
'time_budget'

retval: 1

I have no idea why is this happening. After detecting -1 on time_budget topic I'am ASAP returning from solve_task action. Do you have any idea why is this happening? Should I return when time_budget == 0? (I dont think so, because in technical annex is "Return from the solve_task action call when time_budget goes below 0")

Thanks for your help,
Kapi

Dear Kapi.m,

this was due to a small bug in the evaluation code. (related with the moment of returning from the solve_task action)
The fix will be installed asap.

Best

Thanks for info. I've got another question about inter. evaluation I evaluated task 1.1 and had got these results:

Code: Select all

***********************************************
*************** TASK 1.1 RESULTS **************
***********************************************
gesture correctly recognized: no
gesture correctly recognized: yes
gesture correctly recognized: yes
gesture correctly recognized: yes
gesture correctly recognized: yes
gesture correctly recognized: yes
----------- Scores ---------------
score task 1.1: 0

and

Code: Select all

***********************************************
*************** TASK 1.1 RESULTS **************
***********************************************
gesture correctly recognized: yes
gesture correctly recognized: yes
gesture correctly recognized: no
gesture correctly recognized: yes
gesture correctly recognized: no
gesture correctly recognized: yes
gesture correctly recognized: yes
gesture correctly recognized: yes
gesture correctly recognized: yes
gesture correctly recognized: yes
----------- Scores ---------------
score task 1.1: 2

In the first result, I guess that I should to receive 2 points (because of 5 gestures corectly recognized and no false positives) and in the second 5 points (8 TP, 0 FP). Am I right, or am I missing something?

Best,
Michal

Ok, with new output of evaluation service i see why i get those poins. But i don't understant, why are you penalize false negatives, when in scoring annex you specified penalty only for false positives.

Code: Select all

human trajectory: 7
   is gesture: True
   correctly recognized: False
   counting as: -1

I guess that this should to be for 0 points, not for -1.

Dear Kapi,

you are right the current evaluation penalizes false negatives as well (contrary to the scoring annex).
Until it is fixed you can get calculate your score by just taking into account only the true positives and the false positives.

Best

Hi all,
we recently tested own solution for task 1.1. This is the result:

Code: Select all

***********************************************
*************** TASK 1.1 RESULTS **************
***********************************************
human trajectory: 1
   is gesture: False
   correctly recognized: True
   counting as: 0
human trajectory: 2
   is gesture: False
   correctly recognized: True
   counting as: 0
human trajectory: 3
   is gesture: True
   correctly recognized: True
   counting as: 1
human trajectory: 4
   is gesture: False
   correctly recognized: True
   counting as: 0
human trajectory: 5
   is gesture: True
   correctly recognized: True
   counting as: 1
human trajectory: 6
   is gesture: False
   correctly recognized: True
   counting as: 0
human trajectory: 7
   is gesture: False
   correctly recognized: True
   counting as: 0
human trajectory: 8
   is gesture: True
   correctly recognized: True
   counting as: 1
human trajectory: 9
   is gesture: False
   correctly recognized: True
   counting as: 0
human trajectory: 10
   is gesture: False
   correctly recognized: True
   counting as: 0
human trajectory: 11
   is gesture: False
   correctly recognized: True
   counting as: 0
human trajectory: 12
   is gesture: True
   correctly recognized: True
   counting as: 1
human trajectory: 13
   is gesture: False
   correctly recognized: True
   counting as: 0
human trajectory: 14
   is gesture: False
   correctly recognized: True
   counting as: 0
human trajectory: 15
   is gesture: False
   correctly recognized: True
   counting as: 0
human trajectory: 16
   is gesture: False
   correctly recognized: True
   counting as: 0
human trajectory: 17
   is gesture: False
   correctly recognized: True
   counting as: 0
human trajectory: 18
   is gesture: True
   correctly recognized: True
   counting as: 1
human trajectory: 19
   is gesture: False
   correctly recognized: True
   counting as: 0
human trajectory: 20
   is gesture: False
   correctly recognized: True
   counting as: 0
human trajectory: 21
   is gesture: False
   correctly recognized: True
   counting as: 0
----------- Scores ---------------
score task 1.1: 2


Could it be a possible new bug in the scoring system? Are we doing something wrong?

Thank you.
Best.
Stefano

Hi Stefano,

do you have the newest simulator? In the output you posted there are only 5 pointing gestures. This bug was repaired recently with new version of simulator. Now there should always be 10 pointing and 10 non pointing gestures.

Best,
Kapi

I currently get the following error when uploading a 31 MB compressed Task 2 bag file:

Code: Select all

Error:
error processing your logfile:
cmd output:


retval: 137

Any clue to what might be wrong with it?