EuRoC Forum

We are running into some strange behaviour using the online evaluation of the bag files.

Attached, two links to bag files. The position information in these bags is (as good as) identical. We only changed the orientation. The plots online however show that the x and y axis are inverted.

https://www.dropbox.com/s/3w65j7rbi4j12v0/non-inverted.bag?dl=0
https://www.dropbox.com/s/ugkz2gezl3zp6bf/inverted.bag?dl=0

We understood from the technical annex that scoring would be purely on position errors, so we do not understand the dependence of the score (and the output plot) on the rotation information.

Feel free to verify these outcomes with the bagfiles attached.

Hey,

the scoring relies on the Kitty evaluation code. This computes relative pose errors between two frames first_frame and last_frame in the following way within the calcSequenceErrors() function:

Code: Select all

      // compute rotational and translational errors
      Matrix pose_delta_gt     = Matrix::inv(poses_gt[first_frame])*poses_gt[last_frame];
      Matrix pose_delta_result = Matrix::inv(poses_result[first_frame])*poses_result[last_frame];
      Matrix pose_error        = Matrix::inv(pose_delta_result)*pose_delta_gt;
      float r_err = rotationError(pose_error);
      float t_err = translationError(pose_error);


I believe what is meant in the technical annex is that only the relative translation error (t_err above) is considerd for scoring, not the rotation error (r_err above). But the relative translation error as computed above still depends on the orientation of the first frame. When considering relative pose errors, orientation (of the first frame) has to play a role or else the direction of your odometry estimate would not matter at all.

Cheers,
Sebastian

Thanks Sebsch for your answer.

This is actually exactly the code that we are using as well. We also did a test scenario prior to the release of the datasets where we disturbed the estimated pose estimate by a constant rotation offset. Please note that R_new = R_old*R_offset != R_offset*R_old.

In general, this evaluation implies that as long as you remain _consistent_ within your estimates, there should be no influence of the initial error in orientation. This means, that an initial error in yaw (or roll and pitch) does not influence your evaluation score.

best

Sammy