Unsuccessfully tracker and mapper #5
Description
Hi, I'm attempting stereoscopic 3D reconstruction using the EVK4 camera. I've appropriately reduced the event rate to minimize computational overhead. Since the EVK4 camera cannot capture images, I'm unable to configure bootstrap-related parameters. The pose data originates from the end-effector of the robotic arm. Without images, precise hand-eye calibration is impossible (at least for me), so I've only calibrated the angular relationship between the camera and the robotic arm's end-effector. Additionally, since EVK4 and Davis use different ROS message formats, I manually convert Prophesee-formatted events to DVS format when publishing ROS topics.
While 3D reconstruction runs, the results appear unusual, with depth values distributed across each layer, as shown in the figure. Do you have any suggestions? Do you know what might be causing this?
The calibration type I used is demo, with the approximate parameters as follows.
cam0:
cam_overlaps: [1]
camera_model: pinhole
distortion_coeffs: [-0.067504428925892521, 0.02097316291640404, 0.0034029851581029711, 0.0015693262472667095, 0.09736751449905745]
distortion_model: radtan
intrinsics: [868.96280119256289, 869.36903837500597, 654.70534654060225, 332.75201944013457]
resolution: [1280, 720]
rostopic: /davis/left/events
cam1:
T_cn_cnm1:
- [1., 0., 0., -0.12833783221917505]
- [0., 1., -0., -0.0023026847740744878]
- [-0., 0., 1., -0.0027014002409381627]
- [0.0, 0.0, 0.0, 1.0]
cam_overlaps: [0]
camera_model: pinhole
distortion_coeffs: [-0.06722815529366076, 0.054306850475124159, 0.0041742956869293005, 0.00047949270857777906, 0.011092095869707577]
distortion_model: radtan
intrinsics: [876.17746888315878, 876.2901341486413, 664.0052393034515, 337.71971118626209]
resolution: [1280, 720]
rostopic: /davis/right/events
Live mapper
<launch>
<param name="num_ev_per_map" value="50000" />
<param name="on_demand" value="false" />
<param name="max_duration" value="0.5" />
<param name="min_duration" value="0.001" />
<param name="init_wait_time" value="0.2" />
<param name="auto_copilot" value="true"/>
<param name="calib_path" value="$(find mapper_emvs_stereo)/calib/calib_ids.yaml"/>
<!-- Mapping module -->
<node pkg="mapper_emvs_stereo" type="run_emvs" name="run_emvs" output="screen" args="--flagfile $(find mapper_emvs_stereo)/cfg/tracker/alg1/ids.conf">
<remap from="pointcloud_local" to="/dvs_mapping/pointcloud" />
<remap from="pointcloud_global" to="/dvs_mapping/pointcloud_global" />
<remap from="Inverse_Depth_Map" to="/dvs_mapping/depthmap" />
</node>
</launch>
Live tracker
<launch>
<arg name="auto_trigger" default="true"/> <!-- whether to automatic startup the pipeline at the first data received -->
<arg name="events_topic" default="/davis/left/events" /> <!-- topic used by your sensor to output events -->
<arg name="bootstrap_image_topic" default="/davis/left/image_raw" /> <!-- dvs to use standard frames (relevant for svo-based bootstrapping )-->
<arg name="events_topic_right" default="/davis/right/events" />
<arg name="camera_name" default="ids" /> <!-- name of the camera and of the calibration file -->
<param name="camera_name" value="$(arg camera_name)" />
<param name="calib_file" value="$(find dvs_tracking)/parameters/calib/$(arg camera_name).yaml" />
<remap from="events" to="$(arg events_topic)" /> <!-- remap to topic used by your sensor -->
<remap from="remote_key" to="/evo/remote_key" />
<!-- <remap from="camera_info" to="/dvs/camera_info" /> -->
<param name="world_frame_id" value="world"/>
<param name="dvs_bootstrap_frame_id" value="/camera0" /> <!-- to substitute svo, change this with the frame published by your pipeline -->
<param name="dvs_frame_id" value="dvs0" />
<param name="gt_frame_id" value="hand" />
<!-- EVO: mapping + tracking -->
<param name="min_depth" value="0.2" /> <!-- voxel grid minimum depth -->
<param name="max_depth" value="2" /> <!-- voxel grid maximum depth -->
<param name="num_depth_cells" value="100" /> <!-- number of depth cells in which the depth axis is divided -->
<!-- Angle of view of the DSI (cone) -->
<param name="fov_virtual_camera_deg" value="67.0" />
<!-- Number of horizontal/vertical pixels in the DSI -->
<param name="virtual_width" value="1280" />
<param name="virtual_height" value="720" />
<!-- Map expansion -->
<node name="trigger_map_expansion" pkg="mapper_emvs_stereo" type="trigger_map_expansion.py" output="screen" >
<remap from="remote_key" to="evo/remote_key" />
<remap from="pointcloud" to="dvs_mapping/pointcloud" />
<param name="visibility_threshold" value="0.5" /> <!-- visibility of the map below which the update is triggered -->
<param name="coverage_threshold" value="0.1" /> <!-- minimum amount of pixels covered by reprojected map threshold -->
<param name="baseline_threshold" value="0.1" /> <!-- baseline / mean depth ration above which the update is triggered -->
<param name="rate" value="1" /> <!-- rate at which the node checks whether an expansion is needed -->
<param name="number_of_initial_maps_to_skip" value="0"/> <!-- starts checking updates conditions after this number of maps -->
</node> -->
<!-- Tracking module -->
<node name="dvs_tracking" pkg="dvs_tracking" type="dvs_tracking_ros" required="true" output="screen">
<param name="discard_events_when_idle" value="false"/> <!-- whether not to collect events when idle -->
<param name="batch_size" value="2000" /> <!-- batch-gradient descent batch size -->
<param name="max_iterations" value="100" /> <!-- maximum number of iterations in the optimization -->
<param name="pyramid_levels" value="2" /> <!-- number of pyramid levels used in the KLT process -->
<param name="map_blur" value="3" /> <!-- sigma of the gaussian filter applied to the reprojected map -->
<param name="noise_rate" value="30000" /> <!-- if events rate is below this value, the frame is skipped -->
<param name="frame_size" value="5000000" /> <!-- window of events considered -->
<param name="step_size" value="5000" /> <!-- minimum number of new events to wait before a pose update -->
<param name="max_event_rate" value="200000000" /> <!-- events processed are randomly sampled so that the rate is below this value -->
<param name="pose_mean_filter_size" value="5" /> <!-- median filter size (poses are median filtered) -->
<param name="events_per_kf" value="00000" /> <!-- events required for a new keyframe -->
<param name="event_map_overlap_rate" value="15" /> <!-- publishing rate of the visualizations -->
<param name="min_map_size" value="0"/> <!-- minimum number of map points to proceed with the update -->
<param name="min_n_keypoints" value="0"/> <!-- minimum number of extracted keypoints (LKSE3::keypoints_) required for a reliable tracking -->
<param name="auto_trigger" value="$(arg auto_trigger)"/> <!-- whether to autotrigger the tracking when a map is received -->
<remap from="pointcloud" to="dvs_mapping/pointcloud" />
</node>
<!-- Publish GT pose to tf frame "dvs_bootstrap_frame_id" -->
<node name="pose_to_tf" pkg="mapper_emvs_stereo" type="pose_to_tf.py" output="screen">
<param name="source_topic_name" value="/optitrack/davis_stereo" />
<param name="relative_to_first_pose" value="false" />
</node>
<node name="tf_to_camera_marker" pkg="evo_utils" type="tf_to_camera_markers.py" output="screen" >
<param name="marker_scale" value="0.2" />
</node>
<node name="snakify" pkg="evo_utils" type="snakify.py" output="screen" >
<param name="length" value="100" />
</node>
<node name="rqt_evo" pkg="rqt_evo" type="rqt_evo"></node>
<!-- visualization -->
<node name="dvs_renderer_left" pkg="dvs_renderer" type="dvs_renderer">
<remap from="events" to="$(arg events_topic)" />
<remap from="dvs_rendering" to="dvs_rendering_left" />
</node>
<node name="dvs_renderer_right" pkg="dvs_renderer" type="dvs_renderer">
<remap from="events" to="/davis/right/events" />
<remap from="dvs_rendering" to="dvs_rendering_right" />
</node>
<node type="rviz" name="rviz" pkg="rviz" args="-d $(find dvs_tracking)/rviz/demo.rviz" />
</launch>
```
conf
```
-out_path=/root/eKalibr/calibration_result/test7_dvs_cali
--calib_type=demo
--event_topic0=/davis/left/events
--event_topic1=/davis/right/events
--pose_topic=pose
--min_depth=0.2
--max_depth=2
--dimZ=10
--median_filter_size=5
--process_method=1
--stereo_fusion=2
--adaptive_threshold_c=10
--median_filter_size=7
--radius_search=0.01`
```