InvariantPlaceRecognitionPipeline

A module that implements an algorithm for generating global vector representations of multimodal data outdoors, invariant to changes in weather conditions and seasons. This invariance is achieved by using semantic segmentation masks and highlighting special elements of the scene.

Usage example

You should start with initializing neural model opr.models.place_recognition.base.LateFusionModel with the image, cloud, semantic and SOC modules. The recommended way to do this is to use the configs/model/place_recognition/multimodal_semantic_with_soc_outdoor.yaml config file to instantiate the model with Hydra and load the weights from the "weights/place_recognition/multimodal_semantic_with_soc_outdoor_nclt.pth" .

from hydra.utils import instantiate
from omegaconf import OmegaConf


PR_MODEL_CONFIG_PATH = "configs/model/place_recognition/multimodal_semantic_with_soc_outdoor.yaml"
PR_WEIGHTS_PATH = "weights/place_recognition/multimodal_semantic_with_soc_outdoor_nclt.pth"

pr_model_config = OmegaConf.load(PR_MODEL_CONFIG_PATH)
pr_model = instantiate(pr_model_config)
pr_model.load_state_dict(torch.load(PR_WEIGHTS_PATH))

In the similar manner you should initialize the registration model with the configs/model/registration/hregnet_light_feats.yaml config:

REG_MODEL_CONFIG_PATH = "configs/model/registration/hregnet_light_feats.yaml"
REG_WEIGHTS_PATH = "weights/registration/hregnet_light_feats_nuscenes.pth"

reg_model_config = OmegaConf.load(REGISTRATION_MODEL_CONFIG_PATH)
reg_model = instantiate(reg_model_config)
reg_model.load_state_dict(torch.load(REGISTRATION_WEIGHTS_PATH))

Then you should initialize the opr.pipelines.localization.base.LocalizationPipeline which consists of two sub-pipelines: opr.pipelines.place_recognition.base.PlaceRecognitionPipeline and opr.pipelines.registration.pointcloud.PointcloudRegistrationPipeline.

from opr.pipelines.place_recognition import PlaceRecognitionPipeline
from opr.pipelines.registration import PointcloudRegistrationPipeline
from opr.pipelines.localization import LocalizationPipeline

DATABASE_DIR = "/path/to/database"
DEVICE = "cuda"

pr_pipe = PlaceRecognitionPipeline(
    database_dir=DATABASE_DIR,
    model=pr_model,
    model_weights_path=PR_WEIGHTS_PATH,
    device=DEVICE,
)
reg_pipe = PointcloudRegistrationPipeline(
    model=reg_model,
    model_weights_path=REG_WEIGHTS_PATH,
    device=DEVICE,
    voxel_downsample_size=0.3,
    num_points_downsample=8192,
)
loc_pipe = LocalizationPipeline(
    place_recognition_pipeline=pr_pipe,
    registration_pipeline=reg_pipe,
    precomputed_reg_feats=True,
    pointclouds_subdir="lidar",
)

Then you can use the pipeline to infer the location of the input query data:

query_data = {
    "image_front": image_front,
    "mask_front": mask_front,
    "image_back": image_back,
    "mask_back": mask_back,
    "pointcloud_lidar_coords": pointcloud_lidar_coords,
    "pointcloud_lidar_feats": pointcloud_lidar_feats,
}

loc_pipe.infer(query_data)

The pipeline will return the output dictionary with the following keys:

"db_match_pose": the pose of the most similar record in the database
"db_match_idx": the index of the most similar record in the database
"estimated_pose": the estimated pose of the query data after registration

More usage examples can be found in the following notebooks: