SGAligner: 3D Scene Alignment with Scene Graphs

ICCV 2023

Sayan Deb Sarkar¹, Ondrej Miksik², Marc Pollefeys^1,2, Daniel Barath¹, Iro Armeni¹

¹ETH Zurich ²Microsoft Mixed Reality & AI Labs

SGAligner aligns 3D scene graphs of environments using multi-modal learning and leverage the output for the downstream task of 3D point cloud registration.

[Project Webpage] [Paper]

News 📰

• [2025-04] Codebase updated + Ground truth checkpoints released.
• [2023-07] SGAligner accepted to ICCV 2023. 🔥
• [2023-05] We release SGAligner on arXiv with codebase with pre-trained predicted relationship checkpoint. Checkout our paper and website.

Code Structure 🎬

sgaligner/
├── example-data                  <- examples of data generated post preprocessing
├── data-preprocessing            <- subscan generation + preprocessing
├── configs                       <- configuration files
├── src
│   │── aligner                   <- SGAligner modules
│   │── datasets                  <- dataloader for 3RScan subscans
│   │── engine                    <- trainer classes
│   │── GeoTransformer            <- geotransformer submodule for registration
│   │── inference                 <- inference files for alignment + downstream applications
│   │── trainers                  <- train + validation loop (EVA + SGAligner)
│── utils                         <- util functions
│── README.md                    
│── scripts                       <- bash scripts for data generation + preprocesing + training
└── output                        <- folder that stores models and logs

Dependencies 📝

The main dependencies of the project are the following:

python: 3.8.15
cuda: 11.6

You can set up a conda environment as follows :

git clone --recurse-submodules -j8 git@github.com:sayands/sgaligner.git
cd sgaligner
conda env create -f req.yml

Note: Please follow PointCloudTransformer repository for installing pointnet2_ops_lib if you want to use PCT as the backbone.

Please follow the submodule for additional installation requirements and setup of GeoTransformer.

Downloads 💧

The pre-trained model and other meta files are available here.

Dataset Generation 🔨

Data Download And Preparation

Download 3RScan dataset from the official website and 3DSSG (accompanying GT scene graph annotation) from the official website. We use the official (full list of scan ids including reference + rescans) training split from here and validation split from here. - Download 3RScan.json from here and objects.json from here. - Download the class mapping file 3RScan.v2 Semantic Classes - Mapping.csv from here.

Follow the steps for data preparation:

1. Move all files of 3DSSG and 3RScan metadata files to the files/ directory within 3RScan. The structure would look like the following:

Scan3R/
├── files       <- all 3RScan and 3DSSG meta files (NOT the scan data)  
│    ├── 3RScan.json
|    ├── classes.txt
|    ├── train_scans.txt
|    ├── val_scans.txt
|    ├── wordnet_attributes.txt
|    ├── affordances.txt
|    ├── attributes.txt
|    ├── relationships.txt
|    ├── objects.json
|    ├── relationships.json
|    └── obj_attr.pkl
└── scans       <- scans

2. Generate labels.instances.align.annotated.v2.ply for each 3RScan scan, please refer to the repo from here.

3. Change the absolute paths in utils/define.py and paths in the config files under configs/scan3r/.

Subscan Pair Generation - 3RScan + 3DSSG

First, we create sub-scans from each 3RScan scan using the ground truth scene Graphs from the 3DSSG dataset and then calculate the pairwise overlap ratio for the subscans in a scan. Finally, we preprocess the data for our framework. The relevant code can be found in the data-preprocessing/ directory. You can use the following command to generate the subscans.

bash scripts/generate_data_scan3r_gt.sh

Note To adhere to our evaluation procedure, please do not change the seed value in the files in configs/ directory.

Generating Overlapping and Non-Overlapping Subscan Pairs

To generate overlapping and non-overlapping pairs, use :

python preprocessing/gen_all_pairs_fileset.py

This will create a fileset with the same number of randomly chosen non-overlapping pairs from the generated subscans as overlapping pairs generated before during subscan generation.

Training 🚄

To train SGAligner on 3RScan subscans generated from here, you can use :

cd src
python trainers/trainval_sgaligner.py --config ../configs/scan3r/scan3r_ground_truth.yaml

EVA Training

We also provide training scripts for EVA, used as a baseline after being adapted for scene graph alignment. To train EVA similar to SGAligner on the same data, you can use :

cd src
python trainers/trainval_eva.py --config ../configs/scan3r/scan3r_eva.yaml

We provide config files for the corresponding data in config/ directory. Please change the parameters in the configuration files, if you want to tune the hyper-parameters.

Evaluation 🚦

Graph Alignment + Point Cloud Registration

cd src
python inference/sgaligner/inference_align_reg.py --config ../configs/scan3r/scan3r_ground_truth.yaml --snapshot <path to SGAligner trained model> --reg_snapshot <path to GeoTransformer model trained on 3DMatch>

Finding Overlapping vs Non-Overlapping Pairs

❗ Run Generating Overlapping and Non-Overlapping Subscan Pairs before.

To run the inference, you need to:

cd src
python inference/sgaligner/inference_find_overlapper.py --config ../configs/scan3r/scan3r_gt_w_wo_overlap.yaml --snapshot <path to SGAligner trained model> --reg_snapshot <path to GeoTransformer model trained on 3DMatch>

3D Point Cloud Mosaicking

First, we generate the subscans per 3RScan scan using :

python data-preprocessing/gen_scan_subscan_mapping.py --split <the split you want to generate the mapping for>

And then, to run the inference, you need to:

cd src
python inference/sgaligner/inference_mosaicking.py --config ../configs/scan3r/scan3r_gt_mosaicking.yaml --snapshot <path to SGAligner trained model> --reg_snapshot <path to GeoTransformer model trained on 3DMatch>

Benchmark 📈

We provide detailed results and comparisons here.

3D Scene Graph Alignment (Node Matching)

Method	Mean Reciprocal Rank	Hits@1	Hits@2	Hits@3	Hits@4	Hits@5
EVA	0.867	0.790	0.884	0.938	0.963	0.977
$\mathcal{P}$	0.884	0.835	0.886	0.921	0.938	0.951
$\mathcal{P}$ + $\mathcal{S}$	0.897	0.852	0.899	0.931	0.945	0.955
$\mathcal{P}$ + $\mathcal{S}$ + $\mathcal{R}$	0.911	0.861	0.916	0.947	0.961	0.970
SGAligner	0.950	0.923	0.957	0.974	0.9823	0.987

3D Point Cloud Registration

Method	CD	RRE	RTE	FMR	RR
GeoTr	0.02247	1.813	2.79	98.94	98.49
Ours, K=1	0.01677	1.425	2.88	99.85	98.79
Ours, K=2	0.01111	1.012	1.67	99.85	99.40
Ours, K=3	0.01525	1.736	2.55	99.85	98.81

BibTeX 🙏

@inproceedings{Sarkar_2023_ICCV,
    author    = {Sarkar, Sayan Deb and Miksik, Ondrej and Pollefeys, Marc and Barath, Daniel and Armeni, Iro},
    title     = {SGAligner: 3D Scene Alignment with Scene Graphs},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {21927-21937}
}

Acknowledgments ♻️

In this project we use (parts of) the official implementations of the following works and thank the respective authors for open sourcing their methods:

• SceneGraphFusion (3RScan Dataloader)
• GeoTransformer (Registration)
• MCLEA (Alignment)