## InternImage-based Baseline for CVPR23 OpenLane-V2 Challenge!!!!
We improve our baseline with a more powerful image backbone: **InternImage**, which shows its excellent ability within a series of leaderboards and benchmarks, such as *COCO* and *nuScenes*.
We improve our baseline with a more powerful image backbone: **InternImage**, which shows its excellent ability within a
series of leaderboards and benchmarks, such as *COCO* and *nuScenes*.
#### 1. Requirements
```bash
python>=3.8
torch==1.11
...
...
@@ -13,8 +14,8 @@ mmsegmentation==0.29.1
timm
```
### 2. Install DCNv3 for InternImage
```bash
cd plugin/mmdet3d/baseline/models/backbones/ops_dcnv3
bash make.sh
...
...
@@ -28,11 +29,10 @@ bash make.sh
Notes: InternImage provides abundant pre-trained model weights that can be used!!!
<palign="right">(<ahref="#top">back to top</a>)</p>
## Benchmark and Leaderboard
We will provide an initial benchmark on the OpenLane-V2 dataset, please stay tuned for the release.
Currently, we are maintaining leaderboards on the [*val*](https://paperswithcode.com/sota/3d-lane-detection-on-openlane-v2-2) and [*test*](https://eval.ai/web/challenges/challenge-page/1925/leaderboard/4549) split of `subset_A`.
We will provide an initial benchmark on the OpenLane-V2 dataset, please stay tuned for the release.
Currently, we are maintaining leaderboards on the [\*
val\*](https://paperswithcode.com/sota/3d-lane-detection-on-openlane-v2-2) and [\*
test\*](https://eval.ai/web/challenges/challenge-page/1925/leaderboard/4549) split of `subset_A`.
<palign="right">(<ahref="#top">back to top</a>)</p>
## Highlight - why we are exclusive?
### The world is three-dimensional - Introducing 3D lane
Previous datasets annotate lanes on images in the perspective view. Such a type of 2D annotation is insufficient to fulfill real-world requirements.
Following the [OpenLane](https://github.com/OpenDriveLab/OpenLane) dataset, we annotate **lanes in 3D space** to reflect their properties in the real world.
Previous datasets annotate lanes on images in the perspective view. Such a type of 2D annotation is insufficient to
fulfill real-world requirements. Following the [OpenLane](https://github.com/OpenDriveLab/OpenLane) dataset, we
annotate **lanes in 3D space** to reflect their properties in the real world.
### Be aware of traffic signals - Recognizing Extremely Small road elements
Not only preventing collision but also facilitating efficiency is essential.
Vehicles follow predefined traffic rules for self-disciplining and cooperating with others to ensure a safe and efficient traffic system.
Not only preventing collision but also facilitating efficiency is essential. Vehicles follow predefined traffic rules
for self-disciplining and cooperating with others to ensure a safe and efficient traffic system.
**Traffic elements** on the roads, such as traffic lights and road signs, provide practical and real-time information.
### Beyond perception - Topology Reasoning between lane and road elements
A traffic element is only valid for its corresponding lanes.
Following the wrong signals would be catastrophic.
Also, lanes have their predecessors and successors to build the map.
Autonomous vehicles are required to **reason** about the **topology relationships** to drive in the right way.
In this dataset, we hope to shed light on the task of **scene structure perception and reasoning**.
A traffic element is only valid for its corresponding lanes. Following the wrong signals would be catastrophic. Also,
lanes have their predecessors and successors to build the map. Autonomous vehicles are required to **reason** about
the **topology relationships** to drive in the right way. In this dataset, we hope to shed light on the task of **scene
structure perception and reasoning**.
### Data scale and diversity matters - building on Top of Awesome Benchmarks
Experience from the sunny day does not apply to the dancing snowflakes.
For machine learning, data is the must-have food.
We provide annotations on data collected in various cities, from Austin to Singapore and from Boston to Miami.
Experience from the sunny day does not apply to the dancing snowflakes. For machine learning, data is the must-have
food. We provide annotations on data collected in various cities, from Austin to Singapore and from Boston to Miami.
The **diversity** of data enables models to generalize in different atmospheres and landscapes.
<palign="right">(<ahref="#top">back to top</a>)</p>
## Task
The primary task of the dataset is **scene structure perception and reasoning**, which requires the model to recognize the dynamic drivable states of lanes in the surrounding environment.
The challenge of this dataset includes not only detecting lane centerlines and traffic elements but also recognizing the attribute of traffic elements and topology relationships on detected objects.
We define the **[OpenLane-V2 Score (OLS)](./docs/metrics.md#openlane-v2-score)**, which is the average of various metrics covering different aspects of the primary task:
The metrics of different subtasks are described below.
### 3D Lane Detection 🛣️
The [OpenLane](https://github.com/OpenDriveLab/OpenLane) dataset, which is the first real-world and the largest scaled 3D lane dataset to date, provides lane line annotations in 3D space.
Similarly, we annotate 3D lane centerlines and include the F-Score for evaluating predicted results of undirected lane centerlines.
Furthermore, we define the subtask of 3D lane detection as detecting directed 3D lane centerlines from the given multi-view images covering the whole horizontal FOV.
The instance-level evaluation metric of average precision $\text{DET}_{l}$ is utilized to measure the detection performance on lane centerlines (l).
The [OpenLane](https://github.com/OpenDriveLab/OpenLane) dataset, which is the first real-world and the largest scaled
3D lane dataset to date, provides lane line annotations in 3D space. Similarly, we annotate 3D lane centerlines and
include the F-Score for evaluating predicted results of undirected lane centerlines. Furthermore, we define the subtask
of 3D lane detection as detecting directed 3D lane centerlines from the given multi-view images covering the whole
horizontal FOV. The instance-level evaluation metric of average precision $\\text{DET}\_{l}$ is utilized to measure the
detection performance on lane centerlines (l).
<palign="center">
<imgsrc="./imgs/lane.gif"width="696px">
</p>
### Traffic Element Recognition 🚥
Traffic elements and their attribute provide crucial information for autonomous vehicles.
The attribute represents the semantic meaning of a traffic element, such as the red color of a traffic light.
In this subtask, on the given image in the front view, the location of traffic elements (traffic lights and road signs) and their attributes are demanded to be perceived simultaneously.
Compared to typical 2D detection datasets, the challenge is that the size of traffic elements is tiny due to the large scale of outdoor environments.
Similar to the typical 2D detection task, the metric of $\text{DET}_{t}$ is utilized to measure the performance of traffic elements (t) detection averaged over different attributes.
Traffic elements and their attribute provide crucial information for autonomous vehicles. The attribute represents the
semantic meaning of a traffic element, such as the red color of a traffic light. In this subtask, on the given image in
the front view, the location of traffic elements (traffic lights and road signs) and their attributes are demanded to be
perceived simultaneously. Compared to typical 2D detection datasets, the challenge is that the size of traffic elements
is tiny due to the large scale of outdoor environments. Similar to the typical 2D detection task, the metric of
$\\text{DET}\_{t}$ is utilized to measure the performance of traffic elements (t) detection averaged over different
We first define the task of recognizing topology relationships in the field of autonomous driving.
Given multi-view images, the model learns to recognize the topology relationships among lane centerlines and between lane centerlines and traffic elements.
The most similar task is link prediction in the field of graph, in which the vertices are given and only edges are predicted by models.
In our case, both vertices and edges are unknown for the model.
Thus, lane centerlines and traffic elements are needed to be detected first, and then the topology relationships are built.
Adapted from the task of link prediction, $\text{TOP}$ is used for topology among lane centerlines (ll) and between lane centerlines and traffic elements (lt).
We first define the task of recognizing topology relationships in the field of autonomous driving. Given multi-view
images, the model learns to recognize the topology relationships among lane centerlines and between lane centerlines and
traffic elements. The most similar task is link prediction in the field of graph, in which the vertices are given and
only edges are predicted by models. In our case, both vertices and edges are unknown for the model. Thus, lane
centerlines and traffic elements are needed to be detected first, and then the topology relationships are built. Adapted
from the task of link prediction, $\\text{TOP}$ is used for topology among lane centerlines (ll) and between lane
centerlines and traffic elements (lt).
<palign="center">
<imgsrc="./imgs/topology.gif"width="696px">
...
...
@@ -179,29 +197,34 @@ Adapted from the task of link prediction, $\text{TOP}$ is used for topology amon
<palign="right">(<ahref="#top">back to top</a>)</p>
## Data
The OpenLane-V2 dataset is a large-scale dataset for scene structure perception and reasoning in the field of autonomous driving.
Following [OpenLane](https://github.com/OpenDriveLab/OpenLane), the first 3D lane dataset, we provide lane annotations in 3D space.
The difference is that instead of lane lines, we annotate lane centerlines, which can be served as the trajectory for autonomous vehicles.
Besides, we provide annotations on traffic elements (traffic lights and road signs) and their attribute, and the topology relationships among lane centerlines and between lane centerlines and traffic elements.
The OpenLane-V2 dataset is a large-scale dataset for scene structure perception and reasoning in the field of autonomous
driving. Following [OpenLane](https://github.com/OpenDriveLab/OpenLane), the first 3D lane dataset, we provide lane
annotations in 3D space. The difference is that instead of lane lines, we annotate lane centerlines, which can be served
as the trajectory for autonomous vehicles. Besides, we provide annotations on traffic elements (traffic lights and road
signs) and their attribute, and the topology relationships among lane centerlines and between lane centerlines and
traffic elements.
The dataset is divided into two subsets.
**The `subset_A` serves as the primary subset and is utilized for the coming challenges and leaderboard, in which no external data, including the other subset, is allowed**.
The `subset_B` can be used to test the generalization ability of the model.
For more details, please refer to the corresponding pages: [use of data](./data/README.md), [notes of annotation](./docs/annotation.md), and [dataset statistics](./docs/statistics.md).
**The `subset_A` serves as the primary subset and is utilized for the coming challenges and leaderboard, in which no
external data, including the other subset, is allowed**. The `subset_B` can be used to test the generalization ability
of the model. For more details, please refer to the corresponding pages: [use of data](./data/README.md)
, [notes of annotation](./docs/annotation.md), and [dataset statistics](./docs/statistics.md).
[Download](./data/README.md#download) now to discover our dataset!
<palign="right">(<ahref="#top">back to top</a>)</p>
## Devkit
We provide a devkit for easy access to the OpenLane-V2 dataset.
After installing the package, the use of the dataset, such as loading images, loading meta data, and evaluating results, can be accessed through the API of `openlanv2`.
For more details on the API, please refer to [devkit](./docs/devkit.md).
<palign="right">(<ahref="#top">back to top</a>)</p>
We provide a devkit for easy access to the OpenLane-V2 dataset. After installing the package, the use of the dataset,
such as loading images, loading meta data, and evaluating results, can be accessed through the API of `openlanv2`. For
more details on the API, please refer to [devkit](./docs/devkit.md).
<palign="right">(<ahref="#top">back to top</a>)</p>
## Get Started
Please follow the steps below to get familiar with the OpenLane-V2 dataset.
1. Run the following commands to install the environment for setting up the dataset:
...
...
@@ -221,9 +244,9 @@ Please follow the steps below to get familiar with the OpenLane-V2 dataset.
-<imgsrc="https://ssl.gstatic.com/docs/doclist/images/drive_2022q3_32dp.png"alt="Google Drive"width="18"/> Google Drive,
Then put them into the `data/OpenLane-V2/` folder and unzip them.
The resulting folder hierarchy is described [here](./data/README.md#hierarchy).
Or use the following commands to download example data for a quick glance at the dataset:
Then put them into the `data/OpenLane-V2/` folder and unzip them. The resulting folder hierarchy is
described [here](./data/README.md#hierarchy). Or use the following commands to download example data for a quick
glance at the dataset:
```sh
cd data/OpenLane-V2
...
...
@@ -237,14 +260,16 @@ Please follow the steps below to get familiar with the OpenLane-V2 dataset.
<palign="right">(<ahref="#top">back to top</a>)</p>
## Train a Model
Plug-ins to prevail deep learning frameworks for training models are provided to start training models on the OpenLane-V2 dataset.
We appreciate your valuable feedback and contributions to plug-ins on different frameworks.
Plug-ins to prevail deep learning frameworks for training models are provided to start training models on the
OpenLane-V2 dataset. We appreciate your valuable feedback and contributions to plug-ins on different frameworks.
### mmdet3d
The [plug-in](./plugin/mmdet3d/) to MMDetection3d is built on top of [mmdet3d v1.0.0rc6](https://github.com/open-mmlab/mmdetection3d/tree/v1.0.0rc6) and tested under:
The [plug-in](./plugin/mmdet3d/) to MMDetection3d is built on top
of [mmdet3d v1.0.0rc6](https://github.com/open-mmlab/mmdetection3d/tree/v1.0.0rc6) and tested under:
- Python 3.8.15
- PyTorch 1.9.1
- CUDA 11.1
...
...
@@ -253,22 +278,26 @@ The [plug-in](./plugin/mmdet3d/) to MMDetection3d is built on top of [mmdet3d v1
- mmdet==2.26.0
- mmsegmentation==0.29.1
Please follow the [instruction](https://github.com/open-mmlab/mmdetection3d/blob/v1.0.0rc6/docs/en/getting_started.md) to install mmdet3d.
Assuming OpenLane-V2 is installed under `OpenLane-V2/` and mmdet3d is built under `mmdetection3d/`, create a soft link to the plug-in file:
Please follow the [instruction](https://github.com/open-mmlab/mmdetection3d/blob/v1.0.0rc6/docs/en/getting_started.md)
to install mmdet3d. Assuming OpenLane-V2 is installed under `OpenLane-V2/` and mmdet3d is built under `mmdetection3d/`,
create a soft link to the plug-in file:
```
└── mmdetection3d
└── projects
├── example_project
└── openlanev2 -> OpenLane-V2/plugin/mmdet3d
```
Then you can train or evaluate a model using the config `mmdetection3d/projects/openlanev2/configs/baseline.py`, whose path is replaced accordingly.
Options can be passed to enable supported functions during evaluation, such as `--eval-options dump=True dump_dir=/PATH/TO/DUMP` to save pickle file for submission and `--eval-options visualization=True visualization_dir=/PATH/TO/VIS` for visualization.
Then you can train or evaluate a model using the config `mmdetection3d/projects/openlanev2/configs/baseline.py`, whose
path is replaced accordingly. Options can be passed to enable supported functions during evaluation, such
as `--eval-options dump=True dump_dir=/PATH/TO/DUMP` to save pickle file for submission
and `--eval-options visualization=True visualization_dir=/PATH/TO/VIS` for visualization.
<palign="right">(<ahref="#top">back to top</a>)</p>
## Citation
Please use the following citation when referencing OpenLane-V2:
```bibtex
...
...
@@ -285,10 +314,13 @@ Please use the following citation when referencing OpenLane-V2:
## License
Before using the dataset, you should register on the website and agree to the terms of use of the [nuScenes](https://www.nuscenes.org/nuscenes).
Before using the dataset, you should register on the website and agree to the terms of use of
the [nuScenes](https://www.nuscenes.org/nuscenes).
Our dataset is built on top of the [nuScenes](https://www.nuscenes.org/nuscenes) and [Argoverse 2](https://www.argoverse.org/av2.html) datasets.
Before using the OpenLane-V2 dataset, you should agree to the terms of use of the [nuScenes](https://www.nuscenes.org/nuscenes) and [Argoverse 2](https://www.argoverse.org/av2.html) datasets respectively.
Our dataset is built on top of the [nuScenes](https://www.nuscenes.org/nuscenes)
and [Argoverse 2](https://www.argoverse.org/av2.html) datasets. Before using the OpenLane-V2 dataset, you should agree
to the terms of use of the [nuScenes](https://www.nuscenes.org/nuscenes)
and [Argoverse 2](https://www.argoverse.org/av2.html) datasets respectively.
All code within this repository is under [Apache License 2.0](./LICENSE).
