The dataset is preprocessed into pickle files representing different collections, which then be used for training models or evaluation:
The dataset is preprocessed into pickle files representing different collections, which then be used for training models
or evaluation:
```sh
cd data
python OpenLane-V2/preprocess.py
python OpenLane-V2/preprocess.py
```
## Hierarchy
The hierarchy of folder `OpenLane-V2/` is described below:
```
└── OpenLane-V2
├── train
...
...
@@ -61,12 +66,14 @@ The hierarchy of folder `OpenLane-V2/` is described below:
```
-`[segment_id]` specifies a sequence of frames, and `[timestamp]` specifies a single frame in a sequence.
-`image/` contains images captured by various cameras, and `info/` contains meta data and annotations of a single frame.
-`image/` contains images captured by various cameras, and `info/` contains meta data and annotations of a single
frame.
-`data_dict_[xxx].json` notes the split of train / val / test under the subset of data.
## Meta Data
The json files under the `info/` folder contain meta data and annotations for each frame.
Each file is formatted as follows:
The json files under the `info/` folder contain meta data and annotations for each frame. Each file is formatted as
follows:
```
{
...
...
@@ -84,19 +91,20 @@ Each file is formatted as follows:
'intrinsic': <dict> -- intrinsic parameters of the camera
},
...
}
}
'pose': <dict> -- ego pose
'annotation': <dict> -- anntations for the current frame
}
```
## Annotations
For a single frame, annotations are formatted as follow:
```
{
'lane_centerline': [ (n lane centerlines in the current frame)
{
{
'id': <int> -- unique ID in the current frame
'points': <float> [n, 3] -- 3D coordiate
'confidence': <float> -- confidence, only for prediction
...
...
@@ -104,7 +112,7 @@ For a single frame, annotations are formatted as follow:
...
],
'traffic_element': [ (k traffic elements in the current frame)
{
{
'id': <int> -- unique ID in the current frame
'category': <int> -- traffic element category
1: 'traffic_light',
...
...
@@ -133,11 +141,11 @@ For a single frame, annotations are formatted as follow:
}
```
-`id` is the identifier of a lane centerline or traffic element and is consistent in a sequence.
For predictions, it can be randomly assigned but unique in a single frame.
-`topology_lclc` and `topology_lcte` are adjacent matrices, where row and column are sorted according to the order of the lists `lane_centerline` and `traffic_element`.
It is a MUST to keep the ordering the same for correct evaluation.
For ground truth, only 0 or 1 is a valid boolean value for an element in the matrix.
For predictions, the value varies from 0 to 1, representing the confidence of the predicted relationship.
- #lane_centerline and #traffic_element are not required to be equal between ground truth and predictions.
In the process of evaluation, a matching of ground truth and predictions is determined.
-`id` is the identifier of a lane centerline or traffic element and is consistent in a sequence. For predictions, it
can be randomly assigned but unique in a single frame.
-`topology_lclc` and `topology_lcte` are adjacent matrices, where row and column are sorted according to the order of
the lists `lane_centerline` and `traffic_element`. It is a MUST to keep the ordering the same for correct evaluation.
For ground truth, only 0 or 1 is a valid boolean value for an element in the matrix. For predictions, the value varies
from 0 to 1, representing the confidence of the predicted relationship.
- #lane_centerline and #traffic_element are not required to be equal between ground truth and predictions.
In the process of evaluation, a matching of ground truth and predictions is determined.
The road structure cognition task is defined as inputting the surrounding view images, reconstructing the high-precision map of the self-vehicle, and outputting the recognition result of the direction of the self-vehicle.
The specific expansion is to input the surrounding view images of the vehicle, HDMap; the output is the lane centerlines, the traffic signs, the topology of the lane centerlines, and the correspondence between lanes centerlines and traffic signs.
Below are examples of visualizing annotations and relationships between different elements on 2D images.
The road structure cognition task is defined as inputting the surrounding view images, reconstructing the high-precision
map of the self-vehicle, and outputting the recognition result of the direction of the self-vehicle. The specific
expansion is to input the surrounding view images of the vehicle, HDMap; the output is the lane centerlines, the traffic
signs, the topology of the lane centerlines, and the correspondence between lanes centerlines and traffic signs. Below
are examples of visualizing annotations and relationships between different elements on 2D images.
Given the ground truth and predictions, which are formatted dict or the path to pickle storing the dict that ground truth is preprocessed pickle file and predictions are formatted as described [here](./submission.md#format), this function returns a dict storing all metrics defined by our task.
Given the ground truth and predictions, which are formatted dict or the path to pickle storing the dict that ground
truth is preprocessed pickle file and predictions are formatted as described [here](./submission.md#format), this
function returns a dict storing all metrics defined by our task.
## openlanev2.io
This subpackage wraps all IO operations of the OpenLane-V2 devkit.
It can be modified for different IO operations.
This subpackage wraps all IO operations of the OpenLane-V2 devkit. It can be modified for different IO operations.
Given a data_dict storing identifiers of frames, this function collects meta data the frames and stores it into a pickle file for efficient IO for the following operations.
Given a data_dict storing identifiers of frames, this function collects meta data the frames and stores it into a pickle
file for efficient IO for the following operations.
#### `check_results(results : dict) -> None`
Check format of results.
## openlanev2.visualization
This subpackage provides tools for visualization. Please refer to the [tutorial](../tutorial.ipynb) for examples.
To evaluate performances on different aspects of the task, several metrics are adopted:
- $\text{DET}_{l}$ for mAP on directed lane centerlines,
- $\text{DET}_{t}$ for mAP on traffic elements,
- $\text{TOP}_{ll}$ for mAP on topology among lane centerlines,
- $\text{TOP}_{lt}$ for mAP on topology between lane centerlines and traffic elements.
- $\\text{DET}\_{l}$ for mAP on directed lane centerlines,
- $\\text{DET}\_{t}$ for mAP on traffic elements,
- $\\text{TOP}\_{ll}$ for mAP on topology among lane centerlines,
- $\\text{TOP}\_{lt}$ for mAP on topology between lane centerlines and traffic elements.
We consolidate the above metrics by computing an average of them, resulting in the **OpenLane-V2 Score (OLS)**.
### Lane Centerline
We adopt the average precision (AP) but define a match of lane centerlines by considering the discrete Frechet distance in the 3D space.
The mAP for lane centerlines is averaged over match thresholds of $\\{1.0, 2.0, 3.0\\}$ on the similarity measure.
We adopt the average precision (AP) but define a match of lane centerlines by considering the discrete Frechet distance
in the 3D space. The mAP for lane centerlines is averaged over match thresholds of $\\{1.0, 2.0, 3.0\\}$ on the
similarity measure.
### Traffic Element
Similarly, we use AP to evaluate the task of traffic element detection.
We consider IoU distance as the affinity measure with a match threshold of $0.75$.
Besides, traffic elements have their own attribute.
For instance, a traffic light can be red or green, which indicates the drivable state of the lane.
Therefore, the mAP is then averaged over attributes.
Similarly, we use AP to evaluate the task of traffic element detection. We consider IoU distance as the affinity measure
with a match threshold of $0.75$. Besides, traffic elements have their own attribute. For instance, a traffic light can
be red or green, which indicates the drivable state of the lane. Therefore, the mAP is then averaged over attributes.
### Topology
The topology metrics estimate the goodness of the relationship among lane centerlines and the relationship between lane centerlines and traffic elements.
To formulate the task of topology prediction as a link prediction problem, we first determine a match of ground truth and predicted vertices (lane centerlines and traffic elements) in the relationship graph.
We choose Frechet and IoU distance for the lane centerline and traffic element respectively.
Also, the metric is average over different recalls.
We adopt mAP from link prediction, which is defined as a mean of APs over all vertices.
Two vertices are regarded as connected if the predicted confidence of the edge is greater than $0.5$.
The AP of a vertex is obtained by ranking all predicted edges and calculating the accumulative mean of the precisions:
The topology metrics estimate the goodness of the relationship among lane centerlines and the relationship between lane
centerlines and traffic elements. To formulate the task of topology prediction as a link prediction problem, we first
determine a match of ground truth and predicted vertices (lane centerlines and traffic elements) in the relationship
graph. We choose Frechet and IoU distance for the lane centerline and traffic element respectively. Also, the metric is
where $N(v)$ denotes ordered list of neighbors of vertex $v$ ranked by confidence and $P(v)$ is the precision of the $i$-th vertex $v$ in the ordered list.
where $N(v)$ denotes ordered list of neighbors of vertex $v$ ranked by confidence and $P(v)$ is the precision of the
$i$-th vertex $v$ in the ordered list.
Given ground truth and predicted connectivity of lane centerlines, the mAP is calculated on $G^{l} = (V^{l}, E^{l})$ and $\hat{G}^{l} = (\hat{V}^{l}, \hat{E}^{l})$.
As the given graphs are directed, e.g., the ending point of a lane centerline is connected to the starting point of the next lane centerline, we take the mean of mAP over graphs with only in-going or out-going edges.
Given ground truth and predicted connectivity of lane centerlines, the mAP is calculated on $G^{l} = (V^{l}, E^{l})$ and
$\\hat{G}^{l} = (\\hat{V}^{l}, \\hat{E}^{l})$. As the given graphs are directed, e.g., the ending point of a lane
centerline is connected to the starting point of the next lane centerline, we take the mean of mAP over graphs with only
in-going or out-going edges.
To evaluate the predicted topology between lane centerlines and traffic elements, we ignore the relationship among lane centerlines.
The relationship among traffic elements is also not taken into consideration.
Thus this can be seen as a link prediction problem on a bipartite undirected graph that $G = (V^{l} \cup V^{t}, E)$ and $\hat{G} = (\hat{V}^{l} \cup \hat{V}^{t}, \hat{E})$.
To evaluate the predicted topology between lane centerlines and traffic elements, we ignore the relationship among lane
centerlines. The relationship among traffic elements is also not taken into consideration. Thus this can be seen as a
link prediction problem on a bipartite undirected graph that $G = (V^{l} \\cup V^{t}, E)$ and $\\hat{G} = (\\hat{V}^{l}
\\cup \\hat{V}^{t}, \\hat{E})$.
## Distances
To measure the similarity between ground truth and predicted instances, we adopt Frechet and IoU distances for directed curves and 2D bounding boxes respectively.
To measure the similarity between ground truth and predicted instances, we adopt Frechet and IoU distances for directed
curves and 2D bounding boxes respectively.
### Frechet Distance
Discrete Frechet distance measures the geometric similarity of two ordered lists of points.
Given a pair of curves, namely a ground truth $v = (p_1, ..., p_n)$ and a prediction $\hat{v} = (\hat{p}_1, ..., \hat{p}_k)$, a coupling $L$ is a sequence of distinct pairs between $v$ and $\hat{v}$:
Discrete Frechet distance measures the geometric similarity of two ordered lists of points. Given a pair of curves,
namely a ground truth $v = (p_1, ..., p_n)$ and a prediction $\\hat{v} = (\\hat{p}\_1, ..., \\hat{p}\_k)$, a coupling
$L$ is a sequence of distinct pairs between $v$ and $\\hat{v}$:
where $a_1, ..., a_m$ and $b_1, ..., b_m$ are nondecreasing surjection such that $1 = a_1 \leq a_i \leq a_j \leq a_m = n$ and $1 = b_1 \leq b_i \leq b_j \leq b_m = k$ for all $i < j$. Then the norm $||L||$ of a coupling $L$ is the distance of the most dissimilar pair in $L$ that:
where $a_1, ..., a_m$ and $b_1, ..., b_m$ are nondecreasing surjection such that $1 = a_1 \\leq a_i \\leq a_j \\leq a_m
= n$ and $1 = b_1 \\leq b_i \\leq b_j \\leq b_m = k$ for all $i \< j$. Then the norm $||L||$ of a coupling $L$ is the
The submitted results are required to be stored in a pickle file, which is a dict of identifier and [formatted predictions](../data/README.md#annotations) of a frame:
The submitted results are required to be stored in a pickle file, which is a dict of identifier
and [formatted predictions](../data/README.md#annotations) of a frame:
```
{
...
...
@@ -16,15 +18,19 @@ The submitted results are required to be stored in a pickle file, which is a dic
'traffic_element': ...
'topology_lclc': ...
'topology_lcte': ...
},
},
...
}
}
```
*: For validation, `from iso3166 import countries; countries.get(str)` can be used.
\*: For validation, `from iso3166 import countries; countries.get(str)` can be used.
## Steps
1. Create a team on [EvalAI](https://eval.ai/web/challenges/challenge-page/1925).
2. Click the 'Participate' tag, then choose a team for participation.
3. Choose the phase 'Test Phase (CVPR 2023 Autonomous Driving Challenge)' and upload the file formatted as mentioned above.
4. Check if the submitted file is valid, which is indicated by the 'Status' under the tag of 'My Submissions'. A valid submission would provide performance scores.
3. Choose the phase 'Test Phase (CVPR 2023 Autonomous Driving Challenge)' and upload the file formatted as mentioned
above.
4. Check if the submitted file is valid, which is indicated by the 'Status' under the tag of 'My Submissions'. A valid
raiseException(f'Shape of adjacent matrix of [results/{token}/predictions/topology_lclc] should be (#lane_centerline, #lane_centerline) but not {topology_lclc.shape}')
f'Shape of adjacent matrix of [results/{token}/predictions/topology_lclc] should be (#lane_centerline, #lane_centerline) but not {topology_lclc.shape}')
raiseException(f'Shape of adjacent matrix of [results/{token}/predictions/topology_lcte] should be (#lane_centerline, #traffic_element) but not {topology_lcte.shape}')
f'Shape of adjacent matrix of [results/{token}/predictions/topology_lcte] should be (#lane_centerline, #traffic_element) but not {topology_lcte.shape}')
