Unverified Commit 8e3c0d26 authored by Tushar Sharma's avatar Tushar Sharma Committed by GitHub
Browse files

ci: add workflow to build frontend image (#4285)


Signed-off-by: default avatarAnant Sharma <anants@nvidia.com>
Signed-off-by: default avatarHannah Zhang <hannahz@nvidia.com>
Signed-off-by: default avatarDan Gil <dagil@nvidia.com>
Signed-off-by: default avatarzhongdaor <zhongdaor@nvidia.com>
Signed-off-by: default avatarKeiven Chang <keivenchang@users.noreply.github.com>
Signed-off-by: default avatarJason Zhou <jasonzho@jasonzho-mlt.client.nvidia.com>
Signed-off-by: default avatarJason Zhou <jasonzho@nvidia.com>
Signed-off-by: default avatarDillon Cullinan <dcullinan@nvidia.com>
Signed-off-by: default avatarAryan Bagade <aryan@aryanbagade.com>
Signed-off-by: default avatarmohammedabdulwahhab <furkhan324@berkeley.edu>
Signed-off-by: default avatarTushar Sharma <tusharma@nvidia.com>
Signed-off-by: default avatarAnna Tchernych <atchernych@nvidia.com>
Signed-off-by: default avatarHarrison Saturley-Hall <hsaturleyhal@nvidia.com>
Signed-off-by: default avatarHarrison King Saturley-Hall <hsaturleyhal@nvidia.com>
Signed-off-by: default avatarKyle McGill <101670481+nv-kmcgill53@users.noreply.github.com>
Signed-off-by: default avatarElnifio <elnifio0519@gmail.com>
Signed-off-by: default avataralec-flowers <aflowers@nvidia.com>
Signed-off-by: default avatarHarrison Saturley-Hall <harrison.saturley.hall@gmail.com>
Signed-off-by: default avatarJulien Mancuso <jmancuso@nvidia.com>
Signed-off-by: default avatarPeaBrane <yanrpei@gmail.com>
Signed-off-by: default avataratchernych <atchernych@nvidia.com>
Signed-off-by: default avatartzulingk@nvidia.com <tzulingk@nvidia.com>
Signed-off-by: default avatarOlga Andreeva <oandreeva@nvidia.com>
Signed-off-by: default avatarOlga Andreeva <124622579+oandreeva-nv@users.noreply.github.com>
Signed-off-by: default avatarBen Hamm <ben.hamm@gmail.com>
Signed-off-by: default avatarTanmay Verma <tanmay2592@gmail.com>
Signed-off-by: default avatarKaren Chung <karenc@nvidia.com>
Signed-off-by: default avatarkaren-sy <karenc@nvidia.com>
Signed-off-by: default avatarkrishung5 <krish@nvidia.com>
Signed-off-by: default avatarShuaiyi Zhang <zhangsy28@lenovo.com>
Signed-off-by: default avatarIndrajit Bhosale <iamindrajitb@gmail.com>
Signed-off-by: default avatarVladislav Nosivskoy <vladnosiv@gmail.com>
Signed-off-by: default avatarpvijayakrish <pvijayakrish@nvidia.com>
Signed-off-by: default avatarDmitry Tokarev <dtokarev@nvidia.com>
Co-authored-by: default avatarAnant Sharma <anants@nvidia.com>
Co-authored-by: default avatardagil-nvidia <dagil@nvidia.com>
Co-authored-by: default avatarhhzhang16 <54051230+hhzhang16@users.noreply.github.com>
Co-authored-by: default avatarRyan McCormick <rmccormick@nvidia.com>
Co-authored-by: default avatarzhongdaor-nv <zhongdaor@nvidia.com>
Co-authored-by: default avatarKeiven C <213854356+keivenchang@users.noreply.github.com>
Co-authored-by: default avatarKeiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: default avatarishandhanani <82981111+ishandhanani@users.noreply.github.com>
Co-authored-by: default avatarJason Zhou <jasonzho@nvidia.com>
Co-authored-by: default avatarJason Zhou <jasonzho@jasonzho-mlt.client.nvidia.com>
Co-authored-by: default avatarDillon Cullinan <dcullinan92@gmail.com>
Co-authored-by: default avatarAryan Bagade <73382554+AryanBagade@users.noreply.github.com>
Co-authored-by: default avatarmohammedabdulwahhab <furkhan324@berkeley.edu>
Co-authored-by: default avatartmontfort <tmontfort@nvidia.com>
Co-authored-by: default avatarThomas Montfort <61255722+tmonty12@users.noreply.github.com>
Co-authored-by: default avataratchernych <atchernych@nvidia.com>
Co-authored-by: default avatarHarrison Saturley-Hall <hsaturleyhal@nvidia.com>
Co-authored-by: default avatarKyle McGill <101670481+nv-kmcgill53@users.noreply.github.com>
Co-authored-by: default avatarDmitry Tokarev <dtokarev@nvidia.com>
Co-authored-by: default avatarYunzhou Liu <46603306+Elnifio@users.noreply.github.com>
Co-authored-by: default avatarAlec <35311602+alec-flowers@users.noreply.github.com>
Co-authored-by: default avatarJulien Mancuso <161955438+julienmancuso@users.noreply.github.com>
Co-authored-by: default avatarYan Ru Pei <yanrpei@gmail.com>
Co-authored-by: default avatarBiswa Panda <biswa.panda@gmail.com>
Co-authored-by: default avatarTzu-Ling Kan <tzulingk@nvidia.com>
Co-authored-by: default avatarKris Hung <krish@nvidia.com>
Co-authored-by: default avatarOlga Andreeva <124622579+oandreeva-nv@users.noreply.github.com>
Co-authored-by: default avatarCopilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: default avatarHongkuan Zhou <tedzhouhk@gmail.com>
Co-authored-by: default avatarBen Hamm <ben.hamm@gmail.com>
Co-authored-by: default avatartanmayv25 <tanmay2592@gmail.com>
Co-authored-by: default avatarTanmay Verma <tanmayv@nvidia.com>
Co-authored-by: default avatarKaren Chung <karenc@nvidia.com>
Co-authored-by: default avatarShuaiyi Zhang <576893949@qq.com>
Co-authored-by: default avatarShuaiyi Zhang <zhangsy28@lenovo.com>
Co-authored-by: default avatarAnthony Casagrande <acasagrande@nvidia.com>
Co-authored-by: default avatarIndrajit Bhosale <iamindrajitb@gmail.com>
Co-authored-by: default avatarVladislav Nosivskoy <47858711+vladnosiv@users.noreply.github.com>
Co-authored-by: default avatarPavithra Vijayakrishnan <160681768+pvijayakrish@users.noreply.github.com>
Co-authored-by: default avatarKyle McGill <kmcgill@nvidia.com>
parent 1efc41e0
# SPDX-FileCopyrightText: Copyright (c) 2024-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
name: Build Frontend Image
on:
push:
branches:
- main
- release/*.*.*
concurrency:
group: ${{ github.workflow }}-${{ github.ref_name || github.run_id }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
jobs:
build-frontend-image:
name: Build Frontend Image (${{ matrix.platform.arch }})
strategy:
fail-fast: false
matrix:
platform:
- { arch: amd64, runner: gpu-l40-amd64 }
- { arch: arm64, runner: cpu-arm-r8g-4xlarge }
runs-on: ${{ matrix.platform.runner }}
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
with:
driver: docker
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: '1.24'
- name: Install dependencies
shell: bash
run: |
set -euo pipefail
# Install system dependencies from apt
sudo apt-get update && sudo apt-get install -y git build-essential protobuf-compiler
# Install Rust (cargo + rustc)
curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain stable
# Make cargo available to later steps
echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
- name: Install cbindgen
shell: bash
run: |
set -euo pipefail
cargo install cbindgen
- name: Docker Login
uses: ./.github/actions/docker-login
with:
aws_default_region: ${{ secrets.AWS_DEFAULT_REGION }}
aws_account_id: ${{ secrets.AWS_ACCOUNT_ID }}
azure_acr_hostname: ${{ secrets.AZURE_ACR_HOSTNAME }}
azure_acr_user: ${{ secrets.AZURE_ACR_USER }}
azure_acr_password: ${{ secrets.AZURE_ACR_PASSWORD }}
- name: Build Frontend Container
id: build-image
uses: ./.github/actions/docker-build
env:
PLATFORMS: linux/${{ matrix.platform.arch }}
TARGETARCH: ${{ matrix.platform.arch }}
with:
framework: none
target: frontend
platform: ${{ env.PLATFORMS }}
ci_token: ${{ secrets.CI_TOKEN }}
aws_default_region: ${{ secrets.AWS_DEFAULT_REGION }}
sccache_s3_bucket: ${{ secrets.SCCACHE_S3_BUCKET }}
aws_account_id: ${{ secrets.AWS_ACCOUNT_ID }}
aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
- name: Docker Tag and Push Frontend Image
uses: ./.github/actions/docker-tag-push
with:
local_image: ${{ steps.build-image.outputs.image_tag }}
push_tags: ai-dynamo/dynamo:${{ github.sha }}-frontend-${{ matrix.platform.arch }}
aws_push: 'false'
azure_push: 'true'
aws_account_id: ${{ secrets.AWS_ACCOUNT_ID }}
aws_default_region: ${{ secrets.AWS_DEFAULT_REGION }}
azure_acr_hostname: ${{ secrets.AZURE_ACR_HOSTNAME }}
azure_acr_user: ${{ secrets.AZURE_ACR_USER }}
azure_acr_password: ${{ secrets.AZURE_ACR_PASSWORD }}
\ No newline at end of file
......@@ -23,6 +23,7 @@
ARG BASE_IMAGE
ARG BASE_IMAGE_TAG
ARG EPP_IMAGE="us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/epp:v0.5.1"
ARG PYTHON_VERSION
ARG ENABLE_KVBM
......@@ -545,3 +546,69 @@ USER dynamo
ENTRYPOINT ["/opt/nvidia/nvidia_entrypoint.sh"]
CMD []
##############################################
########## Frontend entrypoint image #########
##############################################
FROM ${EPP_IMAGE} AS epp
FROM nvcr.io/nvidia/base/ubuntu:noble-20250619 AS frontend
ARG PYTHON_VERSION
RUN apt-get update -y \
&& apt-get install -y --no-install-recommends \
# required for EPP
ca-certificates \
libstdc++6 \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# Create dynamo user with group 0 for OpenShift compatibility
RUN userdel -r ubuntu > /dev/null 2>&1 || true \
&& useradd -m -s /bin/bash -g 0 dynamo \
&& [ `id -u dynamo` -eq 1000 ] \
&& mkdir -p /home/dynamo/.cache /opt/dynamo /workspace \
&& chown -R dynamo: /opt/dynamo /home/dynamo/.cache /workspace \
&& chmod -R g+w /opt/dynamo /home/dynamo/.cache /workspace
# Set HOME so ModelExpress can find the cache directory
ENV HOME=/home/dynamo
# Switch to dynamo user
USER dynamo
ENV DYNAMO_HOME=/opt/dynamo
WORKDIR /
COPY --chown=dynamo: --from=epp /epp /epp
COPY --chown=dynamo: container/launch_message/frontend.txt /opt/dynamo/.launch_screen
# Copy tests, benchmarks, deploy and components with correct ownership
COPY --chown=dynamo: tests /workspace/tests
COPY --chown=dynamo: examples /workspace/examples
COPY --chown=dynamo: benchmarks /workspace/benchmarks
COPY --chown=dynamo: deploy /workspace/deploy
COPY --chown=dynamo: components/ /workspace/components/
COPY --chown=dynamo: recipes/ /workspace/recipes/
# Copy attribution files with correct ownership
COPY --chown=dynamo: ATTRIBUTION* LICENSE /workspace/
ENV VIRTUAL_ENV=/opt/dynamo/venv
ENV PATH="/opt/dynamo/venv/bin:$PATH"
# Copy virtual environment directly from dynamo_base (dev image)
# This includes all installed packages: dynamo, nixl, requirements.txt, requirements.test.txt
# Copy uv to system /bin
COPY --chown=dynamo: --from=dev /bin/uv /bin/uvx /bin/
RUN uv python install $PYTHON_VERSION
COPY --chown=dynamo: --from=dev /opt/dynamo/venv/ /opt/dynamo/venv/
# Setup environment for all users
USER root
RUN chmod 755 /opt/dynamo/.launch_screen && \
echo 'source /opt/dynamo/venv/bin/activate' >> /etc/bash.bashrc && \
echo 'cat /opt/dynamo/.launch_screen' >> /etc/bash.bashrc
USER dynamo
ENTRYPOINT ["/epp"]
CMD ["/bin/bash"]
\ No newline at end of file
# syntax=docker/dockerfile:1.10.0
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
ARG DYNAMO_BASE_IMAGE="dynamo:latest-none"
ARG EPP_IMAGE="us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/epp:v0.5.1-dirty"
ARG PYTHON_VERSION=3.12
FROM ${DYNAMO_BASE_IMAGE} AS dynamo_base
FROM ${EPP_IMAGE} AS epp
FROM nvcr.io/nvidia/base/ubuntu:noble-20250619 AS frontend
ARG PYTHON_VERSION
RUN apt-get update -y \
&& apt-get install -y --no-install-recommends \
# required for EPP
ca-certificates \
libstdc++6 \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# Create dynamo user with group 0 for OpenShift compatibility
RUN userdel -r ubuntu > /dev/null 2>&1 || true \
&& useradd -m -s /bin/bash -g 0 dynamo \
&& [ `id -u dynamo` -eq 1000 ] \
&& mkdir -p /home/dynamo/.cache /opt/dynamo /workspace \
&& chown -R dynamo: /opt/dynamo /home/dynamo/.cache /workspace \
&& chmod -R g+w /opt/dynamo /home/dynamo/.cache /workspace
# Set HOME so ModelExpress can find the cache directory
ENV HOME=/home/dynamo
# Switch to dynamo user
USER dynamo
ENV DYNAMO_HOME=/opt/dynamo
WORKDIR /
COPY --chown=dynamo: --from=epp /epp /epp
COPY --chown=dynamo: container/launch_message/frontend.txt /opt/dynamo/.launch_screen
# Copy tests, benchmarks, deploy and components with correct ownership
COPY --chown=dynamo: tests /workspace/tests
COPY --chown=dynamo: examples /workspace/examples
COPY --chown=dynamo: benchmarks /workspace/benchmarks
COPY --chown=dynamo: deploy /workspace/deploy
COPY --chown=dynamo: components/ /workspace/components/
COPY --chown=dynamo: recipes/ /workspace/recipes/
# Copy attribution files with correct ownership
COPY --chown=dynamo: ATTRIBUTION* LICENSE /workspace/
ENV VIRTUAL_ENV=/opt/dynamo/venv
ENV PATH="/opt/dynamo/venv/bin:$PATH"
# Copy virtual environment directly from dynamo_base (dev image)
# This includes all installed packages: dynamo, nixl, requirements.txt, requirements.test.txt
# Copy uv to system /bin
COPY --from=dynamo_base /bin/uv /bin/uvx /bin/
RUN uv python install $PYTHON_VERSION
COPY --chown=dynamo: --from=dynamo_base /opt/dynamo/venv/ /opt/dynamo/venv/
# Setup environment for all users
USER root
RUN chmod 755 /opt/dynamo/.launch_screen && \
echo 'source /opt/dynamo/venv/bin/activate' >> /etc/bash.bashrc && \
echo 'cat /opt/dynamo/.launch_screen' >> /etc/bash.bashrc
USER dynamo
ENTRYPOINT ["/epp"]
CMD ["/bin/bash"]
......@@ -15,7 +15,6 @@ The NVIDIA Dynamo project uses containerized development and deployment to maint
- `Dockerfile.trtllm` - For TensorRT-LLM inference backend
- `Dockerfile.sglang` - For SGLang inference backend
- `Dockerfile` - Base/standalone configuration
- `Dockerfile.frontend` - For Kubernetes Gateway API Inference Extension integration with EPP
- `Dockerfile.epp` - For building the Endpoint Picker (EPP) image
### Stage Summary for Frameworks
......@@ -165,39 +164,22 @@ The `build.sh` script is responsible for building Docker images for different AI
### Building the Frontend Image
The frontend image is a specialized container that includes the Dynamo components (NATS, etcd, dynamo, NIXL, etc) along with the Endpoint Picker (EPP) for Kubernetes Gateway API Inference Extension integration. This image is primarily used for inference gateway deployments.
The frontend image is a specialized container that includes the Dynamo components (Dynamo, NIXL, etc) along with the Endpoint Picker (EPP) for Kubernetes Gateway API Inference Extension integration. This image is primarily used for inference gateway deployments.
**Step 1: Build the Custom Dynamo EPP Image**
Follow the instructions in [`deploy/inference-gateway/README.md`](../deploy/inference-gateway/README.md) under "Build the custom EPP image" section. This process:
- Clones the Gateway API Inference Extension repository
- Applies Dynamo-specific patches for custom routing
- Builds the Dynamo router as a static library
- Creates a custom EPP image with integrated Dynamo routing capabilities
**Step 2: Build the Dynamo Base Image**
The base image contains the core Dynamo runtime components, NATS server, etcd, and Python dependencies:
```bash
# Build the base dev image (framework=none for frontend-only deployment)
# Note: --framework none defaults ENABLE_MEDIA_NIXL=false
./build.sh --framework none --target dev
# Build the frontend image (automatically builds EPP image as a dependency)
./build.sh --framework none --target frontend
```
**Step 3: Build the Frontend Image**
The build process automatically:
1. Clones the Gateway API Inference Extension (GAIE) repository
2. Builds the custom EPP image with Dynamo routing capabilities
3. Builds the frontend image with the EPP binary and Dynamo runtime components
Now build the frontend image that combines the Dynamo base with the EPP:
For more details, see [`deploy/inference-gateway/README.md`](../deploy/inference-gateway/README.md).
**Note:** `--framework none` defaults `ENABLE_MEDIA_NIXL=false`.
```bash
# 2. Build the frontend image using the pre-built EPP
docker buildx build --load --platform linux/amd64 \
--build-arg DYNAMO_BASE_IMAGE=dynamo:latest-none-dev \
--build-arg EPP_IMAGE={EPP_IMAGE_TAG} \
--build-arg PYTHON_VERSION=3.12 \
-f container/Dockerfile.frontend \
-t dynamo:latest-none-frontend \
.
```
#### Frontend Image Contents
The frontend image includes:
......@@ -206,8 +188,6 @@ The frontend image includes:
- **NIXL**: NVIDIA InfiniBand Library for high-performance network communication
- **Benchmarking Tools**: Performance testing utilities (aiperf, aiconfigurator, etc)
- **Python Environment**: Virtual environment with all required dependencies
- **NATS Server**: Message broker for Dynamo's distributed communication
- **etcd**: Distributed key-value store for configuration and coordination
#### Deployment
......
......@@ -121,6 +121,10 @@ SGLANG_BASE_IMAGE_TAG="25.06-cuda12.9-devel-ubuntu24.04"
SGLANG_CUDA_VERSION="12.9.1"
SGLANG_PYTHON_VERSION="3.10"
# GAIE (Gateway API Inference Extension) configuration for frontend (required for EPP binary for frontend image)
GAIE_REPO_URL="https://github.com/kubernetes-sigs/gateway-api-inference-extension.git"
GAIE_VERSION="v0.5.1"
PYTHON_VERSION="3.12"
NIXL_REF=0.8.0
......@@ -946,6 +950,43 @@ fi
show_image_options
# Handle FRONTEND target: build EPP image first
if [[ ${TARGET^^} == "FRONTEND" ]]; then
echo "Building FRONTEND image - requires EPP image"
# Build base dynamo image first (framework=NONE, target=dev)
echo ""
echo "Building EPP image for Frontend..."
# Set up paths for GAIE
GAIE_CLONE_DIR="${BUILD_CONTEXT}/.build/external/gateway-api-inference-extension"
# Clone GAIE repo
echo ""
echo "Cloning GAIE repository at ${GAIE_VERSION}..."
$RUN_PREFIX rm -rf "${GAIE_CLONE_DIR}"
$RUN_PREFIX mkdir -p "$(dirname "${GAIE_CLONE_DIR}")"
$RUN_PREFIX git clone ${GAIE_REPO_URL} "${GAIE_CLONE_DIR}"
$RUN_PREFIX cd "${GAIE_CLONE_DIR}"
$RUN_PREFIX git checkout ${GAIE_VERSION}
$RUN_PREFIX cd "${BUILD_CONTEXT}"
# Build EPP image
echo ""
echo "Building EPP image..."
export GAIE_DIR="${GAIE_CLONE_DIR}"
export DYNAMO_DIR="${BUILD_CONTEXT}"
$RUN_PREFIX bash ${DYNAMO_DIR}/deploy/inference-gateway/build-epp-dynamo.sh
# Set EPP image tag (matches what build-epp-dynamo.sh produces)
EPP_IMAGE_TAG="us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/epp:${GAIE_VERSION}-dirty"
echo "Successfully built EPP image: ${EPP_IMAGE_TAG}"
# Add build args for frontend image
BUILD_ARGS+=" --build-arg EPP_IMAGE=${EPP_IMAGE_TAG}"
fi
# Always build the main image first
# Create build log directory for BuildKit reports
BUILD_LOG_DIR="${BUILD_CONTEXT}/build-logs"
......
......@@ -150,43 +150,20 @@ You can either use the special FrontEnd image for the EPP_IMAGE in the Helm depl
##### 1. Build the custom EPP image #####
If you choose to build your own image use the steps below.
##### 1.1 Clone the official GAIE repo in a separate folder #####
If you choose to build your own image, use the `container/build.sh` script with the `--target frontend` option:
```bash
git clone https://github.com/kubernetes-sigs/gateway-api-inference-extension.git
cd gateway-api-inference-extension
git checkout v0.5.1
./container/build.sh --framework none --target frontend
```
##### 1.2 Build the Dynamo Custom EPP #####
This command automatically:
- Clones the Gateway API Inference Extension (GAIE) repository at the correct version
- Builds the Dynamo Router static library
- Applies the necessary patches to the EPP codebase
- Builds the custom EPP image with Dynamo KV routing support
- Builds the frontend image with the EPP binary and Dynamo runtime components
###### 1.2.1 Clone the official EPP repo ######
```bash
# Clone the official GAIE repo in a separate folder
cd path/to/gateway-api-inference-extension
git clone git@github.com:kubernetes-sigs/gateway-api-inference-extension.git
git checkout v0.5.1
```
###### 1.2.2 Run the script to build the EPP image ######
The script will apply a custom patch to the code with your GAIE repo and build the image for you to use.
```bash
# Use your custom paths
export DYNAMO_DIR=/path/to/dynamo
export GAIE_DIR=/path/to/gateway-api-inference-extension
# Run the script
cd deploy/inference-gateway
./build-epp-dynamo.sh
```
Under the hood the script applies the Dynamo Patch to the EPP code base; creates a Dynamo Router static library and builds a custom EPP image with it.
Re-tag the freshly built image and push it to your registry.
Re-tag the freshly built image and push it to your registry:
```bash
docker images
......@@ -194,7 +171,6 @@ docker tag <your-new-id> <your-image-tag>
docker push <your-image-tag>
```
**Note**
You can also use the standard EPP image`us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/epp:v0.4.0`. For the basic black box integration run:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment