README.md 11.3 KB
Newer Older
Jonas Kaufmann's avatar
Jonas Kaufmann committed
1
2
<img src="doc/simbricks.svg" alt="SimBricks Logo" width="300" />

3
# SimBricks
Jialin Li's avatar
Jialin Li committed
4

Antoine Kaufmann's avatar
Antoine Kaufmann committed
5
[![CI pipeline status](https://gitlab.mpi-sws.org/simbricks/simbricks/badges/main/pipeline.svg)](https://gitlab.mpi-sws.org/simbricks/simbricks/-/commits/main)
6
7
8
9
10
11
12
13
[![Documentation Status](https://readthedocs.org/projects/simbricks/badge/?version=latest)](https://simbricks.readthedocs.io/en/latest/?badge=latest)
[![Docker Hub](https://img.shields.io/badge/docker-hub-brightgreen)](https://hub.docker.com/u/simbricks)
[![Chat on Slack](https://img.shields.io/badge/slack-Chat-brightgreen)](https://join.slack.com/t/simbricks/shared_invite/zt-16y96155y-xspnVcm18EUkbUHDcSVonA)
[![MIT License](https://img.shields.io/github/license/simbricks/simbricks)](https://github.com/simbricks/simbricks/blob/main/LICENSE.md)

## What is SimBricks?

SimBricks is a simulator framework aiming to enable true end-to-end simulation
Jonas Kaufmann's avatar
Jonas Kaufmann committed
14
15
of modern data center network systems, including multiple servers running a full
software stack with unmodified OS and applications, network topologies and
16
17
18
19
20
21
22
devices, as well as other off the shelf and custom hardware components. Instead
of designing a new simulator from scratch, SimBricks combines and connects
multiple existing simulators for different components into a simulated full
system. Our primary use-case for SimBricks is computer systems, networks, and
architecture research. Our [paper](https://arxiv.org/abs/2012.14219) provides a
more detailed discussion of technical details and use-cases.

23
Currently, SimBricks includes the following simulators:
24
25
26

- [QEMU](https://www.qemu.org) (fast host simulator)
- [gem5](https://www.gem5.org/) (flexible and detailed host simulator)
Jonas Kaufmann's avatar
Jonas Kaufmann committed
27
28
29
- [Simics](https://www.intel.com/content/www/us/en/developer/articles/tool/simics-simulator.html)
  (fast, closed-source host simulator supporting modern x86 ISA extensions like
  AVX)
30
31
32
33
34
35
36
37
38
39
40
- [ns-3](https://www.nsnam.org/) (flexible simulator for networks)
- [OMNeT++ INET](https://inet.omnetpp.org/) (flexible simulator for networks)
- [Intel Tofino SDK Simulator](https://www.intel.com/content/www/us/en/products/network-io/programmable-ethernet-switch/p4-suite/p4-studio.html)
  (closed-source vendor-provided simulator for Tofino P4 switches).
- [FEMU](https://github.com/ucare-uchicago/FEMU) (NVMe SSD simulator).
- [Verilator](https://www.veripool.org/verilator/) (Verilog RTL simulator)

## Quick Start

Depending on how you plan to use SimBricks, there are different ways to start
using it. The quickest way to get started just running SimBricks is with our
41
42
43
44
[pre-built Docker container images](https://hub.docker.com/u/simbricks).
However, if you plan to make changes to SimBricks, you will have to build
SimBricks from source, either through Docker, or on your local machine. The
different ways are listed below in order of increasing effort required.
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61

**Please refer to
[our documentation](https://simbricks.readthedocs.io/en/latest/) for more
details.**

### Using Pre-Built Docker Images

**This is the quickest way to get started using SimBricks.**

We provide pre-built Docker images on
[Docker Hub](https://hub.docker.com/u/simbricks). These images allow you to
start using SimBricks without building it yourself or installing any
dependencies. This command will run an interactive shell in a new ephemeral
container (deleted after the shell exits):

```Shell
docker run --rm -it simbricks/simbricks /bin/bash
Antoine Kaufmann's avatar
Antoine Kaufmann committed
62
63
```

64
65
66
If you are running on a Linux system with KVM support enabled, we recommend
passing `/dev/kvm` into the container to drastically speed up some of the
simulators:
Jialin Li's avatar
Jialin Li committed
67

68
69
70
```Shell
docker run --rm -it --device /dev/kvm simbricks/simbricks /bin/bash
```
Antoine Kaufmann's avatar
Antoine Kaufmann committed
71

72
73
74
75
76
77
78
79
Finally, some of our host simulators, e.g., gem5 and Simics, require raw
disk images. Since Docker doesn't handle large, sparse files well leading to
large Docker image sizes, we only include disk images in the qcow format. To
convert these to raw, run the following:

```Shell
make convert-images-raw
```
Jonas Kaufmann's avatar
Jonas Kaufmann committed
80

81
82
Now you are ready to run your first SimBricks simulation. Note that
`simbricks-run` is only available inside our docker containers.
Jialin Li's avatar
Jialin Li committed
83

84
85
```Shell
root@fa76605e3628:/simbricks# cd experiments/
86
root@fa76605e3628:/simbricks/experiments# simbricks-run --verbose --force pyexps/simple_ping.py
87
...
Jialin Li's avatar
Jialin Li committed
88
```
Hejing Li's avatar
readme  
Hejing Li committed
89

90
### Experimental: Interactive SimBricks Jupyter Labs
Antoine Kaufmann's avatar
Antoine Kaufmann committed
91

92
**This is still a work in progress.**
Antoine Kaufmann's avatar
Antoine Kaufmann committed
93

94
95
96
97
98
We are working on a more interactive introduction to SimBricks through Jupyter
Labs in [this repository](https://github.com/simbricks/labs). These also simply
require starting a pre-built docker container and then connecting to it from
your browser. After this you can follow the interactive steps to run SimBricks
simulation directly from your browser.
Antoine Kaufmann's avatar
Antoine Kaufmann committed
99

100
### Building Docker Images
Antoine Kaufmann's avatar
Antoine Kaufmann committed
101

102
103
If you prefer to build the Docker images locally you will need `git`, `make`,
and `docker build` installed on your system. Other dependencies should not be
104
105
required. Now you are ready to build the docker images (depending on your system
this will likely take 15-45 minutes):
Antoine Kaufmann's avatar
Antoine Kaufmann committed
106

107
108
109
110
```Shell
git clone https://github.com/simbricks/simbricks.git
cd simbricks
make docker-images
Antoine Kaufmann's avatar
Antoine Kaufmann committed
111
```
112
113
114
115

This will build a number of Docker images and tag them locally, including the
main `simbricks/simbricks` image.

116
### Building in VS Code Dev Container
117
118
119

**We recommend this approach if you plan to modify or extend SimBricks.**

120
This repository is pre-configured with a [Visual Studio Code Development
121
Container](https://code.visualstudio.com/docs/remote/containers) that includes
122
all required dependencies for building and working on SimBricks. If you have
123
124
125
126
127
128
Docker set up and the VS Code [Dev Containers
extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers)
installed, you just have to press `Ctrl+Shift+P` and execute the `Dev Containers: Reopen in
Container` command to open the repository inside the container. This also means
that all VS Code terminals will automatically run any commands inside the
container.
129
130
131
132
133
134
135
136
137

To compile the core SimBricks components simply run `make` (with `-jN` to
use multiple cores). Note that by default, we do not build the Verilator
simulation as these take longer to compile (one to three minutes typically)
and also skip building the RDMA proxy as it depends on the specific RDMA NIC
libraries. These can be enabled by setting `ENABLE_VERILATOR=y ENABLE_RDMA=y`
on the `make` command-line or by creating `mk/local.mk` and inserting those
settings there.

138
The previous step only builds the simulators directly contained in the SimBricks
139
repository. You likely also want to build at least some of the external
140
141
142
143
144
145
146
147
148
149
150
151
152
153
simulators, such as *gem5*, *QEMU*, or *ns-3*. First, make sure their
corresponding submodules are initialized via `git submodule update --init`. You
can either build all external simulators by running `make -jN external` (this
could take multiple hours depending on your machine), or build them individually
by running e.g. `make -jN sims/external/qemu/ready` (replace `qemu` with `gem5`,
`ns-3`, or `femu` as desired).

Next, to actually run simulations, you also need to build the disk images with
`make -jN build-images` (note this requires QEMU to be built first). This builds
all our disk images, while `make -jN build-images-min` only builds the base disk
image (but not the NOPaxos or Memcached images used for some experiments). This
step will again take 10 - 45 minutes depending on your machine and whether KVM
acceleration is available but only needs to be run once (unless you want to
modify the images).
154
155
156
157
158

Now you are ready to run simulations as with the pre-built docker images.

### Building From Source

159
160
161
162
163
164
Finally, it is of course possible to install the required dependencies directly
on your machine and then build and run SimBricks locally. Note that you will
need to install both the build dependencies for SimBricks but also for the
external simulators you need. We suggest you refer to the
[`docker/Dockerfile.buildenv`](docker/Dockerfile.buildenv) for the authoritative
list of required dependencies.
165

166
167
168
## Questions? Suggestions? Bugs?

If you are using SimBricks or are trying to determine if SimBricks is suitable
169
170
171
172
173
174
175
for what you are trying to do, we would love to hear from you. First off, please
feel free to report bugs or suggestions directly through [GitHub
issues](https://github.com/simbricks/simbricks/issues). If you have questions or
thoughts, please post them on our [GitHub discussion
board](https://github.com/simbricks/simbricks/discussions). Finally, we are also
available on
[Slack](https://join.slack.com/t/simbricks/shared_invite/zt-16y96155y-xspnVcm18EUkbUHDcSVonA)
176
177
for more interactive discussions or to answer quick questions.

178
179
## Repository Structure

180
- `doc/`: Documentation (Sphinx), automatically deployed on
181
182
183
184
185
  [Read The Docs](https://simbricks.readthedocs.io/en/latest/?badge=latest).
- `lib/simbricks/`: Libraries implementing SimBricks interfaces
  - `lib/simbricks/base`: Base protocol implementation responsible for
    connection setup, message transfer, and time synchronization between
    SimBricks component simulators.
186
187
  - `lib/simbricks/network`: Network protocol implementation carrying Ethernet
    packets between network components. Layers over the base protocol.
188
  - `lib/simbricks/pcie`: PCIe protocol implementation, roughly modelling PCIe
189
190
    at the transaction level, interconnecting hosts with PCIe device simulators.
    Layers over base protocol.
191
192
193
194
195
  - `lib/simbricks/nicbm`: Helper C++ library for implementing behavioral
    (high-level) NIC simulation models, offers similar abstractions as device
    models in other simulators such as gem-5.
  - `lib/simbricks/nicif`: *(deprecated)* Thin C library for NIC simulators
    establishing a network and a PCIe connection.
196
- `dist/`: Proxies for distributed SimBricks simulations running on multiple
197
  physical hosts.
198
199
  - `dist/sockets/`: Proxy transporting SimBricks messages over regular TCP
    sockets.
200
201
202
  - `dist/rdma/`: RDMA SimBricks proxy (not compiled by default).
- `sims/`: Component Simulators integrated into SimBricks.
  - `sims/external/`: Submodule pointers to repositories for existing external
Jonas Kaufmann's avatar
Jonas Kaufmann committed
203
    simulators (gem5, QEMU, Simics, ns-3, FEMU).
204
205
  - `sims/nic/`: NIC simulators
    - `sims/nic/i40e_bm`: Behavioral NIC model for Intel X710 40G NIC.
206
    - `sims/nic/corundum`: RTL simulation with Verilator of the
207
208
209
210
211
      [Corundum FPGA NIC](https://corundum.io/).
    - `sims/nic/corundum_bm`: Simple behavioral Corundum NIC model.
    - `sims/nic/e1000_gem5`: E1000 NIC model extracted from gem5.
  - `sims/net/`: Network simulators
    - `sims/net/net_switch`: Simple behavioral Ethernet switch model.
212
    - `sims/net/wire`: Simple Ethernet "wire" connecting two NICs back-to-back.
213
214
215
    - `sims/net/pktgen`: Packet generator.
    - `sims/net/tap`: Linux TAP device adapter.
    - `sims/net/tofino/`: Adapter for Intel Tofino Simulator.
216
    - `sims/net/menshen`: RTL simulation with Verilator for the
217
      [Menshen RMT Pipeline](https://isolation.quest/).
218
- `experiments/`: Python orchestration framework for running simulations.
219
  - `experiments/simbricks/orchestration/`: Orchestration framework implementation.
220
221
222
  - `experiments/run.py`: Main script for running simulation experiments.
  - `experiments/pyexps/`: Example simulation experiments.
- `images/`: Infrastructure to build disk images for host simulators.
223
224
225
226
  - `images/kernel/`: Slimmed down Linux kernel to reduce simulation time.
  - `images/mqnic/`: Linux driver for Corundum NIC.
  - `images/scripts/`: Scripts for installing packages in disk images.
- `docker/`: Scripts for building SimBricks Docker images.