README.md 12.5 KB
Newer Older
Jonas Kaufmann's avatar
Jonas Kaufmann committed
1
2
<img src="doc/simbricks.svg" alt="SimBricks Logo" width="300" />

3
# SimBricks
Jialin Li's avatar
Jialin Li committed
4

Antoine Kaufmann's avatar
Antoine Kaufmann committed
5
[![CI pipeline status](https://gitlab.mpi-sws.org/simbricks/simbricks/badges/main/pipeline.svg)](https://gitlab.mpi-sws.org/simbricks/simbricks/-/commits/main)
6
7
8
9
10
11
12
13
[![Documentation Status](https://readthedocs.org/projects/simbricks/badge/?version=latest)](https://simbricks.readthedocs.io/en/latest/?badge=latest)
[![Docker Hub](https://img.shields.io/badge/docker-hub-brightgreen)](https://hub.docker.com/u/simbricks)
[![Chat on Slack](https://img.shields.io/badge/slack-Chat-brightgreen)](https://join.slack.com/t/simbricks/shared_invite/zt-16y96155y-xspnVcm18EUkbUHDcSVonA)
[![MIT License](https://img.shields.io/github/license/simbricks/simbricks)](https://github.com/simbricks/simbricks/blob/main/LICENSE.md)

## What is SimBricks?

SimBricks is a simulator framework aiming to enable true end-to-end simulation
Jonas Kaufmann's avatar
Jonas Kaufmann committed
14
15
of modern data center network systems, including multiple servers running a full
software stack with unmodified OS and applications, network topologies and
16
17
18
19
20
21
22
devices, as well as other off the shelf and custom hardware components. Instead
of designing a new simulator from scratch, SimBricks combines and connects
multiple existing simulators for different components into a simulated full
system. Our primary use-case for SimBricks is computer systems, networks, and
architecture research. Our [paper](https://arxiv.org/abs/2012.14219) provides a
more detailed discussion of technical details and use-cases.

23
Currently, SimBricks includes the following simulators:
24
25
26

- [QEMU](https://www.qemu.org) (fast host simulator)
- [gem5](https://www.gem5.org/) (flexible and detailed host simulator)
Jonas Kaufmann's avatar
Jonas Kaufmann committed
27
28
29
- [Simics](https://www.intel.com/content/www/us/en/developer/articles/tool/simics-simulator.html)
  (fast, closed-source host simulator supporting modern x86 ISA extensions like
  AVX)
30
31
32
33
34
35
36
37
38
39
40
- [ns-3](https://www.nsnam.org/) (flexible simulator for networks)
- [OMNeT++ INET](https://inet.omnetpp.org/) (flexible simulator for networks)
- [Intel Tofino SDK Simulator](https://www.intel.com/content/www/us/en/products/network-io/programmable-ethernet-switch/p4-suite/p4-studio.html)
  (closed-source vendor-provided simulator for Tofino P4 switches).
- [FEMU](https://github.com/ucare-uchicago/FEMU) (NVMe SSD simulator).
- [Verilator](https://www.veripool.org/verilator/) (Verilog RTL simulator)

## Quick Start

Depending on how you plan to use SimBricks, there are different ways to start
using it. The quickest way to get started just running SimBricks is with our
41
42
43
44
[pre-built Docker container images](https://hub.docker.com/u/simbricks).
However, if you plan to make changes to SimBricks, you will have to build
SimBricks from source, either through Docker, or on your local machine. The
different ways are listed below in order of increasing effort required.
45
46
47
48
49

**Please refer to
[our documentation](https://simbricks.readthedocs.io/en/latest/) for more
details.**

50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
### SimBricks Usage Examples

**This is the quickest way to get started using SimBricks.**

We provide a [repository](https://github.com/simbricks/simbricks-examples) 
that contains a series of examples including an easy-to-use and more interactive 
introduction to SimBricks through a Jupyter Notebook showing first steps on how to orchestrate 
and run simulations using SimBricks. The repository is set up for usage 
with [GitHub codespaces](https://github.com/codespaces/new/simbricks/simbricks-examples/?quickstart=1) 
which in conjunction with the Jupyter Notebook provide a very easy to use step-by-step introduction 
to SimBricks.

For more advanced use-cases the repository provides an example for users 
on how one can provide custom disk images to simulated SimBricks hosts for non-trivial 
use-cases, and a project on hardware acceleration composed of several milestones 
guiding users through a complete SimBricks based development flow.  

The repository is set up with a dev container configuration that makes running the 
examples a breeze. For quick tests you can just open the repo in CodeSpaces. The 
easiest way to use this locally is with vscode and its Dev Containers Extension. You can 
find more details on how to use the provided experiments in the repositories 
[README](https://github.com/simbricks/simbricks-examples/blob/main/README.md).

73
74
75
76
77
78
79
80
81
### Using Pre-Built Docker Images

We provide pre-built Docker images on
[Docker Hub](https://hub.docker.com/u/simbricks). These images allow you to
start using SimBricks without building it yourself or installing any
dependencies. This command will run an interactive shell in a new ephemeral
container (deleted after the shell exits):

```Shell
82
docker run --rm -it --device /dev/kvm --privileged simbricks/simbricks /bin/bash
Antoine Kaufmann's avatar
Antoine Kaufmann committed
83
84
```

85
86
If you are running on a Linux system with KVM support enabled, we recommend
passing `/dev/kvm` into the container to drastically speed up some of the
87
88
89
90
91
92
simulators. It is even required for some of them, e.g. gem5.

Further, if you plan to use gem5, the container needs to be started with
`--privileged` since it requires access to the `perf_event_open` syscall. In
addition, `/proc/sys/kernel/perf_event_paranoid` has to to be set to 1 or lower
on your host system. You can do so with
Jialin Li's avatar
Jialin Li committed
93

94
```Shell
95
sudo sysctl -w kernel.perf_event_paranoid=1
96
```
Antoine Kaufmann's avatar
Antoine Kaufmann committed
97

98
Finally, some of our host simulators, e.g. gem5 and Simics, require raw
99
100
disk images. Since Docker doesn't handle large, sparse files well leading to
large Docker image sizes, we only include disk images in the qcow format. To
101
convert these to raw, run the following inside the container:
102
103
104
105

```Shell
make convert-images-raw
```
Jonas Kaufmann's avatar
Jonas Kaufmann committed
106

107
108
Now you are ready to run your first SimBricks simulation. Note that
`simbricks-run` is only available inside our docker containers.
Jialin Li's avatar
Jialin Li committed
109

110
111
```Shell
root@fa76605e3628:/simbricks# cd experiments/
112
root@fa76605e3628:/simbricks/experiments# simbricks-run --verbose --force pyexps/simple_ping.py
113
...
Jialin Li's avatar
Jialin Li committed
114
```
Hejing Li's avatar
readme  
Hejing Li committed
115

116
### Building Docker Images
Antoine Kaufmann's avatar
Antoine Kaufmann committed
117

118
119
If you prefer to build the Docker images locally you will need `git`, `make`,
and `docker build` installed on your system. Other dependencies should not be
120
121
required. Now you are ready to build the docker images (depending on your system
this will likely take 15-45 minutes):
Antoine Kaufmann's avatar
Antoine Kaufmann committed
122

123
124
125
126
```Shell
git clone https://github.com/simbricks/simbricks.git
cd simbricks
make docker-images
Antoine Kaufmann's avatar
Antoine Kaufmann committed
127
```
128
129
130
131

This will build a number of Docker images and tag them locally, including the
main `simbricks/simbricks` image.

132
### Building in VS Code Dev Container
133
134
135

**We recommend this approach if you plan to modify or extend SimBricks.**

136
This repository is pre-configured with a [Visual Studio Code Development
137
Container](https://code.visualstudio.com/docs/remote/containers) that includes
138
all required dependencies for building and working on SimBricks. If you have
139
140
141
142
143
144
Docker set up and the VS Code [Dev Containers
extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers)
installed, you just have to press `Ctrl+Shift+P` and execute the `Dev Containers: Reopen in
Container` command to open the repository inside the container. This also means
that all VS Code terminals will automatically run any commands inside the
container.
145
146
147
148
149
150
151
152
153

To compile the core SimBricks components simply run `make` (with `-jN` to
use multiple cores). Note that by default, we do not build the Verilator
simulation as these take longer to compile (one to three minutes typically)
and also skip building the RDMA proxy as it depends on the specific RDMA NIC
libraries. These can be enabled by setting `ENABLE_VERILATOR=y ENABLE_RDMA=y`
on the `make` command-line or by creating `mk/local.mk` and inserting those
settings there.

154
The previous step only builds the simulators directly contained in the SimBricks
155
repository. You likely also want to build at least some of the external
156
157
158
159
160
161
162
163
164
165
166
167
168
169
simulators, such as *gem5*, *QEMU*, or *ns-3*. First, make sure their
corresponding submodules are initialized via `git submodule update --init`. You
can either build all external simulators by running `make -jN external` (this
could take multiple hours depending on your machine), or build them individually
by running e.g. `make -jN sims/external/qemu/ready` (replace `qemu` with `gem5`,
`ns-3`, or `femu` as desired).

Next, to actually run simulations, you also need to build the disk images with
`make -jN build-images` (note this requires QEMU to be built first). This builds
all our disk images, while `make -jN build-images-min` only builds the base disk
image (but not the NOPaxos or Memcached images used for some experiments). This
step will again take 10 - 45 minutes depending on your machine and whether KVM
acceleration is available but only needs to be run once (unless you want to
modify the images).
170
171
172
173
174

Now you are ready to run simulations as with the pre-built docker images.

### Building From Source

175
176
177
178
179
180
Finally, it is of course possible to install the required dependencies directly
on your machine and then build and run SimBricks locally. Note that you will
need to install both the build dependencies for SimBricks but also for the
external simulators you need. We suggest you refer to the
[`docker/Dockerfile.buildenv`](docker/Dockerfile.buildenv) for the authoritative
list of required dependencies.
181

182
183
184
## Questions? Suggestions? Bugs?

If you are using SimBricks or are trying to determine if SimBricks is suitable
185
186
187
188
189
190
191
for what you are trying to do, we would love to hear from you. First off, please
feel free to report bugs or suggestions directly through [GitHub
issues](https://github.com/simbricks/simbricks/issues). If you have questions or
thoughts, please post them on our [GitHub discussion
board](https://github.com/simbricks/simbricks/discussions). Finally, we are also
available on
[Slack](https://join.slack.com/t/simbricks/shared_invite/zt-16y96155y-xspnVcm18EUkbUHDcSVonA)
192
193
for more interactive discussions or to answer quick questions.

194
195
## Repository Structure

196
- `doc/`: Documentation (Sphinx), automatically deployed on
197
198
199
200
201
  [Read The Docs](https://simbricks.readthedocs.io/en/latest/?badge=latest).
- `lib/simbricks/`: Libraries implementing SimBricks interfaces
  - `lib/simbricks/base`: Base protocol implementation responsible for
    connection setup, message transfer, and time synchronization between
    SimBricks component simulators.
202
203
  - `lib/simbricks/network`: Network protocol implementation carrying Ethernet
    packets between network components. Layers over the base protocol.
204
  - `lib/simbricks/pcie`: PCIe protocol implementation, roughly modelling PCIe
205
206
    at the transaction level, interconnecting hosts with PCIe device simulators.
    Layers over base protocol.
207
208
209
210
211
  - `lib/simbricks/nicbm`: Helper C++ library for implementing behavioral
    (high-level) NIC simulation models, offers similar abstractions as device
    models in other simulators such as gem-5.
  - `lib/simbricks/nicif`: *(deprecated)* Thin C library for NIC simulators
    establishing a network and a PCIe connection.
212
- `dist/`: Proxies for distributed SimBricks simulations running on multiple
213
  physical hosts.
214
215
  - `dist/sockets/`: Proxy transporting SimBricks messages over regular TCP
    sockets.
216
217
218
  - `dist/rdma/`: RDMA SimBricks proxy (not compiled by default).
- `sims/`: Component Simulators integrated into SimBricks.
  - `sims/external/`: Submodule pointers to repositories for existing external
Jonas Kaufmann's avatar
Jonas Kaufmann committed
219
    simulators (gem5, QEMU, Simics, ns-3, FEMU).
220
221
  - `sims/nic/`: NIC simulators
    - `sims/nic/i40e_bm`: Behavioral NIC model for Intel X710 40G NIC.
222
    - `sims/nic/corundum`: RTL simulation with Verilator of the
223
224
225
226
227
      [Corundum FPGA NIC](https://corundum.io/).
    - `sims/nic/corundum_bm`: Simple behavioral Corundum NIC model.
    - `sims/nic/e1000_gem5`: E1000 NIC model extracted from gem5.
  - `sims/net/`: Network simulators
    - `sims/net/net_switch`: Simple behavioral Ethernet switch model.
228
    - `sims/net/wire`: Simple Ethernet "wire" connecting two NICs back-to-back.
229
230
231
    - `sims/net/pktgen`: Packet generator.
    - `sims/net/tap`: Linux TAP device adapter.
    - `sims/net/tofino/`: Adapter for Intel Tofino Simulator.
232
    - `sims/net/menshen`: RTL simulation with Verilator for the
233
      [Menshen RMT Pipeline](https://isolation.quest/).
234
- `experiments/`: Python orchestration framework for running simulations.
235
  - `experiments/simbricks/orchestration/`: Orchestration framework implementation.
236
237
238
  - `experiments/run.py`: Main script for running simulation experiments.
  - `experiments/pyexps/`: Example simulation experiments.
- `images/`: Infrastructure to build disk images for host simulators.
239
240
241
242
  - `images/kernel/`: Slimmed down Linux kernel to reduce simulation time.
  - `images/mqnic/`: Linux driver for Corundum NIC.
  - `images/scripts/`: Scripts for installing packages in disk images.
- `docker/`: Scripts for building SimBricks Docker images.