..
  Copyright 2022 Max Planck Institute for Software Systems, and
  National University of Singapore
..
  Permission is hereby granted, free of charge, to any person obtaining
  a copy of this software and associated documentation files (the
  "Software"), to deal in the Software without restriction, including
  without limitation the rights to use, copy, modify, merge, publish,
  distribute, sublicense, and/or sell copies of the Software, and to
  permit persons to whom the Software is furnished to do so, subject to
  the following conditions:
..
  The above copyright notice and this permission notice shall be
  included in all copies or substantial portions of the Software.
..
  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
  IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
  CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
  TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
  SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

###################################
gem5
###################################

`gem5 <https://www.gem5.org/>`_ is a modular computer architecture simulator that
can be configured to simulate a very broad range of different systems. For now,
we maintain our own fork of gem5 `on github
<https://github.com/simbricks/gem5>`_, which contains our SimBricks adapters, a
python configuration script for full system x86 simulations with SimBricks
adapters, and a few other extensions, such as MSI-X support. In the long term,
we hope to upstream these changes.

SimBricks Adapters
==================
We have added SimBricks adapters for PCI and Ethernet. The source for these
adapters and additional helper code are in `src/simbricks
<https://github.com/simbricks/gem5/tree/main/src/simbricks>`_ in the gem5 repo.

Common Functionality
--------------------
Much of the functionality for initialization and synchronization is agnostic to
the specific SimBricks protocol used. We have factored out this code into a
base class ``simbricks::base::Adapter`` that just implements the SimBricks base
protocol layer. The ``simbricks::base::GenericBaseAdapter`` class is a generic
helper class that casts incoming and outgoing messages to the appropriate type
specified in the template. Both adapters use this. Note that the adapters do not
extend this class, but instead inherit and implement the ``Interface`` member
class. The adapters then create an instance of the ``GenericBaseAdapter`` and
pass in a reference to themselves through the interface parameter in the
constructor.

The base adapter code also takes care of properly instantiating and connecting
multiple adapters. The ``simbricks::base::InitManager`` singleton object,
collects references to all adapters and exposes functions to wait until a
specific adapter is ready and connected. The ``InitManager`` does this
asynchronously to avoid deadlocks due to mismatched initialization order in
different simulators.

Ethernet
--------
The Ethernet adapter implements a gem5 ``EtherInt`` port, through which it can
be connected to gem5-internal NICs. We have tested this with gem5's e1000 NIC.
The adapter is implemented in ``simbricks::ethernet::Adapter``. This adapter is
also a good starting point for future adapters, as it is simple but demonstrates
how to use the base adapter functionality, and both receives and sends messages.

PCI
----
The PCI adapter ``simbricks::pci::adapter`` is more complicated as it handles
many more message types and interacts with the re st of gem5 in multiple ways
(MMIO, DMA, Interrupts).

To make matters worse, it also does some gymnastics to implement a PCI device in
gem5 that uses the asynchronous timing memory protocol in gem5, instead of the
default atomic protocol semantics for PCI devices in gem5. For this we override
the MMIO port created by the pci device super class, and implement our own
timing port. Hopefully in the future gem5 will offer a less backward way of
doing this but for now it works without drastically changing gem5's abstractions
and all the other devices using them.

Configuration
=============
gem5 is configured through python scripts. These scripts can be parametrized
through the command line. Part of the configuration can be specified and adapted
by varying the command line parameters, while many will require you to directly
change the python configurations. We include our reference configuration for x86
full system simulation capable of running Linux and with various SimBricks
adapter configurations in ``configs/simbricks/simbricks.py``. This script
heavily includes parts of the common gem5 configuration.

Checkpointing
=============
gem5 is currently the only simulator we use that supports checkpoint and
restore. The most common use-case for this is accelerating repeated simulations
by checkpointing system state after boot and running future simulations from
there. Note that SimBricks does not currently support distributed checkpoints,
and none of our other simulators support checkpointing either. To still leverage
this feature for accelerating boot, we carefully configure our simulations to
checkpoint before executing anything that affects state in other simulators, in
particular before loading device drivers. On resume, each gem5 instance will
restore from its own checkpoint while the rest of the simulators will just start
again from their respective initial state. As this state never changed in the
checkpointed system either for these components, this is still a consistent
system state.

Usage Notes
===========
  * gem5 only supports raw hard disk images. The SimBricks Makefile contains
    commands to build the raw images from the qcow2 images. (**TODO** link to
    docker documentation). 

  * gem5-kvm simulations require ``kvm`` support on the host and appropriate
    permissions for the user to access ``/dev/kvm``. Note that unlike qemu, gem5
    will fail with an error and not silently fall back to something slower.

  * gem5-kvm configurations require ``/proc/sys/kernel/perf_event_paranoid`` to
    be set to ``-1`` on the host system.
