doc: refine existing and add documentation for disk images

doc/external_links.py

@@ -2,6 +2,8 @@
 
 extlinks = {
     'simbricks-repo': ('https://github.com/simbricks/simbricks%s', 'README%s'),
+    'simbricks-examples':
+        ('https://github.com/simbricks/simbricks-examples%s', None),
     'gem5-fork': ('https://github.com/simbricks/gem5%s', None),
     'slack': (
         'https://join.slack.com/t/simbricks/shared_invite/zt-16y96155y-xspnVcm18EUkbUHDcSVonA%s',

doc/user/howto.rst

@@ -121,6 +121,36 @@ be found in the module :mod-orchestration:`nodeconfig.py` directly.
 Add a Custom Image
 ******************************
 
+First off, make sure you actually need a custom image. You can find more
+information on this under :ref:`sec-images`. We have a tutorial in our
+:simbricks-examples:`examples repository <>` that highlights how to add a
+custom, out-of-tree image for a simple Memcached experiment. Further, we are
+currently reworking our orchestration framework to greatly simplify the process
+of defining your own custom image, abstracting away the need to manually build
+and manage it.
+
+For the moment, here is some additional information on the inner details of the
+image building process: We use `Packer <https://www.packer.io/>`_, which
+essentially starts a QEMU VM with internet access, boots up the kernel and then
+runs an installation script inside to change configs and install required
+packages either through ``apt`` or from source. You can find examples of
+installation scripts for our in-tree images under
+:simbricks-repo:`images/scripts </blob/main/images/scripts>`. The commands for
+building them reside in the file :simbricks-repo:`images/rules.mk
+</blob/main/images/rules.mk>`.
+
+The produced images are stored as ``images/output-<image_name>/<image-name>``.
+By default, we produce disk images in the compressed ``qcow2`` format. Some of
+our host simulators, e.g. gem5 and Simics, require raw disk images though, which
+are substantially larger. The ``qcow2`` can be easily converted to raw with
+``qemu-img convert``. For our in-tree images there's a Makefile target for this,
+which stores the converted images as
+``images/output-<image_name>/<image-name>.raw``.
+
+.. code-block:: bash
+
+  $ make convert-images-raw
+
 ******************************
 Integrate a New Simulator
 ******************************

doc/user/orchestration.rst

@@ -112,8 +112,7 @@ To configure the workload and the software environment of nodes, use the classes
 :class:`~simbricks.orchestration.nodeconfig.AppConfig`. The former is passed to
 every host simulator and defines, for example, the networking configuration like
 IP address and subnet mask, how much system memory the node has, but also which
-disk image to run. You can read more about the latter under
-:ref:`sec-howto-custom_image`.
+disk image to run. You can read more about the latter under :ref:`sec-images`.
 
 The :class:`~simbricks.orchestration.nodeconfig.NodeConfig` contains an
 attribute for an instance of
@@ -173,30 +172,28 @@ also possible to increase simulation performance by trading-off accuracy using a
 higher setting. For more information, refer to the section on
 :ref:`sec-synchronization` in the :ref:`page-architectural-overview`.
 
+.. _sec-images:
+
 ******************************
 Images
 ******************************
 
-All host simulators require a disk image. We already provide a base image with
-Ubuntu. If you just want to copy in additional files, e.g., drivers and
-executables for your workload, you don't need to build your own image. You can
-use the method
-:meth:`~simbricks.orchestration.nodeconfig.NodeConfig.config_files` of
-:class:`~simbricks.orchestration.nodeconfig.NodeConfig` and
-:class:`~simbricks.orchestration.nodeconfig.AppConfig` to mount additional files
-under ``/tmp/guest`` inside the simulated OS.
-
-For more than that, you need to build your own images. You can find the commands
-to do so in ``images/rules.mk``. When building an image, it is started with Qemu
-in a VM with active internet access. The image-specific script located in
-``images/scripts`` is then executed on the guest system to modify the image,
-e.g., changing config files, installing packages, or building required projects
-from source. In order to use your image in experiments, set the attribute
-:attr:`~simbricks.orchestration.nodeconfig.NodeConfig.disk_image` on
-:class:`~simbricks.orchestration.nodeconfig.NodeConfig`. This requires that your
-image is stored as ``images/out-<image_name>/<image_name>``. If your host
-simulator requires a raw image, execute ``make
-images/out-<image_name>/<image_name>.raw`` to convert your image.
+All our host simulators boot up a proper Operating System and therefore require
+a disk image. We already provide a minimal base image using Ubuntu and some
+experiment-specific derivatives with additional packages installed. If you just
+want to copy in additional files for your experiment, such as drivers and
+executables, you don't need to build your own image. You can just override the
+method :meth:`~simbricks.orchestration.nodeconfig.NodeConfig.config_files` of
+:class:`~simbricks.orchestration.nodeconfig.AppConfig` or
+:class:`~simbricks.orchestration.nodeconfig.NodeConfig` to mount additional
+files under ``/tmp/guest`` inside the simulated OS.
+
+For anything more than that, for example to install additional packages, you
+need to build your own image. You can find information on how to do so under
+:ref:`sec-howto-custom_image`. The specific image that you want to use for a
+host in your experiment is specified in the
+:class:`~simbricks.orchestration.nodeconfig.NodeConfig` class via the attribute
+:attr:`~simbricks.orchestration.nodeconfig.NodeConfig.disk_image`.
 
 *************************************
 Checkpoints