tensor_attributes.rst

.. currentmodule:: oneflow

.. _tensor-attributes-doc:

Tensor Attributes
=============================================================

.. The documentation is referenced from: https://pytorch.org/docs/1.10/tensor_attributes.html.


Each local ``oneflow.Tensor`` has a :class:`oneflow.dtype`, :class:`oneflow.device`, and global ``oneflow.Tensor`` has a :class:`oneflow.dtype`, :class:`oneflow.placement`, :class:`oneflow.sbp`.

.. contents:: oneflow
    :depth: 2
    :local:
    :class: this-will-duplicate-information-and-it-is-still-useful-here
    :backlinks: top


.. _dtype-doc:

oneflow.dtype
-----------------------

.. class:: dtype

A :class:`oneflow.dtype` is an object that represents the data type of a
:class:`oneflow.Tensor`. Oneflow has eight different data types:

======================================= =============================================== =============================== ==================================
Data type                               dtype                                           CPU tensor                      GPU tensor
======================================= =============================================== =============================== ==================================
Boolean                                 ``oneflow.bool``                                :class:`oneflow.BoolTensor`     :class:`oneflow.cuda.BoolTensor`
8-bit integer (unsigned)                ``oneflow.uint8``                               :class:`oneflow.ByteTensor`     :class:`oneflow.cuda.ByteTensor`
8-bit integer (signed)                  ``oneflow.int8``                                :class:`oneflow.CharTensor`     :class:`oneflow.cuda.CharTensor`
64-bit floating point                   ``oneflow.float64`` or ``oneflow.double``       :class:`oneflow.DoubleTensor`   :class:`oneflow.cuda.DoubleTensor`
32-bit floating point                   ``oneflow.float32`` or ``oneflow.float``        :class:`oneflow.FloatTensor`    :class:`oneflow.cuda.FloatTensor`
16-bit floating point                   ``oneflow.float16`` or ``oneflow.half``         :class:`oneflow.HalfTensor`     :class:`oneflow.cuda.HalfTensor`
32-bit integer (signed)                 ``oneflow.int32`` or ``oneflow.int``            :class:`oneflow.IntTensor`      :class:`oneflow.cuda.IntTensor`
64-bit integer (signed)                 ``oneflow.int64`` or ``oneflow.long``           :class:`oneflow.LongTensor`     :class:`oneflow.cuda.LongTensor`
======================================= =============================================== =============================== ==================================


To find out if a :class:`oneflow.dtype` is a floating point data type, the property :attr:`is_floating_point`
can be used, which returns ``True`` if the data type is a floating point data type.

.. _type-promotion-doc:

When the dtypes of inputs to an arithmetic operation (`add`, `sub`, `div`, `mul`) differ, we promote
by finding the minimum dtype that satisfies the following rules:

* If the type of a scalar operand is of a higher category than tensor operands
  (where complex > floating > integral > boolean), we promote to a type with sufficient size to hold
  all scalar operands of that category.
* If a zero-dimension tensor operand has a higher category than dimensioned operands,
  we promote to a type with sufficient size and category to hold all zero-dim tensor operands of
  that category.
* If there are no higher-category zero-dim operands, we promote to a type with sufficient size
  and category to hold all dimensioned operands.

A floating point scalar operand has dtype `oneflow.get_default_dtype()` and an integral
non-boolean scalar operand has dtype `oneflow.int64`. Unlike numpy, we do not inspect
values when determining the minimum `dtypes` of an operand.  Quantized and complex types
are not yet supported.

Promotion Examples::

    >>> float_tensor = oneflow.ones(1, dtype=oneflow.float)
    >>> double_tensor = oneflow.ones(1, dtype=oneflow.double)
    >>> int_tensor = oneflow.ones(1, dtype=oneflow.int)
    >>> long_tensor = oneflow.ones(1, dtype=oneflow.long)
    >>> uint_tensor = oneflow.ones(1, dtype=oneflow.uint8)
    >>> double_tensor = oneflow.ones(1, dtype=oneflow.double)
    >>> bool_tensor = oneflow.ones(1, dtype=oneflow.bool)
    # zero-dim tensors
    >>> long_zerodim = oneflow.tensor(1, dtype=oneflow.long)
    >>> int_zerodim = oneflow.tensor(1, dtype=oneflow.int)

    >>> a,b=oneflow.tensor(5),oneflow.tensor(5)
    >>> oneflow.add(a, b).dtype
    oneflow.int64
    # 5 is an int64, but does not have higher category than int_tensor so is not considered.
    >>> (int_tensor + 5).dtype
    oneflow.int32
    >>> (int_tensor + long_zerodim).dtype
    oneflow.int64
    >>> (long_tensor + int_tensor).dtype
    oneflow.int64
    >>> (bool_tensor + long_tensor).dtype
    oneflow.int64
    >>> (bool_tensor + uint_tensor).dtype
    oneflow.uint8
    >>> (float_tensor + double_tensor).dtype
    oneflow.float64
    >>> (bool_tensor + int_tensor).dtype
    oneflow.int32
    # Since long is a different kind than float, result dtype only needs to be large enough
    # to hold the float.
    >>> oneflow.add(long_tensor, float_tensor).dtype
    oneflow.float32

When the output tensor of an arithmetic operation is specified, we allow casting to its `dtype` except that:
  * An integral output tensor cannot accept a floating point tensor.
  * A boolean output tensor cannot accept a non-boolean tensor.
  * A non-complex output tensor cannot accept a complex tensor

Casting Examples::

    # allowed:
    >>> float_tensor *= float_tensor
    >>> float_tensor *= int_tensor
    >>> float_tensor *= uint_tensor
    >>> float_tensor *= bool_tensor
    >>> int_tensor *= uint_tensor

    # disallowed (RuntimeError: result type can't be cast to the desired output type):
    >>> float_tensor *= double_tensor
    >>> int_tensor *= float_tensor
    >>> int_tensor *= long_tensor
    >>> uint_tensor *= int_tensor
    >>> bool_tensor *= int_tensor
    >>> bool_tensor *= uint_tensor

.. _device-doc:

oneflow.device
------------------------

.. class:: device

A :class:`oneflow.device` is an object representing the device on which a :class:`oneflow.Tensor` is
or will be allocated.

The :class:`oneflow.device` contains a device type (``'cpu'`` or ``'cuda'``) and optional device
ordinal for the device type. If the device ordinal is not present, this object will always represent
the current device for the device type, even after :func:`oneflow.cuda.set_device()` is called; e.g.,
a :class:`oneflow.Tensor` constructed with device ``'cuda'`` is equivalent to ``'cuda:X'`` where X is
the result of :func:`oneflow.cuda.current_device()`.

A :class:`oneflow.Tensor`'s device can be accessed via the :attr:`Tensor.device` property.

A :class:`oneflow.device` can be constructed via a string or via a string and device ordinal

Via a string:
::

    >>> oneflow.device('cuda:0')
    device(type='cuda', index=0)

    >>> oneflow.device('cpu')
    device(type='cpu', index=0)

    >>> oneflow.device('cuda')  # current cuda device
    device(type='cuda', index=0)

Via a string and device ordinal:

::

    >>> oneflow.device('cuda', 0)
    device(type='cuda', index=0)

    >>> oneflow.device('cpu', 0)
    device(type='cpu', index=0)

.. note::
   The :class:`oneflow.device` argument in functions can generally be substituted with a string.
   This allows for fast prototyping of code.

   >>> # Example of a function that takes in a oneflow.device
   >>> cuda1 = oneflow.device('cuda:1')
   >>> oneflow.randn((2,3), device=cuda1)

   >>> # You can substitute the oneflow.device with a string
   >>> oneflow.randn((2,3), device='cuda:1')

.. note::
   For legacy reasons, a device can be constructed via a single device ordinal, which is treated
   as a cuda device.  This matches :meth:`Tensor.get_device`, which returns an ordinal for cuda
   tensors and is not supported for cpu tensors.

   >>> oneflow.device(1)
   device(type='cuda', index=1)

.. note::
   Methods which take a device will generally accept a (properly formatted) string
   or (legacy) integer device ordinal, i.e. the following are all equivalent:

   >>> oneflow.randn((2,3), device=oneflow.device('cuda:1'))
   >>> oneflow.randn((2,3), device='cuda:1')
   >>> oneflow.randn((2,3), device=1)  # legacy

oneflow.placement
--------------------------------------------------------------
.. autoclass:: oneflow.placement

oneflow.placement.all
--------------------------------------------------------------
.. autofunction:: oneflow.placement.all

oneflow.env.all_device_placement
--------------------------------------------------------------
.. autofunction:: oneflow.env.all_device_placement

oneflow.sbp.sbp
--------------------------------------------------------------
.. autoclass:: oneflow.sbp.sbp