• Nicolas Hug's avatar
    Add quantized version of nms (#3601) · f74bfab6
    Nicolas Hug authored
    * Add quantized version of nms
    
    * Added tests
    
    * Compute areas only once
    
    * remove calls to dequantize_val
    
    * fix return type for empty tensor
    
    * flake8
    
    * remove use of scale as it gets cancelled out
    
    * simpler int convertion in tests
    
    * explicitly set ovr to double
    
    * add tests for more values of scale and zero_point
    
    * comment about underflow
    
    * remove unnecessary accessor
    
    * properly convert to float for division
    
    * Add comments about underflow
    
    * explicitely cast coordinates to float to allow vectorization
    
    * clang
    
    * clang  again
    
    * hopefully OK now
    f74bfab6
setup.py 17.3 KB