fix FX mode rcnn avgpool output config

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/327

Fix the bug for not leaving avgpool quantized when its input is quantized. This is not captured by CI because currently we set `small_pooler_resolution` to True to make test running faster, but this skips the `avgpool` everytime.

Reviewed By: jerryzh168

Differential Revision: D37706151

fbshipit-source-id: fde9a190324a23b97ba803e000f4caa54f6f3738

d2go/modeling/meta_arch/rcnn.py

@@ -282,7 +282,10 @@ def _fx_quant_prepare(self, cfg, example_input):
         self.roi_heads.box_head.avgpool,
         qconfig,
         fqn_to_example_inputs["roi_heads.box_head.avgpool"],
-        prepare_custom_config={"input_quantized_idxs": [0]},
+        prepare_custom_config={
+            "input_quantized_idxs": [0],
+            "output_quantized_idxs": [0],
+        },
     )
     self.roi_heads.box_predictor.cls_score = prep_fn(
         self.roi_heads.box_predictor.cls_score,