task: text-classification

Fixed parameters:

• model_name_or_path: Bhumika/roberta-base-finetuned-sst2
• dataset:
  • path: glue
  • eval_split: validation
  • data_keys: {'primary': 'sentence'}
  • ref_keys: ['label']
  • name: sst2
• quantization_approach: dynamic
• node_exclusion: []
• per_channel: False
• framework: onnxruntime
• framework_args:
  • opset: 15
  • optimization_level: 1
• aware_training: False

Benchmarked parameters:

• operators_to_quantize: ['Add', 'MatMul'], ['Add']

Evaluation

Below, time metrics for

• Batch size: 8
• Input length: 128 | operators_to_quantize | | latency_mean (original, ms) | latency_mean (optimized, ms) | | throughput (original, /s) | throughput (optimized, /s) | | accuracy (original) | accuracy (optimized) | | :-------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: | :-: | :-----------------: | :------------------: | | ['Add', 'MatMul'] | | | 619.76 | 161.66 | | | 1.80 | 6.20 | | | 1.000 | 1.000 | | ['Add'] | | | 611.74 | 478.48 | | | 1.80 | 2.20 | | | 1.000 | 1.000 |