task: question-answering

Fixed parameters:

• model_name_or_path: distilbert-base-uncased-distilled-squad
• dataset:
  • path: squad
  • name: None
  • calibration_split: train
  • eval_split: validation
  • data_keys: {'question': 'question', 'context': 'context'}
  • ref_keys: ['answers']
  • max_seq_length: 128
• node_exclusion: []
• per_channel: False
• calibration:
  • method: minmax
  • num_calibration_samples: 20
  • calibration_histogram_percentile: None
  • calibration_moving_average: None
  • calibration_moving_average_constant: None
• framework: onnxruntime
• framework_args:
  • opset: 15
  • optimization_level: 1
• aware_training: False

Benchmarked parameters:

• quantization_approach: dynamic, static
• operators_to_quantize: ['Add', 'MatMul'], ['Add']

Evaluation

Below, time metrics for

• Batch size: 8
• Input length: 128 | quantization_approach | operators_to_quantize | | latency_mean (original, ms) | latency_mean (optimized, ms) | | throughput (original, /s) | throughput (optimized, /s) | | exact_match (original) | exact_match (optimized) | | f1 (original) | f1 (optimized) | | :-------------------: | :-------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: | :-: | :--------------------: | :---------------------: | :-: | :-----------: | :------------: | | dynamic | ['Add', 'MatMul'] | | | 247.65 | 75.80 | | | 4.50 | 13.50 | | | 90.000 | 85.000 | | | 90.000 | 86.667 | | dynamic | ['Add'] | | | 234.61 | 191.78 | | | 4.50 | 5.50 | | | 90.000 | 90.000 | | | 90.000 | 90.000 | | static | ['Add', 'MatMul'] | | | 238.04 | 131.81 | | | 4.50 | 8.00 | | | 90.000 | 50.000 | | | 90.000 | 59.362 | | static | ['Add'] | | | 241.84 | 257.85 | | | 4.50 | 4.00 | | | 90.000 | 70.000 | | | 90.000 | 76.333 |