Pilota model for scud2query

A model for Pilota trained with https://github.com/megagonlabs/scud2query.

• scud
  • Fine tuned model of t5-base-japanese-web (with Byte-fallback, 8K)
  • The original model is distributed in the Apache License 2.0
• scorer
  • Fine tuned model of LINE DistilBERT Japanese
  • The original model is distributed in the Apache License 2.0

Usage

1. Install Pilota

2. Prepare inputs

   • Command

     echo -e '部屋に冷蔵庫があると良い。レンタカーサービスがあるホテルを【customer】が希望する。' | python -m pilota.convert.plain2request | tee input.jsonl
     

   • Output

     {"context":null,"utterance":"部屋に冷蔵庫があると良い。レンタカーサービスがあるホテルを【customer】が希望する。","sentences":null,"meta":{}}
     

3. Feed it to Pilota

   • Command

     pilota -m megagonlabs/pilota_scud2query --batch_size 1 --outlen 60 --nbest 1 --beam 5 < input.jsonl
     

   • Output (Formatted by jq .)

     [
       {
         "scuds_nbest": [
           [
             "部屋に冷蔵庫がある。"
           ]
         ],
         "original_ranks": [
           0
         ],
         "scores": [
           0.9769772589206696
         ],
         "scores_detail": [
           {
             "OK": 0.9232575297355652,
             "incorrect_none": 0.0034886503126472235,
             "lack": 0.023772092536091805,
             "limited": 0.013821585103869438,
             "untruth": 0.04332486167550087
           }
         ],
         "sentence": "部屋に冷蔵庫があると良い。"
       },
       {
         "scuds_nbest": [
           [
             "レンタカーサービスがあるホテルだ。"
           ]
         ],
         "original_ranks": [
           0
         ],
         "scores": [
           0.9876023113727569
         ],
         "scores_detail": [
           {
             "OK": 0.9586743712425232,
             "incorrect_none": 0.004059707745909691,
             "lack": 0.0024317132774740458,
             "limited": 0.007630097679793835,
             "untruth": 0.04025880992412567
           }
         ],
         "sentence": "レンタカーサービスがあるホテルを【customer】が希望する。"
       }
     ]
     

License

Apache License 2.0