Wizard Coder SQL-Generation Model

Overview

• Model Name: WizardCoderSQL-15B-V1.0
• Repository: WizardLM/WizardCoder-15B-V1.0
• License: OpenRAIL-M
• Fine-Tuned Model Name: WizardCoderSQL-15B-V1.0
• Fine-Tuned Dataset: bugdaryan/spider-natsql-wikisql-instruct

Description

This is a fine-tuned version of the Wizard Coder 15B model specifically designed for SQL generation tasks. The model has been fine-tuned on the bugdaryan/spider-natsql-wikisql-instruct dataset to empower it with the ability to generate SQL queries based on natural language instructions.

Model Details

• Base Model: Wizard Coder 15B
• Fine-Tuned Model Name: WizardCoderSQL-15B-V1.0
• Fine-Tuning Parameters:
  • QLoRA Parameters:
    • LoRA Attention Dimension (lora_r): 64
    • LoRA Alpha Parameter (lora_alpha): 16
    • LoRA Dropout Probability (lora_dropout): 0.1
  • bitsandbytes Parameters:
    • Use 4-bit Precision Base Model (use_4bit): True
    • Compute Dtype for 4-bit Base Models (bnb_4bit_compute_dtype): float16
    • Quantization Type (bnb_4bit_quant_type): nf4
    • Activate Nested Quantization (use_nested_quant): False
  • TrainingArguments Parameters:
    • Number of Training Epochs (num_train_epochs): 1
    • Enable FP16/BF16 Training (fp16/bf16): False/True
    • Batch Size per GPU for Training (per_device_train_batch_size): 48
    • Batch Size per GPU for Evaluation (per_device_eval_batch_size): 4
    • Gradient Accumulation Steps (gradient_accumulation_steps): 1
    • Enable Gradient Checkpointing (gradient_checkpointing): True
    • Maximum Gradient Norm (max_grad_norm): 0.3
    • Initial Learning Rate (learning_rate): 2e-4
    • Weight Decay (weight_decay): 0.001
    • Optimizer (optim): paged_adamw_32bit
    • Learning Rate Scheduler Type (lr_scheduler_type): cosine
    • Maximum Training Steps (max_steps): -1
    • Warmup Ratio (warmup_ratio): 0.03
    • Group Sequences into Batches with Same Length (group_by_length): True
    • Save Checkpoint Every X Update Steps (save_steps): 0
    • Log Every X Update Steps (logging_steps): 25
  • SFT Parameters:
    • Maximum Sequence Length (max_seq_length): 500

Performance

• Fine-Tuned Model Metrics: ([TBA])

Dataset

• Fine-Tuned Dataset: bugdaryan/spider-natsql-wikisql-instruct
• Dataset Description: This dataset contains natural language instructions paired with SQL queries. It serves as the training data for fine-tuning the Wizard Coder model for SQL generation tasks.

Model Card Information

• Maintainer: Spartak Bughdaryan
• Contact: bugdaryan@gmail.com
• Date Created: September 15, 2023
• Last Updated: September 15, 2023

Usage

To use this fine-tuned model for SQL generation tasks, you can load it using the Hugging Face Transformers library in Python. Here's an example of how to use it:

from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    pipeline
)
import torch

model_name = 'bugdaryan/WizardCoderSQL-15B-V1.0'

model = AutoModelForCausalLM.from_pretrained(model_name, device_map='auto')
tokenizer = AutoTokenizer.from_pretrained(model_name)

pipe = pipeline('text-generation', model=model, tokenizer=tokenizer)

tables = "CREATE TABLE sales ( sale_id number PRIMARY KEY, product_id number, customer_id number, salesperson_id number, sale_date DATE, quantity number, FOREIGN KEY (product_id) REFERENCES products(product_id), FOREIGN KEY (customer_id) REFERENCES customers(customer_id), FOREIGN KEY (salesperson_id) REFERENCES salespeople(salesperson_id)); CREATE TABLE product_suppliers ( supplier_id number PRIMARY KEY, product_id number, supply_price number, FOREIGN KEY (product_id) REFERENCES products(product_id)); CREATE TABLE customers ( customer_id number PRIMARY KEY, name text, address text ); CREATE TABLE salespeople ( salesperson_id number PRIMARY KEY, name text, region text ); CREATE TABLE product_suppliers ( supplier_id number PRIMARY KEY, product_id number, supply_price number );"

question = 'Find the salesperson who made the most sales.'

prompt = f"Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. ### Instruction: Convert text to SQLite query: {question} {tables} ### Response:"

ans = pipe(prompt, max_new_tokens=200)
print(ans[0]['generated_text'])

Disclaimer

WizardCoderSQL model follows the same license as WizardCoder. The content produced by any version of WizardCoderSQL is influenced by uncontrollable variables such as randomness, and therefore, the accuracy of the output cannot be guaranteed by this project. This project does not accept any legal liability for the content of the model output, nor does it assume responsibility for any losses incurred due to the use of associated resources and output results.