Experiment Logs of "A Theoretical Explanation of Activation Sparsity through Flat Minima and Adversarial Robustness"
This repo holds the experiment logs of our manuscript. They include raw data of main results displayed in the paper, as well as some tentative experiments that are removed due to limit of length. They can be cloned and viewed by TensorBoard opened on corresponding subdirectories, or directly viewed online on Huggingface by clicking "Training metrics".
The terminology in log database may be different with those in the paper. For example, "concentration" is used instead of "sparsity", because in experiments for validation the activations are not sparse but concentrate around non-zero values.
Main Results
The raw data of results displayed in the manuscript can be found in
- Experiments for Validation (Sec 6):
runs/manipulate_width_implicit_relu2/vit_(activation_layer=weird_relu2/*/dataset=cifar10/*/lr=0.0001/ - Training ViT-Base/16 on ImageNet-1k from Scratch (Sec 7.1):
runs/imagenet1k/from_scratch5/ - Finetuning ViT-Base/16 for Sparsity (Sec 7.2):
runs/imagenet1k/finetuning_hard_uplifting/ - Training T5-Base on C4 from Scratch (Sec 7.1):
runs/T5/from_scratch/ - Finetuning T5-Base for Sparsity (Sec 7.2):
runs/T5/finetuning/ - Applicability of Theorem 8 (Sec 8.4):
runs/marchenko_pastur/
Tentative Experiments
Other subdirectories in runs/manipulate_width_implicit_relu2/ show activation concentration manipulation with non-squared weird activation functions. It can be seen that the jumpy-squaring drastically strengthens the dominance of gradient sparsity, because under non-squared activation functions, significantly less activations concentrate in the expected flat interval.
Training ViT from scratch with non-refined Zeroth Biases, i.e., without ZB clamping and LayerNorm restricting, can be found in runs/imagenet1k/from_scratch3 (from_scratch1, from_scratch2 and from_scratch4 are short training during which implementation bugs are found). The gradient clipping norm is 10 for both vanilla and modified training because we believed that this leads to more intense gradient noises and better flatness. These non-refined sparsifications bring about 20% sparsity improvements, showing the effectiveness of, even not refined, our modification. On the other hand, the improvements after the refinements reveal the critical role of LayerNorm when it comes to sparsity.
Finetuning ViT without LayerNorm uplifting can be found in runs/imagenet1k/finetuning. The sparsity hardly changed. This, again, indicates the significance of LayerNorm on sparsity.
Experiments using T5 happen after those of ViT so T5 does not have tentative experiments.