ParchGrad: Controlled Internal Gradients for Reliable Saliency Map [Preprint]
We prune gradient signals in convolutional layer while preserving the variance of gradient magnitudes to balance interaction between combined modules. We propose ParChGrad to obtain more reliable saliency map!
ParchGrad: Controlled Internal Gradients for Reliable Saliency Map 🚀
TDLR;
- Motivation : Gradients are shattered, and more refined internal signals can improve the saliency map.
- Problem : Comparison of internal gradients in a convolutional layer and make a measure properly compare gradient signals.
- Method : We propose $\gamma$-dominance based on the random variable formulation of gradients.
- Contribution : We propose variance conservation and class-wise channel selection to safely prune internal gradient signals.
- Keywords : saliency map, channel pruning
Explaining the decision of neural networks is crucial to ensure their reliability in various image tasks. Among several input attribution methods, the saliency map explains the decision with the propagated gradients on pixels. One of the fundamental problems of the saliency map is gradient shattering which leaves noise on pixels and hinders the interpretation of attribution maps. To provide a clear explanation, controlling the internal gradient signals is essential. However, to the best of our knowledge, there is no work on modifying the internal gradient signals so that the saliency map provides reliable explanations. To initialize the control of internal gradient signals in the saliency map, we provide theoretical analysis on the controllable modification of gradients in multiple convolutional channels and propose ParchGrad, which prunes channels by partitioning multiple channels into two groups. In addition, we propose a class-wise selection method based on statistical tests to select which channels to magnify or reduce the impact of gradients.
Visualization of ParchGrad vs Gradient
Introduction
Explaining the decision of neural networks is crucial to ensure their reliability in various image tasks. Among several input attribution methods, the saliency map explains the decision with the propagated gradients on pixels. One of the fundamental problems of the saliency map is gradient shattering which leaves noise on pixels and hinders the interpretation of attribution maps.
To provide a clear explanation, controlling the internal gradient signals is essential. However, to the best of our knowledge, there is no work on modifying the internal gradient signals so that the saliency map provides reliable explanations.
To initialize the control of internal gradient signals in the saliency map, we provide theoretical analysis on the controllable modification of gradients in multiple convolutional channels and propose ParchGrad, which prunes channels by partitioning multiple channels into two groups. In addition, we propose a class-wise selection method based on statistical tests to select which channels to magnify or reduce the impact of gradients.
Contribution: Comparison with other explainable methods.
| Method | Category | All layers | Prune Channels | #Propagation* | Sample Size |
|---|---|---|---|---|---|
| ScoreCAM |
CAM | X | O | O(C) | O(C) |
| AblationCAM |
CAM | X | O | O(C) | O(C) |
| LRP |
LRP | O | X | O(1) | O(1) |
| CRP |
LRP | O | O | O(1) | O(1) |
| Gradient |
Saliency | O | X | O(1) | O(1) |
| InputGrad |
Saliency | O | X | O(1) | O(1) |
| Integrated Gradient |
Saliency | O | X | O(1) | O(M) |
| SmoothGrad |
Saliency | O | X | O(M) | O(M) |
| ParchGrad (proposed) | Saliency | O | O | O(1) | O(1) |
- “All Layers” refers to whether including most of convolutional layers in a model for the explanation,
- “Pruning” refers to whether manual modification is applied,
- “Propagation” is the complexity of individual back-propagation to obtain the explanation.
- “Sample-size” is the total number of samples to obtain the explanation.
Problems
1. Gradients Are Noisy
Gradients are noisy to be interpreted as large number of small signals could exists on a input. A toy experiment below shows the impact of noisy gradients, which are randomly distributed.
Let $\mathcal{H}$ be a group of 10 random gradients whose first entry is sampled from $\mathcal{N}(0, 0.1)$ and the second entry is sampled from $\mathcal{N}(0.0, 1.0)$ and $\mathcal{L}$ be a group of 100 random gradients whose elements are both sampled from N(0, 0.1). The summed vector in each group and the combined group is shown in Figure below. Although H group is dominant in Dim2 (0,1), the summed vector of L group is dominant in Dim1 (1,0) because of large number of vectors. Consequently, the final direction is (1,1) direction (black). This experiment shows the impact of large number of irrelevant channels (red).
2. Zero Pruning of Gradients Result in Artifacts
One trivial method to remove noisy gradients is to set zero. However, we observe that the zero gradient signals on selected channels results in artifacts.
The artifacts are due to the fact that multiple multiple modules are interact to make gradient signal. When zero pruning is applied to the gradients in a convolutional layer, the impact of the convolutional layer is reduced and lost the dominance over other modules.
Methods : $\gamma$-dominance and variance conservation
To theoretically understand the pruning and impact of gradients, we define $\gamma$-dominance and provides theorems to preserve the magnitude of the gradients. Please see the paper for the detailed definition and the theorems.
Conclusion
Gradient signal is crucial in explainable AI for various reasons including training and explaining. As the saliency map is model-agnostic, gradient signals are the first option to explain models. However, as gradients are noisy, users cannot believe easily whether the attribution is reliable. To provide a more reliable saliency map, tackling the internal mechanism of gradients and controlling the signals is essential. This paper provides a theoretical and empirical analysis of the gradients so that the XAI community can clarify how the gradients impact the saliency map and how to control them. In detail, this paper shows the impact of variance over many channels and provides a solution to safely turn off the channel impact while conserving the magnitude with other modules.
We also propose class-wise channel selection, which selects channels based on the statistical importance of GAP. This selection could be understood as adding a forward bias in the backward (adding activation information to gradient computation). We acknowledge that class-wise selection bias is not the best option, but possibly a good option when there is no knowledge of channels. For the ongoing research, we will study other channel selection methods, such as CLIP-based or circuit-based channel selection, so that constructing proper channel propagation paths improves the explainability of the saliency maps.
We believe ParchGrad can broaden the overall understanding of saliency maps in many image application fields where obtaining a reliable saliency map is important.
