Selective Brain Damage: Measuring the Disparate Impact of Model Pruning
S. Hooker, A. Courville, Y. Dauphin, and A. Frome / November 2019 / paper / blog
Neural network pruning techniques have demonstrated that it is possible to remove the majority of weights in a network with surprisingly little degradation to test set accuracy. However, this measure of performance conceals significant differences in how different classes and images are impacted by pruning. We find that certain examples, which we term pruning identified exemplars (PIEs), and classes are systematically more impacted by the introduction of sparsity. Removing PIE images from the test-set greatly improves top-1 accuracy for both pruned and non-pruned models. These hard-to-generalize-to images tend to be mislabelled, of lower image quality, depict multiple objects or require fine-grained classification. These findings shed light on previously unknown trade-offs, and suggest that a high degree of caution should be exercised before pruning is used in sensitive domains.
A trained neural network consists of a model architecture and a set of weights (the learned parameters of the model). These are typically large (they can be very large – the largest of Open AI’s pre-trained GPT-2 language model referred to in Section 1 is 6.2GB!). The size inhibits their storage and transmission and limits where they can be deployed. In resource-constrained settings, such as ‘at the edge’, compact models are clearly preferable.
With this in mind, methods to compress models have been developed. ‘Model pruning’ is one such method, in which some of the neural network’s weights are removed (set to zero) and hence do not need to be stored (reducing memory requirements) and do not contribute to computation at run time (reducing energy consumption and latency). Rather surprisingly, numerous experiments have shown that removing weights in this way has negligible effect on the performance of the model. The inspiration behind this approach is the human brain, which loses ‘50% of all synapses between the ages of two and ten’ in a process called synaptic pruning that improves ‘efficiency by removing redundant neurons and strengthening synaptic connections that are most useful for the environment’.
Because it’s ‘puzzling’ that neural networks are so robust to high levels of pruning, the authors of this paper probe what is actually lost. They find that although global degradation of a pruned network may be almost negligible, certain inputs or classes are disproportionately impacted, and this can have knock-on effects for other objectives such as fairness. They call this ‘selective brain damage’.
The pruning method that the authors use is ‘magnitude’ pruning, which is easy to understand and to implement (weights are successively removed during training if they are below a certain magnitude until a model sparsity target is reached) and very commonly used. The same formal evaluation methodology can be extended to other pruning techniques.
Why is it interesting?
The paper was rejected for ICLR2020. The discussion on OpenReview suggested that it was obvious that pruning would result in non-uniform degradation of performance and that, having proven this to be the case, the authors did not provide a solution.
It may indeed be obvious to the research community, but I wonder whether it is still obvious when it comes to production. Pruning is already a standard library function for compression. For instance, a magnitude pruning algorithm is part of Tensorflow’s Model Optimization toolkit, and pruning is one of the optimisation strategies implemented by semiconductor IP and embedded device manufacturers to automate compression of models for use on their technologies. What this paper highlights is that a naive use of pruning in production, one that looks only at overall model performance, might have negative implications for robustness and fairness objectives.
Will we see the impact of this in the next year?
I hope we’ll see much more discussion like this in the next year. As machine learning models increasingly become part of complex supply chains and integrations, we will require new methods to ensure that.
Signal: Rejected for ICLR2020
Who to follow: @sarahookr
Other things to read:
- One of the first papers (published in 1990) to investigate whether a version of synaptic pruning in human brains might work for artificial neural networks: ‘Optimal Brain Damage’ (paper)
- This is the paper that introduced me to the idea that one could remove 90% or more of weights without losing accuracy, at NeurIPS 2015: ‘Learning Both Weights and Connections for Efficient Neural Networks’ (paper)
- The ‘magnitude’ pruning algorithm, workshop track ICLR2018: ‘To Prune, or Not to Prune: Exploring the Efficacy of Pruning for Model Compression’ (paper)
Explainable Machine Learning in Deployment
U. Bhatt, A. Xiang et al. / 10 December 2019 / paper
Explainable machine learning offers the potential to provide stakeholders with insights into model behavior by using various methods such as feature importance scores, counterfactual explanations, or influential training data. Yet there is little understanding of how organizations use these methods in practice. This study explores how organizations view and use explainability for stakeholder consumption. We find that, currently, the majority of deployments are not for end users affected by the model but rather for machine learning engineers, who use explainability to debug the model itself. There is thus a gap between explainability in practice and the goal of transparency, since explanations primarily serve internal stakeholders rather than external ones. Our study synthesizes the limitations of current explainability techniques that hamper their use for end users. To facilitate end user interaction, we develop a framework for establishing clear goals for explainability. We end by discussing concerns raised regarding explainability.
The goal of this research was to study who is actually using ‘explainability’ techniques and how they are using them. The team conducted interviews with ‘roughly fifty people in approximately thirty organisations’. Twenty of these were data scientists not currently using explainability tools; the other 30 were individuals in organisations that have deployed explainability techniques.
The authors supply local definitions of terms such as ‘explainability’, ‘transparency’ and ‘trustworthiness’ for clarity and because they realised that they needed a shared language for the interviews, which were with individuals from a range of backgrounds – data science, academia and civil society. Therefore, ‘[e]xplainability refers to attempts to provide insights into a model’s behavior’, ‘[t]ransparency refers to attempts to provide stakeholders (particularly external stakeholders) with relevant information about how the model works’, while ‘[t]rustworthiness refers to the extent to which stakeholders can reasonably trust a model’s outputs’.
The authors found that where explanation techniques were in use, they were normally in the service of providing data scientists with insight to debug their systems or to ‘sanity check’ model outputs with domain experts rather than to help those affected by a model output to understand it. The types of technique favoured were those that were easy to implement rather than potentially the most illuminating; and causal explanations were desired but unavailable.
Why is it interesting?
Explainable machine learning is a busy area of research, driven by regulations such as Europe’s GDPR and by its centrality to the many attempts to articulate what is responsible AI – either as an explicit goal or as an enabler for higher-level principles such as justice, fairness and autonomy (explanations should facilitate informed consent and meaningful recourse for the subjects of algorithmic decision-making).
What this paper finds is that the explanation techniques that are currently available fall short of what is required in practice, substantively and technically. The authors offer some suggestions of what organisations should do and where research should focus next, to achieve the aim of building ‘trustworthy explainability solutions’. Their approach, of interviewing practitioners – i.e. of ‘user centred design’, is one I like and would like to see more of (the Holstein et al. paper (see ‘More’) is another notable recent example). It allows for a much broader consideration of utility than simple evaluation metrics – in this case, asking how easy a given solution is to use or to scale, and how useful the explanations it gives are for a given type of stakeholder – and highlights that there is often a gap between research evaluation criteria and deployment needs.
Complementary work has sought to understand what constitutes an explanation in a given context. I recommend looking up Project ExplAIn by the UK’s Information Commissioner’s Office (ICO) and The Alan Turing Institute, which sought to establish norms for the type of explanation required in given situations via Citizen Juries and put together practical guidance on AI explanation.
Starting with the type of explanation that is expected allows us to ask ourselves whether it is actually feasible. This is why it has been argued that there are some domains in which black box algorithms ought never to be deployed (see Cynthia Rudin under ‘More’). The finding that organisations want causal explanations is illustrative of this concern. Deep learning algorithms are good at capturing correlations between phenomena, but not at establishing which caused the other. Attempts to integrate causality into machine learning are an exciting frontier of research (see Bernhard Scholkopf under ‘More’).
As such, the paper is a welcome check to the proliferation of software libraries and commercial services that claim to offer explanation solutions and from which it’s easy to imagine that this problem is essentially solved. As lead author Umang Bhatt said when he presented the paper at FAT* in January, ‘Express scepticism about anyone claiming to be providing explanations.’
Will we see the impact of this in the next year?
Like research into machine learning algorithms themselves, research into explainability has been subject to cycles of hype and correction, and this is already leading to more nuanced discussions which should benefit everyone.
Signal: Accepted for FAT*2020
Who to follow: @umangsbhatt, @alicexiang, @RDBinns, @ICOnews, @turinginst
Other things to read:
- Holstein et al.: ‘Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?’ (paper)
- At NeurIPS 2018’s Critiquing and Correcting Trends in Machine Learning workshop, Cynthia Rudin argued, ‘Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead’ (paper)
- Bernhard Schölkopf: ‘Causality for Machine Learning’ (paper)
- The Alan Turing Institute and the Information Commissioner’s Office (ICO) Project ExplAIn (Interim report here, Guidance (draft) here). Final guidance due to be released later in 2020.