Where Do Models Go Wrong? Parameter-Space Saliency Maps for Explainability

Thanks!

The parameter saliency approach can be naturally generalized to any architecture, not just vision. The changes that need to be made: 1. Aggregation. Currently, our code aggregates saliency of the model parameters by averaging on the conv filter level. We used that because in the literature filters have been shown to be interpretable. However, no aggregation can be used and the saliency profile can be computed on individual parameter level allowing for any architecture. 2. Loss. The loss is also not limited to classification losses, any other loss function can be used, e.g. metric learning or regression.

These should be fairly simple modifications of the code. Happy to help if needed!

I have benn working so so much with explainability. I think the main thing to know when trying to extend this to non vision (which I assume to be standard feature engineering based ml, let's ignore texts for a second) is that all the explainability is in the features. You have to design features that make sense. It really doesn't matter if you have the best explainability method in the world if that perfect explainability method retrasces the models prediction to features nobody understands.

Reading the first part of the titles, I thought maybe it is about poor career choices of fashion models...