Introduction In real-world applications of machine learning, reliable and safe systems must considermeasures of performance beyond standard test set accuracy. These other goalsinclude out-of-distribution (OOD) robustness, prediction consistency, resilience toadversaries, calibrated uncertainty estimates, and the ability to detect anomalousinputs. However, improving performance towards these goals is often a balancingact that today’s methods cannot achieve without sacrificing performance on othersafety axes. For instance, adversarial training improves adversarial robustnessbut sharply degrades other classifier performance metrics. Similarly, strong dataaugmentation and regularization techniques often […]