Importantly, a reliable model should aim to do well in all of these areas simultaneously out-of-the-box, without requiring any customization for individual tasks. We posit three general categories of requirements for reliable machine learning (ML) systems: (1) they should accurately report uncertainty about their predictions ( “know what they don’t know”) (2) they should generalize robustly to new scenarios (distribution shift) and (3) they should be able to efficiently adapt to new data (adaptation).
#Ml net model builder how to#
Robust Generalization - A naïve model is sensitive to spurious correlations (“destination”), whereas Plex is robust.Īdaptation - Plex can actively choose the data from which it learns to improve performance more quickly.įirst, we explore how to understand the reliability of a model in novel scenarios. Cat classifier: Plex can say “I don’t know” for inputs that are neither cat nor dog.
#Ml net model builder code#
We are also open-sourcing the code to encourage further research into this approach. We illustrate the efficacy of Plex in the vision and language domains by applying these extensions to the current state-of-the-art Vision Transformer and T5 models, which results in significant improvement in their reliability. We also introduce Plex, a set of pre-trained large model extensions that can be applied to many different architectures. In “ Plex: Towards Reliability Using Pre-trained Large Model Extensions”, we present a framework for reliable deep learning as a new perspective about a model’s abilities this includes a number of concrete tasks and datasets for stress-testing model reliability. For models to be reliable in deployment, they must be able to accommodate shifts in data distribution and make useful decisions in a broad array of scenarios. In addition, the models may not be well-suited for applications where predictive performance is only part of the equation.
However, in practice, the data confronting models in real-world settings rarely match the training distribution.
Such models are most accurate when applied to test data drawn from the same distribution as their training set. Posted by Dustin Tran and Balaji Lakshminarayanan, Research Scientists, Google Researchĭeep learning models have made impressive progress in vision, language, and other modalities, particularly with the rise of large-scale pre-training.
0 kommentar(er)
