Presentation Information
[4M4-GS-2e-01]Neural network - additive GPR hybrid with optimized redundant features for robust and insightful MLSuppression of overfitting, simplicity, and possibility to obviate deep NNs in certain applications
〇Sergei Manzhos1, Manabu Ihara1 (1. Institute of Science Tokyo)
Keywords:
additive kernel,neural network,overfitting,insightful ML
Neural networks (NN) and kernel regressions are the most widely used methods for regression-type machine learning. Deep NNs are popular but come at a cost of a large number of nonlinear optimization parameters that complicate training, lead to overfitting, and increase the CPU cost of recall. Kernel regression avoids the use of many nonlinear parameters, but trails the expressive power of deep NNs.
We show that additive kernel-based Gaussian process regression (GPR) in redundant features that are optimized with a Monte Carlo algorithm possesses simultaneously robustness of kernel regression and expressive power of a neural network (NN). The method is further development of the GPR-NN hybrid idea of [J. Phys. Chem. A 127 (2023) 7823] that used rule-based weights and is equivalent to a single hidden layer neural network with optimized (for given data) neuron activation functions and optimized weights. We show that this results in an expressive power closer to that of a multilayer NN and could obviate the need for deep NNs in some applications. A dimensionality reduction regime is also possible. All terms of the representation are obtained in a single linear step, and there is no overfitting when the number of terms / neurons is increased beyond optimal.
The method can be used to generate insight and is naturally amenable to symbolic regression.
We show that additive kernel-based Gaussian process regression (GPR) in redundant features that are optimized with a Monte Carlo algorithm possesses simultaneously robustness of kernel regression and expressive power of a neural network (NN). The method is further development of the GPR-NN hybrid idea of [J. Phys. Chem. A 127 (2023) 7823] that used rule-based weights and is equivalent to a single hidden layer neural network with optimized (for given data) neuron activation functions and optimized weights. We show that this results in an expressive power closer to that of a multilayer NN and could obviate the need for deep NNs in some applications. A dimensionality reduction regime is also possible. All terms of the representation are obtained in a single linear step, and there is no overfitting when the number of terms / neurons is increased beyond optimal.
The method can be used to generate insight and is naturally amenable to symbolic regression.