Researchers from Alibaba Group and Peking University have open sourced an efficient AutoML algorithm called Kernel Neural Architecture Search. The study sought a green Neural Architecture Search (NAS) solution that evaluates architectures without training.
KNAS uses a gradient kernel as a proxy for model quality and consumes less computational resources compared to standard techniques. The team proposed the following hypothesis: “Gradients can be used as a coarse proxy of downstream training to evaluate randomly initialized architectures.” The researchers found a gradient kernel (the mean of the Gram matrix (MGM) gradients) has a strong correlation with the accuracy of a model. The KNAS algorithm calculates the MGM for each proposed model architecture, keeping only the best ones, calculating the model accuracy for these candidates, and selecting the model with the highest accuracy as the final result.
Usually, neural architecture search systems are used to find the best deep learning model architecture for a task. To do this, the system finds a well-suited architecture to provide the best performance metric on the given task dataset and the search space of possible architectures. But, this method requires full training of each proposed model on the dataset, resulting in longer training times.