One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking

@article{Chen2021OneShotNE,
  title={One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking},
  author={Minghao Chen and Houwen Peng and Jianlong Fu and Haibin Ling},
  journal={2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2021},
  pages={16525-16534}
}
Despite remarkable progress achieved, most neural architecture search (NAS) methods focus on searching for one single accurate and robust architecture. To further build models with better generalization capability and performance, model ensemble is usually adopted and performs better than stand-alone models. Inspired by the merits of model ensemble, we propose to search for multiple diverse models simultaneously as an alternative way to find powerful models. Searching for ensembles is non… 

Figures and Tables from this paper

Evolutionary Neural Cascade Search across Supernetworks
TLDR
ENCAS can be used to search over multiple pretrained supernetworks to achieve a trade-off front of cascades of different neural network architectures, maximizing accuracy while minimizing FLOPs count and leading to Pareto dominance in all computation regimes.
Bi-level Alignment for Cross-Domain Crowd Counting
TLDR
This work designs a bi-level alignment framework (BLA) consisting of task-driven data alignment and feature alignment, and introduces Au-toML to search for an optimal transform on source, which well serves for the downstream task.
HyperSegNAS: Bridging One-Shot Neural Architecture Search with 3D Medical Image Segmentation using HyperNet
TLDR
This work introduces a HyperNet to assist super-net training by incorporating architecture topology information, and shows that HyperSegNAS yields better performing and more intuitive architectures compared to the previous state-of-the-art segmentation networks; it can quickly and accurately find good architecture candidates under different computing constraints.
Hypernet-Ensemble Learning of Segmentation Probability for Medical Image Segmentation with Ambiguous Labels
TLDR
This paper proposes novel methods to improve the segmentation probability estimation without sacrificing performance in a real-world scenario that the authors have only one ambiguous annotation per image and proposes a unified hypernetwork ensemble method to alleviate the computational burden of training multiple networks.

References

SHOWING 1-10 OF 58 REFERENCES
MixPath: A Unified Approach for One-shot Neural Architecture Search
TLDR
This paper discovers that in the studied search space, feature vectors summed from multiple paths are nearly multiples of those from a single path, which perturbs supernet training and its ranking ability, and proposes a novel mechanism called Shadow Batch Normalization (SBN) to regularize the disparate feature statistics.
Single Path One-Shot Neural Architecture Search with Uniform Sampling
TLDR
A Single Path One-Shot model is proposed to construct a simplified supernet, where all architectures are single paths so that weight co-adaption problem is alleviated.
BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
TLDR
The proposed BigNAS, an approach that challenges the conventional wisdom that post-processing of the weights is necessary to get good prediction accuracies, is proposed, able to train a single set of shared weights on ImageNet and use these weights to obtain child models whose sizes range from 200 to 1000 MFLOPs.
Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search
TLDR
This work introduces the concept of prioritized path, which refers to the architecture candidates exhibiting superior performance during training, and directly select the most promising one from the prioritized paths as the final architecture, without using other complex search methods, such as reinforcement learning or evolution algorithms.
Block-Wisely Supervised Neural Architecture Search With Knowledge Distillation
TLDR
This work proposes to modularize the large search space of NAS into blocks to ensure that the potential candidate architectures are fully trained, and distill the neural architecture (DNA) knowledge from a teacher model to supervise the block-wise architecture search, which significantly improves the effectiveness of NAS.
GreedyNAS: Towards Fast One-Shot NAS With Greedy Supernet
TLDR
This paper proposes a multi-path sampling strategy with rejection, and greedily filter the weak paths to ease the burden of supernet by encouraging it to focus more on evaluation of those potentially-good ones, which are identified using a surrogate portion of validation data.
Angle-based Search Space Shrinking for Neural Architecture Search
TLDR
Comp comprehensive evidences are provided showing that, in weight-sharing supernet, the proposed metric is more stable and accurate than accuracy-based and magnitude-based metrics to predict the capability of child models.
Improving One-Shot NAS by Suppressing the Posterior Fading
TLDR
This paper analyzes existing weight sharing one-shot NAS approaches from a Bayesian point of view and identifies the Posterior Fading problem, which compromises the effectiveness of shared weights, and presents a novel approach to guide the parameter posterior towards its true distribution.
Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation
TLDR
This paper presents an efficient algorithm which allows the depth of searched architectures to grow gradually during the training procedure, and solves two issues, namely, heavier computational overheads and weaker search stability, which are solved using search space approximation and regularization.
ASAP: Architecture Search, Anneal and Prune
TLDR
A differentiable search space is proposed that allows the annealing of architecture weights, while gradually pruning inferior operations, in this way, the search converges to a single output network in a continuous manner.
...
...