• Corpus ID: 54438210

ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

@article{Cai2019ProxylessNASDN,
  title={ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware},
  author={Han Cai and Ligeng Zhu and Song Han},
  journal={ArXiv},
  year={2019},
  volume={abs/1812.00332}
}
Neural architecture search (NAS) has a great impact by automatically designing effective neural network architectures. However, the prohibitive computational demand of conventional NAS algorithms (e.g. $10^4$ GPU hours) makes it difficult to \emph{directly} search the architectures on large-scale tasks (e.g. ImageNet). Differentiable NAS can reduce the cost of GPU hours via a continuous representation of network architecture but suffers from the high GPU memory consumption issue (grow linearly… 
View PDF on arXiv
969 Citations
Highly Influential Citations
242
Background Citations
487
Methods Citations
462
Results Citations
24

969 Citations

Citation Type

Citation Type

DA-NAS: Data Adapted Pruning for Efficient Neural Architecture Search
TLDR
DA-NAS is presented that can directly search the architecture for large-scale target tasks while allowing a large candidate set in a more efficient manner, and supports an argument search space to efficiently search the best-performing architecture.
Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
TLDR
This work proposes a novel framework called training-free neural architecture search (TE-NAS), which ranks architectures by analyzing the spectrum of the neural tangent kernel (NTK) and the number of linear regions in the input space and shows that these two measurements imply the trainability and expressivity of a neural network.
MemNAS: Memory-Efficient Neural Architecture Search With Grow-Trim Learning
TLDR
The proposed MemNAS is a novel growing and trimming based neural architecture search framework that optimizes not only performance but also memory requirement of an inference network and considers running memory use as an optimization objective along with performance.
FOUR GPU HOURS: A THEORETICALLY INSPIRED PERSPECTIVE
Neural Architecture Search (NAS) has been explosively studied to automate the discovery of top-performer neural networks. Current works require heavy training of supernet or intensive architecture…
Adapting Neural Architectures Between Domains
TLDR
The theoretical analyses lead to AdaptNAS, a novel and principled approach to adapt neural architectures between domains in NAS, which shows that only a small part of ImageNet will be sufficient for AdaptNAS to extend its architecture success to the entire ImageNet and outperform state-of the-art comparison algorithms.
XferNAS: Transfer Neural Architecture Search
TLDR
This work proposes a generally applicable framework that introduces only minor changes to existing optimizers to leverage this feature of Neural Architecture Search and demonstrates that the proposed framework generally gives better results and, in the worst case, is just as good as the unmodified optimizer.
Masters Thesis: Efficient Neural Network Architecture Search
One-Shot Neural Architecture Search (NAS) is a promising method to significantly reduce search time without any separate training. It can be treated as a Network Compression problem on the…
NASIB: Neural Architecture Search withIn Budget
TLDR
A new approach for NAS is proposed, called NASIB, which adapts and attunes to the computation resources available by varying the exploration vs. exploitation trade-off, which could lead to novel architectures that require lesser domain expertise, compared to the majority of the existing methods.
S3NAS: Fast NPU-aware Neural Architecture Search Methodology
TLDR
This paper presents a fast NPU-aware NAS methodology, called S3NAS, to find a CNN architecture with higher accuracy than the existing ones under a given latency constraint, and applies a modified Single-Path NAS technique to the proposed supernet structure.
Core-set Sampling for Efficient Neural Architecture Search
TLDR
This paper attempts to formulate the problem of reducing the large search time imposed by the heavy computational burden of neural architecture search based on the data curation manner, and proposes a key strategy to search the architecture using summarized data distribution, i.e., core-set.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 38 REFERENCES
Efficient Architecture Search by Network Transformation
TLDR
This paper proposes a new framework toward efficient architecture search by exploring the architecture space based on the current network and reusing its weights, and employs a reinforcement learning agent as the meta-controller, whose action is to grow the network depth or layer width with function-preserving transformations.
Efficient Neural Architecture Search via Parameter Sharing
TLDR
Efficient Neural Architecture Search is a fast and inexpensive approach for automatic model design that establishes a new state-of-the-art among all methods without post-training processing and delivers strong empirical performances using much fewer GPU-hours.
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
TLDR
This work proposes to evaluate the direct metric on the target platform, beyond only considering FLOPs, and derives several practical guidelines for efficient network design, called ShuffleNet V2.
SMASH: One-Shot Model Architecture Search through HyperNetworks
TLDR
A technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model's architecture is proposed, achieving competitive performance with similarly-sized hand-designed networks.
MnasNet: Platform-Aware Neural Architecture Search for Mobile
TLDR
An automated mobile neural architecture search (MNAS) approach, which explicitly incorporate model latency into the main objective so that the search can identify a model that achieves a good trade-off between accuracy and latency.
Multi-objective Architecture Search for CNNs
TLDR
This work proposes NASH, an architecture search which considerable reduces the computational resources required for training novel architectures by applying network morphisms and aggressive learning rate schedules and proposes Pareto-NASH, a method for multi-objective architecture search that allows approximating the Pare to-front of architectures under multiple objective, such as predictive performance and number of parameters, in a single run of the method.
Simple And Efficient Architecture Search for Convolutional Neural Networks
TLDR
Surprisingly, this simple method to automatically search for well-performing CNN architectures based on a simple hill climbing procedure whose operators apply network morphisms, followed by short optimization runs by cosine annealing yields competitive results.
DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures
TLDR
DPP-Net is proposed: Device-aware Progressive Search for Pareto-optimal Neural Architectures, optimizing for both device-related and device-agnostic objectives, which achieves better performances: higher accuracy & shorter inference time on various devices.
DARTS: Differentiable Architecture Search
TLDR
The proposed algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques.
Practical Block-Wise Neural Network Architecture Generation
TLDR
A block-wise network generation pipeline called BlockQNN which automatically builds high-performance networks using the Q-Learning paradigm with epsilon-greedy exploration strategy and offers tremendous reduction of the search space in designing networks which only spends 3 days with 32 GPUs.
...
1
2
3
4
...