Corpus ID: 54438210

ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

@article{Cai2019ProxylessNASDN,
  title={ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware},
  author={Han Cai and Ligeng Zhu and Song Han},
  journal={ArXiv},
  year={2019},
  volume={abs/1812.00332}
}
Neural architecture search (NAS) has a great impact by automatically designing effective neural network architectures. However, the prohibitive computational demand of conventional NAS algorithms (e.g. $10^4$ GPU hours) makes it difficult to \emph{directly} search the architectures on large-scale tasks (e.g. ImageNet). Differentiable NAS can reduce the cost of GPU hours via a continuous representation of network architecture but suffers from the high GPU memory consumption issue (grow linearly… Expand
View PDF on arXiv
942 Citations
Highly Influential Citations
236
Background Citations
475
Methods Citations
445
Results Citations
24

Supplemental Code

Github Repo
Via Papers with Code
mit-han-lab/proxylessnas
[ICLR 2019] ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
1236 Stars • 249 Forks
Datasets
CIFAR-10
ImageNet
Github Repo
Via Papers with Code
osmr/imgclsmob
Sandbox for training deep learning networks
2177 Stars • 451 Forks
Datasets
CIFAR-10
ImageNet

942 Citations

Citation Type

Citation Type

DA-NAS: Data Adapted Pruning for Efficient Neural Architecture Search
TLDR
DA-NAS is presented that can directly search the architecture for large-scale target tasks while allowing a large candidate set in a more efficient manner, and supports an argument search space to efficiently search the best-performing architecture. Expand
Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
TLDR
This work proposes a novel framework called training-free neural architecture search (TE-NAS), which ranks architectures by analyzing the spectrum of the neural tangent kernel (NTK) and the number of linear regions in the input space and shows that these two measurements imply the trainability and expressivity of a neural network. Expand
MemNAS: Memory-Efficient Neural Architecture Search With Grow-Trim Learning
TLDR
The proposed MemNAS is a novel growing and trimming based neural architecture search framework that optimizes not only performance but also memory requirement of an inference network and considers running memory use as an optimization objective along with performance. Expand
FOUR GPU HOURS: A THEORETICALLY INSPIRED PERSPECTIVE
Neural Architecture Search (NAS) has been explosively studied to automate the discovery of top-performer neural networks. Current works require heavy training of supernet or intensive architecture… Expand
Adapting Neural Architectures Between Domains
TLDR
The theoretical analyses lead to AdaptNAS, a novel and principled approach to adapt neural architectures between domains in NAS, which shows that only a small part of ImageNet will be sufficient for AdaptNAS to extend its architecture success to the entire ImageNet and outperform state-of the-art comparison algorithms. Expand
XferNAS: Transfer Neural Architecture Search
TLDR
This work proposes a generally applicable framework that introduces only minor changes to existing optimizers to leverage this feature of Neural Architecture Search and demonstrates that the proposed framework generally gives better results and, in the worst case, is just as good as the unmodified optimizer. Expand
Masters Thesis: Efficient Neural Network Architecture Search
One-Shot Neural Architecture Search (NAS) is a promising method to significantly reduce search time without any separate training. It can be treated as a Network Compression problem on the… Expand
NASIB: Neural Architecture Search withIn Budget
TLDR
A new approach for NAS is proposed, called NASIB, which adapts and attunes to the computation resources available by varying the exploration vs. exploitation trade-off, which could lead to novel architectures that require lesser domain expertise, compared to the majority of the existing methods. Expand
S3NAS: Fast NPU-aware Neural Architecture Search Methodology
TLDR
This paper presents a fast NPU-aware NAS methodology, called S3NAS, to find a CNN architecture with higher accuracy than the existing ones under a given latency constraint, and applies a modified Single-Path NAS technique to the proposed supernet structure. Expand
Core-set Sampling for Efficient Neural Architecture Search
TLDR
This paper attempts to formulate the problem of reducing the large search time imposed by the heavy computational burden of neural architecture search based on the data curation manner, and proposes a key strategy to search the architecture using summarized data distribution, i.e., core-set. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 38 REFERENCES
Efficient Architecture Search by Network Transformation
TLDR
This paper proposes a new framework toward efficient architecture search by exploring the architecture space based on the current network and reusing its weights, and employs a reinforcement learning agent as the meta-controller, whose action is to grow the network depth or layer width with function-preserving transformations. Expand
Efficient Neural Architecture Search via Parameter Sharing
TLDR
Efficient Neural Architecture Search is a fast and inexpensive approach for automatic model design that establishes a new state-of-the-art among all methods without post-training processing and delivers strong empirical performances using much fewer GPU-hours. Expand
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
TLDR
This work proposes to evaluate the direct metric on the target platform, beyond only considering FLOPs, and derives several practical guidelines for efficient network design, called ShuffleNet V2. Expand
SMASH: One-Shot Model Architecture Search through HyperNetworks
TLDR
A technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model's architecture is proposed, achieving competitive performance with similarly-sized hand-designed networks. Expand
MnasNet: Platform-Aware Neural Architecture Search for Mobile
TLDR
An automated mobile neural architecture search (MNAS) approach, which explicitly incorporate model latency into the main objective so that the search can identify a model that achieves a good trade-off between accuracy and latency. Expand
Multi-objective Architecture Search for CNNs
TLDR
This work proposes NASH, an architecture search which considerable reduces the computational resources required for training novel architectures by applying network morphisms and aggressive learning rate schedules and proposes Pareto-NASH, a method for multi-objective architecture search that allows approximating the Pare to-front of architectures under multiple objective, such as predictive performance and number of parameters, in a single run of the method. Expand
Simple And Efficient Architecture Search for Convolutional Neural Networks
TLDR
Surprisingly, this simple method to automatically search for well-performing CNN architectures based on a simple hill climbing procedure whose operators apply network morphisms, followed by short optimization runs by cosine annealing yields competitive results. Expand
DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures
TLDR
DPP-Net is proposed: Device-aware Progressive Search for Pareto-optimal Neural Architectures, optimizing for both device-related and device-agnostic objectives, which achieves better performances: higher accuracy & shorter inference time on various devices. Expand
DARTS: Differentiable Architecture Search
TLDR
The proposed algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques. Expand
Practical Block-Wise Neural Network Architecture Generation
TLDR
A block-wise network generation pipeline called BlockQNN which automatically builds high-performance networks using the Q-Learning paradigm with epsilon-greedy exploration strategy and offers tremendous reduction of the search space in designing networks which only spends 3 days with 32 GPUs. Expand
...
1
2
3
4
...