Please use this identifier to cite or link to this item: https://doi.org/10.21256/zhaw-20804
Publication type: Article in scientific journal
Type of review: Peer review (publication)
Title: Design patterns for resource-constrained automated deep-learning methods
Authors: Tuggener, Lukas
Amirian, Mohammadreza
Benites de Azevedo e Souza, Fernando
von Däniken, Pius
Gupta, Prakhar
Schilling, Frank-Peter
Stadelmann, Thilo
et. al: No
DOI: 10.3390/ai1040031
10.21256/zhaw-20804
Published in: AI
Volume(Issue): 1
Issue: 4
Pages: 510
Pages to: 538
Issue Date: 6-Nov-2020
Publisher / Ed. Institution: MDPI
ISSN: 2673-2688
Language: English
Subjects: Automated machine learning; Architecture design; Computer vision; Audio processing; Natural language processing; Weakly supervised learning
Subject (DDC): 006: Special computer methods
Abstract: We present an extensive evaluation of a wide variety of promising design patterns for automated deep-learning (AutoDL) methods, organized according to the problem categories of the 2019 AutoDL challenges, which set the task of optimizing both model accuracy and search efficiency under tight time and computing constraints. We propose structured empirical evaluations as the most promising avenue to obtain design principles for deep-learning systems due to the absence of strong theoretical support. From these evaluations, we distill relevant patterns which give rise to neural network design recommendations. In particular, we establish (a) that very wide fully connected layers learn meaningful features faster; we illustrate (b) how the lack of pretraining in audio processing can be compensated by architecture search; we show (c) that in text processing deep-learning-based methods only pull ahead of traditional methods for short text lengths with less than a thousand characters under tight resource limitations; and lastly we present (d) evidence that in very data- and computing-constrained settings, hyperparameter tuning of more traditional machine-learning methods outperforms deep-learning systems.
URI: https://digitalcollection.zhaw.ch/handle/11475/20804
Fulltext version: Published version
License (according to publishing contract): CC BY 4.0: Attribution 4.0 International
Departement: School of Engineering
Organisational Unit: Institute of Applied Information Technology (InIT)
Published as part of the ZHAW project: Ada – Advanced Algorithms for an Artificial Data Analyst
Appears in Collections:Publikationen School of Engineering

Files in This Item:
File Description SizeFormat 
2020_Tuggener-etal_Design-patterns-deep-learning-methods_AI.pdf6.28 MBAdobe PDFThumbnail
View/Open


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.