Digitala Vetenskapliga Arkivet

Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Evolving intelligence: Overcoming challenges for Evolutionary Deep Learning
Halmstad University, School of Information Technology.ORCID iD: 0000-0002-6040-2269
2024 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Deep Learning (DL) has achieved remarkable results in both academic and industrial fields over the last few years. However, DL models are often hard to design and require proper selection of features and tuning of hyper-parameters to achieve high performance. These selections are tedious for human experts and require substantial time and resources. A difficulty that encouraged a growing number of researchers to use Evolutionary Computation (EC) algorithms to optimize Deep Neural Networks (DNN); a research branch called Evolutionary Deep Learning (EDL).

This thesis is a two-fold exploration within the domains of EDL, and more broadly Evolutionary Machine Learning (EML). The first goal is to makeEDL/EML algorithms more practical by reducing the high computational costassociated with EC methods. In particular, we have proposed methods to alleviate the computation burden using approximate models. We show that surrogate-models can speed up EC methods by three times without compromising the quality of the final solutions. Our surrogate-assisted approach allows EC methods to scale better for both, expensive learning algorithms and large datasets with over 100K instances. Our second objective is to leverage EC methods for advancing our understanding of Deep Neural Network (DNN) design. We identify a knowledge gap in DL algorithms and introduce an EC algorithm precisely designed to optimize this uncharted aspect of DL design. Our analytical focus revolves around revealing avant-garde concepts and acquiring novel insights. In our study of randomness techniques in DNN, we offer insights into the design and training of more robust and generalizable neural networks. We also propose, in another study, a novel survival regression loss function discovered based on evolutionary search.

Place, publisher, year, edition, pages
Halmstad: Halmstad University Press, 2024. , p. 32
Series
Halmstad University Dissertations ; 109
Keywords [en]
neural networks, evolutionary deep learning, evolutionary machine learning, feature selection, hyperparameter optimization, evolutionary computation, particle swarm optimization, genetic algorithm
National Category
Computer Systems Signal Processing
Identifiers
URN: urn:nbn:se:hh:diva-52469ISBN: 978-91-89587-31-1 (electronic)ISBN: 978-91-89587-32-8 (print)OAI: oai:DiVA.org:hh-52469DiVA, id: diva2:1831077
Public defence
2024-02-16, Wigforss, Kristian IV:s väg 3, Halmstad, 08:00 (English)
Opponent
Supervisors
Available from: 2024-01-24 Created: 2024-01-24 Last updated: 2024-03-07
List of papers
1. Surrogate-Assisted Genetic Algorithm for Wrapper Feature Selection
Open this publication in new window or tab >>Surrogate-Assisted Genetic Algorithm for Wrapper Feature Selection
2021 (English)In: 2021 IEEE Congress on Evolutionary Computation (CEC), IEEE, 2021, p. 776-785Conference paper, Published paper (Refereed)
Abstract [en]

Feature selection is an intractable problem, therefore practical algorithms often trade off the solution accuracy against the computation time. In this paper, we propose a novel multi-stage feature selection framework utilizing multiple levels of approximations, or surrogates. Such a framework allows for using wrapper approaches in a much more computationally efficient way, significantly increasing the quality of feature selection solutions achievable, especially on large datasets. We design and evaluate a Surrogate-Assisted Genetic Algorithm (SAGA) which utilizes this concept to guide the evolutionary search during the early phase of exploration. SAGA only switches to evaluating the original function at the final exploitation phase.

We prove that the run-time upper bound of SAGA surrogate-assisted stage is at worse equal to the wrapper GA, and it scales better for induction algorithms of high order of complexity in number of instances. We demonstrate, using 14 datasets from the UCI ML repository, that in practice SAGA significantly reduces the computation time compared to a baseline wrapper Genetic Algorithm (GA), while converging to solutions of significantly higher accuracy. Our experiments show that SAGA can arrive at near-optimal solutions three times faster than a wrapper GA, on average. We also showcase the importance of evolution control approach designed to prevent surrogates from misleading the evolutionary search towards false optima.

Place, publisher, year, edition, pages
IEEE, 2021
Keywords
Feature selection, Wrapper, Genetic Algorithm, Progressive Sampling, Surrogates, Meta-models, Evolution Control, Optimization
National Category
Computer Sciences
Identifiers
urn:nbn:se:hh:diva-45893 (URN)10.1109/CEC45853.2021.9504718 (DOI)000703866100098 ()2-s2.0-85122940013 (Scopus ID)978-1-7281-8393-0 (ISBN)
Conference
2021 IEEE Congress on Evolutionary Computation (CEC), Kraków, Poland, 28 June - 1 July, 2021
Projects
EVE – Extending Life of Vehicles within Electromobility Era
Note

Som manuscript i avhandling/As manuscript in thesis

Available from: 2021-11-17 Created: 2021-11-17 Last updated: 2024-01-24Bibliographically approved
2. Extracting Invariant Features for Predicting State of Health of Batteries in Hybrid Energy Buses
Open this publication in new window or tab >>Extracting Invariant Features for Predicting State of Health of Batteries in Hybrid Energy Buses
Show others...
2021 (English)In: 2021 IEEE 8th International Conference on Data Science and Advanced Analytics (DSAA), Porto, Portugal, 6-9 Oct., 2021, IEEE, 2021, p. 1-6Conference paper, Published paper (Refereed)
Abstract [en]

Batteries are a safety-critical and the most expensive component for electric vehicles (EVs). To ensure the reliability of the EVs in operation, it is crucial to monitor the state of health of those batteries. Monitoring their deterioration is also relevant to the sustainability of the transport solutions, through creating an efficient strategy for utilizing the remaining capacity of the battery and its second life. Electric buses, similar to other EVs, come in many different variants, including different configurations and operating conditions. Developing new degradation models for each existing combination of settings can become challenging from different perspectives such as unavailability of failure data for novel settings, heterogeneity in data, low amount of data available for less popular configurations, and lack of sufficient engineering knowledge. Therefore, being able to automatically transfer a machine learning model to new settings is crucial. More concretely, the aim of this work is to extract features that are invariant across different settings.

In this study, we propose an evolutionary method, called genetic algorithm for domain invariant features (GADIF), that selects a set of features to be used for training machine learning models, in such a way as to maximize the invariance across different settings. A Genetic Algorithm, with each chromosome being a binary vector signaling selection of features, is equipped with a specific fitness function encompassing both the task performance and domain shift. We contrast the performance, in migrating to unseen domains, of our method against a number of classical feature selection methods without any transfer learning mechanism. Moreover, in the experimental result section, we analyze how different features are selected under different settings. The results show that using invariant features leads to a better generalization of the machine learning models to an unseen domain.

Place, publisher, year, edition, pages
IEEE, 2021
Keywords
State of Health Estimation, Remaining Useful Life Prediction, Invariant Features, Lithium-ion Battery, Transfer Learning, Electric vehicles, Predictive maintenance
National Category
Computer Sciences
Identifiers
urn:nbn:se:hh:diva-45895 (URN)10.1109/DSAA53316.2021.9564184 (DOI)000783799800049 ()2-s2.0-85126144193 (Scopus ID)
Conference
2021 IEEE 8th International Conference on Data Science and Advanced Analytics (DSAA), Porto, Portugal, 6-9 Oct., 2021
Funder
Vinnova
Note

Som manuscript i avhandling/As manuscript in thesis

Available from: 2021-11-17 Created: 2021-11-17 Last updated: 2024-01-24Bibliographically approved
3. Fast Genetic Algorithm for feature selection — A qualitative approximation approach
Open this publication in new window or tab >>Fast Genetic Algorithm for feature selection — A qualitative approximation approach
2023 (English)In: Expert systems with applications, ISSN 0957-4174, E-ISSN 1873-6793, Vol. 211, article id 118528Article in journal (Refereed) Published
Abstract [en]

Evolutionary Algorithms (EAs) are often challenging to apply in real-world settings since evolutionary computations involve a large number of evaluations of a typically expensive fitness function. For example, an evaluation could involve training a new machine learning model. An approximation (also known as meta-model or a surrogate) of the true function can be used in such applications to alleviate the computation cost. In this paper, we propose a two-stage surrogate-assisted evolutionary approach to address the computational issues arising from using Genetic Algorithm (GA) for feature selection in a wrapper setting for large datasets. We define “Approximation Usefulness” to capture the necessary conditions to ensure correctness of the EA computations when an approximation is used. Based on this definition, we propose a procedure to construct a lightweight qualitative meta-model by the active selection of data instances. We then use a meta-model to carry out the feature selection task. We apply this procedure to the GA-based algorithm CHC (Cross generational elitist selection, Heterogeneous recombination and Cataclysmic mutation) to create a Qualitative approXimations variant, CHCQX. We show that CHCQX converges faster to feature subset solutions of significantly higher accuracy (as compared to CHC), particularly for large datasets with over 100K instances. We also demonstrate the applicability of the thinking behind our approach more broadly to Swarm Intelligence (SI), another branch of the Evolutionary Computation (EC) paradigm with results of PSOQX, a qualitative approximation adaptation of the Particle Swarm Optimization (PSO) method. A GitHub repository with the complete implementation is available. © 2022 The Author(s)

Place, publisher, year, edition, pages
Oxford: Elsevier, 2023
Keywords
Evolutionary computation, Feature selection, Fitness approximation, Genetic Algorithm, Meta-model, Optimization, Particle Swarm Intelligence
National Category
Computer Sciences
Identifiers
urn:nbn:se:hh:diva-48909 (URN)10.1016/j.eswa.2022.118528 (DOI)000992359900001 ()2-s2.0-85137157028 (Scopus ID)
Note

Som manuscript i avhandling/As manuscript in thesis

Available from: 2022-12-16 Created: 2022-12-16 Last updated: 2024-01-24Bibliographically approved
4. Rolling The Dice For Better Deep Learning Performance: A Study Of Randomness Techniques In Deep Neural Networks
Open this publication in new window or tab >>Rolling The Dice For Better Deep Learning Performance: A Study Of Randomness Techniques In Deep Neural Networks
Show others...
2024 (English)In: Information Sciences, ISSN 0020-0255, E-ISSN 1872-6291, Vol. 667, p. 1-17, article id 120500Article in journal (Refereed) Published
Abstract [en]

This paper presents a comprehensive empirical investigation into the interactions between various randomness techniques in Deep Neural Networks (DNNs) and how they contribute to network performance. It is well-established that injecting randomness into the training process of DNNs, through various approaches at different stages, is often beneficial for reducing overfitting and improving generalization. However, the interactions between randomness techniques such as weight noise, dropout, and many others remain poorly understood. Consequently, it is challenging to determine which methods can be effectively combined to optimize DNN performance. To address this issue, we categorize the existing randomness techniques into four key types: data, model, optimization, and learning. We use this classification to identify gaps in the current coverage of potential mechanisms for the introduction of noise, leading to proposing two new techniques: adding noise to the loss function and random masking of the gradient updates.

In our empirical study, we employ a Particle Swarm Optimizer (PSO) to explore the space of possible configurations to answer where and how much randomness should be injected to maximize DNN performance. We assess the impact of various types and levels of randomness for DNN architectures applied to standard computer vision benchmarks: MNIST, FASHION-MNIST, CIFAR10, and CIFAR100. Across more than 30\,000 evaluated configurations, we perform a detailed examination of the interactions between randomness techniques and their combined impact on DNN performance. Our findings reveal that randomness in data augmentation and in weight initialization are the main contributors to performance improvement. Additionally, correlation analysis demonstrates that different optimizers, such as Adam and Gradient Descent with Momentum, prefer distinct types of randomization during the training process. A GitHub repository with the complete implementation and generated dataset is available. © 2024 The Author(s)

Place, publisher, year, edition, pages
Philadelphia, PA: Elsevier, 2024
Keywords
Neural Networks, Randomized Neural Networks, Convolutional Neural Network, hyperparameter optimization, Particle swarm optimization
National Category
Computer Systems
Identifiers
urn:nbn:se:hh:diva-52467 (URN)10.1016/j.ins.2024.120500 (DOI)001224296500001 ()2-s2.0-85188777216& (Scopus ID)
Available from: 2024-01-24 Created: 2024-01-24 Last updated: 2024-06-11Bibliographically approved
5. Improving Concordance Index in Regression-based Survival Analysis: Discovery of Loss Function for Neural Networks
Open this publication in new window or tab >>Improving Concordance Index in Regression-based Survival Analysis: Discovery of Loss Function for Neural Networks
Show others...
2024 (English)In: GECCO '24 Companion: Proceedings of the Genetic and Evolutionary Computation Conference Companion, New York: Association for Computing Machinery (ACM), 2024, p. 1863-1869Conference paper, Published paper (Other academic)
Abstract [en]

In this work, we use an Evolutionary Algorithm (EA) to discover a novel Neural Network (NN) regression-based survival loss function with the aim of improving the C-index performance. Our contribution is threefold; firstly, we propose an evolutionary meta-learning algorithm SAGA$_{loss}$ for optimizing a neural-network regression-based loss function that maximizes the C-index; our algorithm consistently discovers specialized loss functions that outperform MSCE. Secondly, based on our analysis of the evolutionary search results, we highlight a non-intuitive insight that signifies the importance of the non-zero gradient for the censored cases part of the loss function, a property that is shown to be useful in improving concordance. Finally, based on this insight, we propose MSCE$_{Sp}$, a novel survival regression loss function that can be used off-the-shelf and generally performs better than the Mean Squared Error for censored cases. We performed extensive experiments on 19 benchmark datasets to validate our findings. © 2024 is held by the owner/author(s).

Place, publisher, year, edition, pages
New York: Association for Computing Machinery (ACM), 2024
Keywords
evolutionary meta-learning, loss function, neural networks, survival analysis, regression
National Category
Computer Systems
Identifiers
urn:nbn:se:hh:diva-52468 (URN)10.1145/3638530.3664129 (DOI)2-s2.0-85200800944& (Scopus ID)979-8-4007-0495-6 (ISBN)
Conference
The Genetic and Evolutionary Computation Conference, Melbourne, Australia, July 14-18, 2024
Note

Som manuscript i avhandling/As manuscript in thesis

Available from: 2024-01-24 Created: 2024-01-24 Last updated: 2025-01-09Bibliographically approved

Open Access in DiVA

fulltext(1097 kB)273 downloads
File information
File name FULLTEXT02.pdfFile size 1097 kBChecksum SHA-512
c319e4b5e55043c003cd79cc3902e87b7cb8feac3418dfc49822bf1955cea642c341a1e59f41cb13835d3c0fa6c27cb6fe485f7b55ebe4addf3563f5459fdddb
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Altarabichi, Mohammed Ghaith
By organisation
School of Information Technology
Computer SystemsSignal Processing

Search outside of DiVA

GoogleGoogle Scholar
Total: 290 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 1589 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf