# Parallel hyperparameter optimization of spiking neural networks
> [Français](README_fr.md)
This repository contains hyperlinks to redirect the reader to the source code of each chapter from the thesis [Parallel hyperparameter optimization of spiking neural networks](https://theses.fr/s327519).
The thesis is accessible at (_available once published_):
*[HAL]()
## Summary
> Artificial Neural Networks (ANNs) are a machine learning technique that has become indispensable. By learning from data, ANNs make it possible to solve certain complex cognitive tasks. Over the last three decades, ANNs have seen numerous major advances. These advances have enabled the development of image recognition, large language models or text-to-image conversion. Undeniably, ANNs have become an invaluable tool for many applications and this growing interest led in 2020 to the boom of generative models. However, several new barriers could put the brakes on the interest in these models. The first brake is the end of Moore's Law, due to the physical limits reached by transistors. But also, while research has long focused on the predictive performances of ANNs, other aspects have been neglected. These include energy efficiency, robustness, security, interpretability, transparency and so on [(1)](https://futureoflife.org/open-letter/pause-giant-ai-experiments/). This is why we need to go beyond von Neumann architectures for reducing the energy footprint, and the neuromorphic approach is a serious breakthrough candidate through biomimicry of the human brain via Spiking Neural Networks (SNNs).
> Unfortunately, SNNs are currently struggling to outperform conventional methods. As they are more recent and therefore less studied, a better approach to their design could make it possible to combine performance and low-energy cost. That is why, the automatic design of SNNs is studied within this thesis, with a focus on HyperParameter Optimization (HPO). The aim is to improve the HPO algorithms and to better understand the behavior of SNNs regarding their hyperparameters.
## Zellij
Zellij is the main Python package made for this thesis, including both FBD and HPO algorithms.
The actual version of the thesis was freezed. The documentation is not up-to-date.
> [Github to Zellij](https://github.com/ThomasFirmin/zellij/tree/thesis_freeze)
## Chapter 4 - Partition-based global optimization
Chapter 4 describes a generalization of a family of algorithms, based on a hierarchical decomposition of the search space. We introduce Zellij a framework unifying various algorithms from different research fields. The chapter ends on a discussion about a new algorithm based on a decomposition via Latin Hypercubes.
> [Code of chapter 4](https://gitlab.cristal.univ-lille.fr/tfirmin/thesis_code_chap4)
## Chapter 5 - Silent networks: a vicious trap for Hyperparameter Optimization
Chapter 5 tackles hyperparameter optimization applied to spiking neural networks. The work focuses on a specific type of spiking neural networks, named silent networks. This peculiar networks emits only wether a few or no spikes at all. This phenomenon is a generalization of the signal loss problem, as it the extreme decrease, yet the absence of spiking activity is explained by mistuned architecture or hyperparameters. We empirically emphasize that silent networks have an impact on the optimization algorithm. We leverage this phenomenon to improve the hyperparameter optimization.
> [Code of chapter 5](https://gitlab.cristal.univ-lille.fr/tfirmin/thesis_code_chap5)
## Chapter 6 - Accelerating Hyperparameter Optimization with Multi-Fidelity
Chapter 6 enhances the results previously obtained by leveraging both silent networks and multi-fidelity optimization. Indeed, silent networks can slow down the convergence of the optimization by about 10 hours for experiments lasting 100 hours. The methodology designed in chapter 5, allowed to accelerate the optimization by a few hours. Then, by combining it with multi-fidelity, we were able to reduce the necessary budget by 7.
> [Code of chapter 6](https://gitlab.cristal.univ-lille.fr/tfirmin/thesis_code_chap6)
## Authors and acknowledgment
* Author: Thomas Firmin
* Supervisor: El-Ghazali Talbi
* Co-Supervisor: Pierre Boulet
Experiments presented in this work were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see \url{https://www.grid5000.fr}).
This work was granted access to the HPC resources of IDRIS under the allocation 2023-AD011014347 made by GENCI.
This work has been supported by the University of Lille, the ANR-20-THIA-0014 program AI\_PhD$@$Lille and the ANR PEPR AI and Numpex. It was also supported by IRCICA(CNRS and Univ. Lille USR-3380).
# Parallel hyperparameter optimization of spiking neural networks
> [English](README_en.md)
This repository contains hyperlinks to redirect the reader to the source code of each chapter from the thesis [Parallel hyperparameter optimization of spiking neural networks]().
## Getting started
The thesis is accessible at (_will be available once published_):
*[HAL]()
## Summary
>Les Réseaux de Neurones Artificiels (RNAs) sont une technique d'apprentissage machine devenue aujourd'hui incontournable, permettant de résoudre certaines tâches cognitives complexes par un apprentissage automatique. Depuis ces trois dernières décennies, les RNAs ont connu de nombreuses avancées majeures. Ces avancées ont permis le développement de la reconnaissance d'images, des modèles de langage géants ou la conversion texte-image. Indéniablement, les RNAs sont devenus un outil précieux ayant mené depuis 2020, au boom des modèles générationnels. Cependant, certaines barrières pourraient freiner l'intérêt pour ces modèles. Notamment la fin de la loi de Moore, dû aux limites physiques atteintes par les transistors. Mais aussi, tandis que la recherche s'est longtemps concentrée sur les performances prédictives des RNAs, d'autres aspects ont été négligés. C'est le cas de l'efficacité énergétique, mais également de la robustesse, de la sécurité, de l'interprétabilité, de la transparence, etc [(1)](https://futureoflife.org/open-letter/pause-giant-ai-experiments/). Il faut donc aller au-delà des architectures de von Neumann afin de réduire l'empreinte énergétique, et l'approche neuromorphique est un candidat de rupture sérieux utilisant le biomimétisme du cerveau via des Réseaux de Neurones à Impulsions (RNIs).
>Malheureusement, les RNIs peinent à surpasser les performances des RNAs. Les RNIs étant plus récents, et donc moins étudiés, une meilleure approche de leur conception pourrait permettre d'allier performances et faible coût énergétique. C'est pourquoi la conception automatique des RNIs est étudiée dans cette thèse. L'intérêt est notamment porté sur l'Optimisation des HyperParamètres (OHP). Ainsi, nous étudions l'impact de l'OHP sur les RNIs, et l'impact des RNIs sur l'OHP. Le but étant d'améliorer les algorithmes utilisés et de mieux comprendre le comportement des RNIs au regard de leurs hyperparamètres.
To make it easy for you to get started with GitLab, here's a list of recommended next steps.
