Digitala Vetenskapliga Arkivet

Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Comparative Analysis of Knowledge Distillation Methods for Convolutional Neural Networks in Image Classification
KTH, School of Electrical Engineering and Computer Science (EECS).
KTH, School of Electrical Engineering and Computer Science (EECS).
2025 (English)Independent thesis Basic level (degree of Bachelor), 10 credits / 15 HE creditsStudent thesisAlternative title
Jämförande analys av kunskapsdestillationsmetoder för konvolutionella neurala nätverk inom bildklassificering (Swedish)
Abstract [en]

In the past few years, artificial neural networks (ANNs) have achieved impressive results in numerous tasks. In image classification, convolutional neural networks (CNNs) perform well by applying convolutional filters to extract features of the images. One lightweight CNN architecture is ShuffleNet V2, which can be scaled to trade off model size and accuracy. However, higher-accuracy variants can be too large for deployment in resource-constrained environments. This report investigates whether a larger, pretrained ShuffleNet V2 teacher model can transfer knowledge to a smaller ShuffleNet V2 student via knowledge distillation (KD). We compare the knowledge distillation described by Hinton et al. (HKD), decoupled knowledge distillation (DKD), distillation-oriented training (DOT), and attention transfer (AT) — using a subset of the ImageNet-1k dataset. Although the average top-1 accuracies across five runs vary, every KD method’s single best run exceeds the best top-1 accuracy of the same-sized, non-distilled baseline (73.32%). Among students, DKD achieved the highest maximum top-1 score (74.08%), followed in order by AT (73.98%), HKD (73.88%), and DOT (73.72%). The limited hyperparameter search (due to compute constraints) indicates that more extensive tuning may yield improvements, and additional runs can generate statistically significant results. These efforts should be the focus of future work.

Abstract [sv]

Under de senaste åren har artificiella neurala nätverk (ANNs) uppnått imponerande resultat i en mängd olika uppgifter. Inom bildigenkänning presterar konvolutionella neurala nätverk (CNNs) väl genom att applicera konvolutionsfilter som extraherar egenskaper ur bilder. En lättviktig CNN-arkitektur är Shuffle-Net V2, som kan skalas om för att balansera mellan modellstorlek och träffsäkerhet. Däremot kan varianter med hög träffsäkerhet vara för stora för att användas i resursbegränsade miljöer. Denna rapport undersöker huruvida en större, förtränad ShuffleNet V2-lärarmodell kan överföra kunskap till en mindre ShuffleNet V2-studentmodell via kunskapsdestillering (KD). Vi jämför den kunskapsdestillering som beskrivs av Hinton et al. (HKD), decoupled knowledge distillation (DKD), distillationoriented training (DOT) och attention transfer (AT) — med hjälp av en delmängd av ImageNet-1K-datasetet. Även om den genomsnittliga top-1-träffsäkerheten under fem körningar varierar, överträffar varje KD-metods bästa enskilda körning den bästa top-1-träffsäkerheten hos en icke-destillerad baseline-modell av samma storlek (73,32%). Bland studentmodellerna uppnådde DKD den högsta maximala top-1-precisionen (74,08%), följt av AT (73,98%), HKD (73,88%) och DOT (73,72%). Den begränsade hyperparameteroptimeringen (på grund av begränsade beräkningsresurser) antyder att ytterligare finjusteringar kan leda till förbättringar och fler körningar kan ge statistiskt signifikanta resultat. Dessa aspekter bör prioriteras i framtida arbete.

Place, publisher, year, edition, pages
2025. , p. 44
Series
TRITA-EECS-EX ; 2025:299
Keywords [en]
Knowledge distillation, ShuffleNet V2, Decoupled knowledge distillation, Distillation-oriented trainer, Attention transfer
Keywords [sv]
Kunskapsdestillering, ShuffleNet V2, Decoupled knowledge distillation, Distillation-oriented trainer, Attention transfer
National Category
Computer and Information Sciences
Identifiers
URN: urn:nbn:se:kth:diva-367635OAI: oai:DiVA.org:kth-367635DiVA, id: diva2:1985704
Supervisors
Examiners
Available from: 2025-08-05 Created: 2025-07-27 Last updated: 2025-08-05Bibliographically approved

Open Access in DiVA

fulltext(3442 kB)519 downloads
File information
File name FULLTEXT01.pdfFile size 3442 kBChecksum SHA-512
e49bd0dbeada5bb6ea3ed94eb886ae80970a9de3097e3a6417a749900eafcaf1aebef021d692d7f69baa1c56bc3eb693ffcd3e83a84a73ddad20d09927b96422
Type fulltextMimetype application/pdf

By organisation
School of Electrical Engineering and Computer Science (EECS)
Computer and Information Sciences

Search outside of DiVA

GoogleGoogle Scholar
Total: 521 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 854 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf