Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Incrementally Expanding Environment in Deep Reinforcement Learning
KTH, School of Engineering Sciences (SCI).
KTH, School of Engineering Sciences (SCI).
2018 (English)Independent thesis Basic level (degree of Bachelor), 10 credits / 15 HE creditsStudent thesisAlternative title
Stegvis expanderande miljö inom djupt förstärkande inlärning (Swedish)
Abstract [sv]

Tillräckligt snabb konvergens till optimala strategier är ett mål inom maskininlärning och speciellt förstärkande inlärning. Realtidslösningar till komplexa inlärningsproblem behövs för att expandera fältet till nya områden där maskininlärning tidigare varit en omöjlighet. I denna rapport introducerar vi en ny metod för att träna djupa Q-learning agenter i en miljö vars storlek är skalbar, i hopp om att förkorta inlärningstiden. I denna metod börjar agenten i en mycket liten miljö där den snabbt kan utforska olika situationer på en liten skala och lära sig att hantera dem. Miljön expanderar sedan bitvis när agenten har lärt sig miljön till en förbestämd standard. På sådant sätt kommer agenten inte att behövs lära om sig miljön utan kan anpassa sina kunskaper från ett mindre sammanhang till ett större. Jämförelsen mellan den Stegvis Expanderade Miljö (IEE) metoden samt den konventionella metoden att från början ge hela miljön till agenten visade att metoderna hade jämförbar prestanda i vissa fall medan IEE metoden visade sig överlägsen i andra situationer. Speciellt i scenarier med en högre learning rate så förbättrade vår metod sin strategi snabbare. Den vanliga metoden presterade aldrig bättre än IEE metoden. Vi drog slutsatsen att vår föreslagna metod är överlägsen den vanliga metoden, den är robustare i val av parametrar och presterar bättre eller lika bra i alla testade fall.

Abstract [en]

The very long times required to converge to optimal policies is a problem affecting machine learning and reinforcement learning in particular. Real-time solution on complex learning problems are necessary to expand the field into new domains where machine learning has previously been unfeasible. In this paper we introduce a novel method for training deep q-learning agents in an environment where the size can be dynamically scaled, in order to improve learning time. In this framework the agent starts in a very small environment where it can quickly experience different situations in a small scale and learn to handle them properly. As the agent learns the environment enough to reach certain predefined performance goals, the environment is expanded to increase complexity. The agent should then not have to relearn the environment completely, but simply adapt to the larger environment. This Incrementally Expanding Environment (IEE) method was compared to the conventional, deep q-learning method of training the agent on the full environment size from the beginning. Results showed that in some situations the methods performed identically, where in some situations the IEE method performed better. Particularly in scenarios with higher learning rates our framework improved its policy noticeably faster than the conventional method. The conventional method also notably never performed better than the IEE method. We thus conclude that the proposed framework is superior the the conventional one, being more robust to parameter choice and performing as well or better in all observed cases.

Place, publisher, year, edition, pages
2018. , p. 14
Series
TRITA-SCI-GRU ; 2018-131
National Category
Engineering and Technology
Identifiers
URN: urn:nbn:se:kth:diva-230754OAI: oai:DiVA.org:kth-230754DiVA, id: diva2:1219174
Supervisors
Examiners
Available from: 2018-06-15 Created: 2018-06-15 Last updated: 2018-06-15Bibliographically approved

Open Access in DiVA

fulltext(730 kB)12 downloads
File information
File name FULLTEXT01.pdfFile size 730 kBChecksum SHA-512
c151aae6080683ef17a23ed41f6dff8b4ee54ccfe5b2462cfd0c6357a633fac8f047f97ce733f16dc7fadbe11fa2f6dac00c569df707be45c33c377a9ac3e94a
Type fulltextMimetype application/pdf

By organisation
School of Engineering Sciences (SCI)
Engineering and Technology

Search outside of DiVA

GoogleGoogle Scholar
Total: 12 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 20 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf