Change search
ReferencesLink to record
Permanent link

Direct link
Virtual infrastructures for computational science: software and architectures for distributed job and resource management
Umeå University, Faculty of Science and Technology, Department of Computing Science.
2011 (English)Doctoral thesis, comprehensive summary (Other academic)Alternative title
Virtuella infrastrukturer för beräkningsvetenskap : programvaror och arkitekturer för distribuerad jobb- och resurshantering (Swedish)
Abstract [en]

In computational science, the scale of problems addressed and the resolution of solu- tions achieved are often limited by the available computational capacity. The current methodology of scaling computational capacity to large scale (i.e. larger than individ- ual resource site capacity) includes aggregation and federation of distributed resource systems. Regardless of how this aggregation manifests, scaling of scientific compu- tational problems typically involves (re)formulation of computational structures and problems to exploit problem and resource parallelism. Efficient parallelization and scaling of scientific computations to large scale is difficult and further complicated by a number of factors introduced by resource aggregation, e.g., resource heterogene- ity and coupling of computational methodology. Scaling complexity severely impacts computation enactment and necessitates the use of mechanisms that provide higher abstractions for management of computations in distributed computing environments.This work addresses design and construction of virtual infrastructures for scientific computation that abstract computation enactment complexity, decouple computation specification from computation enactment, and facilitate large-scale use of compu- tational resource systems. In particular, this thesis discusses job and resource man- agement in distributed virtual scientific infrastructures intended for Grid and Cloud computing environments. The main area studied is Grid computing, which is ap- proached using Service-Oriented Computing and Architecture methodology. Thesis contributions discuss both methodology and mechanisms for construction of virtual infrastructures, and address individual problems such as job management, application integration, scheduling job prioritization, and service-based software development.I addition to scientific publications, this work also makes contributions in the form of software artifacts that demonstrate the concepts discussed. The Grid Job Manage- ment Framework (GJMF) abstracts job enactment complexity and provides a range of middleware-agnostic job submission, control, and monitoring interfaces. The FSGrid framework provides a generic model for specification and delegation of resource allo- cations in virtual organizations, and enacts allocations based on distributed fairshare job prioritization. Mechanisms such as these decouple job and resource management from computational infrastructure systems and facilitate the construction of scalable virtual infrastructures for computational science.

Abstract [sv]

Inom beräkningsvetenskap begränsar ofta mängden tillgänglig beräkningskraft både storlek på problem som kan ansättas såväl som kvalitet på lösningar som kan uppnås. Metodik för skalning av beräkningskapacitet till stor skala (dvs större än kapaciteten hos enskilda resurscentras) baseras för närvarande på aggregering och federation av distribuerade beräkningsresurser. Oavsett hur denna resursaggregering tar sig uttryck tenderar skalning av vetenskapliga beräkningar till storskalig nivå att inkludera omformulering av problemställningar och beräkningsstrukturer för att bättre utnyttja problem- och resursparallellism. Effektiv parallellisering och skalning av vetenskapliga beräkningar är svårt och kompliceras ytterligare av faktorer som medföljer resursaggregering, t.ex. heterogeneitet i resursmiljöer och beroenden i programmeringsmodeller och beräkningsmetoder. Detta utbytesförhållande illustrerar komplexiteten i utförande av beräkningar och behovet av mekanismer som erbjuder högre abstraktionsnivåer för hantering av beräkningar i distribuerade beräkningsmiljöer.Denna avhandling diskuterar design och konstruktion av virtuella beräkningsinfrastrukturer som abstraherar komplexitet i utförande av beräkningar, frikopplar design av beräkningar från utförande av beräkningar samt underlättar storskalig användning av beräkningsresurser för vetenskapliga beräkningar. I synnerhet behandlas jobb- och resurshantering i distribuerade virtuella vetenskapliga infrastrukturer avsedda för Grid och Cloud computing miljöer. Det huvudsakliga området för avhandlingen är Grid computing, vilket adresseras med service-orienterad beräknings- och arkitekturmetodik. Arbetet diskuterar metodik och mekanismer för konstruktion av virtuella beräkningsinfrastrukturer samt gör bidrag inom enskilda områden som jobbhantering, applikationsintegrering, jobbprioritering och service-baserad programvaruutveckling.Utöver vetenskapliga publikationer bidrar detta arbete också med bidrag i form av programvarusystem som illustrerar de metoder som diskuteras. The Grid Job Management Framework (GJMF) abstraherar komplexitet i hantering av beräkningsjobb och erbjuder en uppsättning middleware-agnostiska gränssnitt för körning, kontroll och övervakning av beräkningsjobb i distribuerade beräkningsmiljöer. FSGrid erbjuder en generisk modell för specifikation och delegering av resurstilldelning i virtuella organisationer och grundar sig på distribuerad rättvisebaserad jobbprioritering. Mekanismer som dessa frikopplar jobb- och resurshantering från fysiska infrastruktursystem samt underlättar konstruktion av skalbara virtuella infrastrukturer för beräkningsvetenskap.

Place, publisher, year, edition, pages
Umeå: Institutionen för datavetenskap, Umeå universitet , 2011. , 238 p.
Series
Report / UMINF, ISSN 0348-0542 ; 11.02
Identifiers
URN: urn:nbn:se:umu:diva-42428ISBN: 978-91-7459-194-1OAI: oai:DiVA.org:umu-42428DiVA: diva2:409249
Public defence
2011-05-05, MIT-huset, MA121, Umeå universitet, Umeå, 13:30
Opponent
Supervisors
Available from: 2011-04-11 Created: 2011-04-07 Last updated: 2011-04-29Bibliographically approved
List of papers
1. Designing service-based resource management tools for a healthy grid ecosystem
Open this publication in new window or tab >>Designing service-based resource management tools for a healthy grid ecosystem
2008 (Swedish)In: Parallel processing and applied mathematics: 7th International Conference on Parallel Processing and Applied Mathematics, Springer-Verlag , 2008, 259-270 p.Conference paper (Refereed)
Abstract [en]

We present an approach for development of Grid resource management tools, where we put into practice internationally established high-level views of future Grid architectures. The approach addresses fundamental Grid challenges and strives towards a future vision of the Grid where capabilities are made available as independent and dynamically assembled utilities, enabling run-time changes in the structure, behavior, and location of software. The presentation is made in terms of design heuristics, design patterns, and quality attributes, and is centered around the key concepts of co-existence, composability, adoptability, adaptability, changeability, and interoperability. The practical realization of the approach is illustrated by five case studies (recently developed Grid tools) high-lighting the most distinct aspects of these key concepts for each tool. The approach contributes to a healthy Grid ecosystem that promotes a natural selection of “surviving” components through competition, innovation, evolution, and diversity. In conclusion, this environment facilitates the use and composition of components on a per-component basis.

Place, publisher, year, edition, pages
Springer-Verlag, 2008
Series
, Lecture Notes in Computer Science, ISSN 0302-9743 ; 4967
National Category
Computer Science
Research subject
Computing Science
Identifiers
urn:nbn:se:umu:diva-9614 (URN)10.1007/978-3-540-68111-3_28 (DOI)
Conference
7th International Conference on Parallel Processing and Applied Mathematics, 2008
Available from: 2008-05-05 Created: 2008-05-05 Last updated: 2011-04-11Bibliographically approved
2. Dynamic and transparent service compositions techniques for service-oriented grid architectures
Open this publication in new window or tab >>Dynamic and transparent service compositions techniques for service-oriented grid architectures
2008 (English)In: Integrated research in Grid computing / [ed] S. Gorlatch, P Fragopoulou and T. Priol, Crete University Press , 2008, 323-334 p.Chapter in book (Refereed)
Abstract [en]

With the introduction of the Service-Oriented Architecture design paradigm, service composition has become a central methodology for developing Grid software. We present an approach to Grid software development consisting of architectural design patterns for service de-composition and service re-composition. The patterns presented can each be used individually, but provide synergistic effects when combined as described in a unified framework. Software design patterns are employed to provide structure in design for service-based software development. Service APIs and immutable data wrappers are used to simplify service client development and isolate service clients from details of underlying service engine architectures. The use of local call structures greatly reduces inter-service communication overhead for co-located services, and service API factories are used to make local calls transparent to service client developers. Light-weight and dynamically replaceable plug-ins provide structure for decision support and integration points. A dynamic configuration scheme provides coordination of service efforts and synchronization of service interactions in a user-centric manner. When using local calls and dynamic configuration for creating networks of cooperating services, the need for generic service monitoring solutions becomes apparent and is addressed by service monitoring interfaces. We present these techniques along with their intended use in the context of software development for service-oriented Grid architectures.

Place, publisher, year, edition, pages
Crete University Press, 2008
Identifiers
urn:nbn:se:umu:diva-9613 (URN)
Available from: 2008-05-05 Created: 2008-05-05 Last updated: 2011-04-11Bibliographically approved
3. Designing general, composable, and middleware-independent Grid infrastructure tools for multi-tiered job management
Open this publication in new window or tab >>Designing general, composable, and middleware-independent Grid infrastructure tools for multi-tiered job management
Show others...
2007 (English)In: Towards Next Generation Grids / [ed] T. Priol and M. Vaneschi, Springer-Verlag , 2007, 175-184 p.Conference paper (Refereed)
Abstract [en]

We propose a multi-tiered architecture for middleware-independent Grid job management. The architecture consists of a number of services for well-defined tasks in the job management process, offering complete user-level isolation of servicecapabilities, multiple layers of abstraction, control, and fault tolerance. The middleware abstraction layer comprises components for targeted job submission, job control and resource discovery. The brokered job submission layer offers a Grid view on resources, including functionality for resource brokering and submission of jobs to selected resources. The reliable job submission layer includes components for fault tolerant execution of individual jobs and groups of independentjobs, respectively. The architecture is proposed as a composable set of tools rather than a monolithic solution, allowing users to select the individual components of interest. The prototype presented is implemented using the Globus Toolkit 4, integrated with the Globus Toolkit 4 and NorduGrid/ARC middlewares and based on existing and emerging Grid standards. A performance evaluation reveals that the overhead for resource discovery, brokering, middleware-specific format conversions, job monitoring, fault tolerance, and management of individual and groups of jobs is sufficiently small to motivate the use of the framework.

Place, publisher, year, edition, pages
Springer-Verlag, 2007
Keyword
Grid Job management infrastructure, standards-based architecture, fault-tolerance, middleware-independence, Grid ecosystem
Identifiers
urn:nbn:se:umu:diva-9305 (URN)978-0-387-72497-3 (ISBN)
Conference
CoreGRID Symposium 2007
Available from: 2008-05-05 Created: 2008-05-05 Last updated: 2012-02-06
4. GJMF - a composable service-oriented Grid job management framework
Open this publication in new window or tab >>GJMF - a composable service-oriented Grid job management framework
2013 (English)In: Future generations computer systems, ISSN 0167-739X, Vol. 29, no 1, 144-157 p.Article in journal (Refereed) Published
Abstract [en]

We investigate best practices for Grid software design and development, and propose a composable, loosely coupled Service-Oriented Architecture for Grid job management. The architecture focuses on providing a transparent Grid access model for concurrent use of multiple Grid middlewares and aims to decouple Grid applications from Grid middlewares and infrastructure. The notion of an ecosystem of Grid infrastructure components is extended, and Grid job management software design is discussed in this context. Non- intrusive integration models and abstraction of Grid middleware function- ality through hierarchical aggregation of autonomous Grid job management services are emphasized, and service composition techniques facilitating this process are explored. A proof-of-concept implementation of the architecture is presented along with a discussion of architecture implementation details and trade-offs introduced by the service composition techniques used.

Keyword
Grid computing, Grid job management, Grid ecosystem
National Category
Computer and Information Science
Identifiers
urn:nbn:se:umu:diva-42412 (URN)10.1016/j.future.2012.04.004 (DOI)
Available from: 2011-04-07 Created: 2011-04-07 Last updated: 2013-01-02Bibliographically approved
5. Impact of service overhead on service-oriented Grid architectures
Open this publication in new window or tab >>Impact of service overhead on service-oriented Grid architectures
2011 (English)Conference paper (Refereed)
Abstract [en]

Grid computing applications and infrastructures build heavily on Service-Oriented Computing development methodology and are often realized as Service-Oriented Ar- chitectures. Current Service-Oriented Architecture methodology renders service components as Web Services, and suffers per- formance limitations from Web Service overhead. The Grid Job Management Framework (GJMF) is a flexible Grid in- frastructure and application support component realized as a loosely coupled network of Web Services that offers a range of abstractive and platform independent interfaces for middleware- agnostic Grid job submission, monitoring, and control. In this paper we a present a performance evaluation aimed to characterize the impact of service overhead on Grid Service- Oriented Architectures and evaluate the efficiency of the GJMF architecture and optimization mechanisms designed to mediate impact of Web Service overhead on architecture performance.

Identifiers
urn:nbn:se:umu:diva-42419 (URN)
Note
Submitted for conference publication, 2011.Available from: 2011-04-07 Created: 2011-04-07 Last updated: 2012-01-04Bibliographically approved
6. Empowering a flexible application portal with a SOA-based Grid job management framework
Open this publication in new window or tab >>Empowering a flexible application portal with a SOA-based Grid job management framework
Show others...
2009 (English)In: Applied Parallel Computing (PARA 08): State of art in scientific computing / [ed] A.C. Elster et al., Springer , 2009Conference paper (Refereed)
Place, publisher, year, edition, pages
Springer, 2009
Series
, Lecture notes in computer science, ISSN 0302-9743 ; 6127
National Category
Computer Science
Identifiers
urn:nbn:se:umu:diva-40381 (URN)
Conference
PARA 08
Available from: 2011-02-22 Created: 2011-02-22 Last updated: 2012-01-04Bibliographically approved
7. Decentralized scalable fairshare scheduling
Open this publication in new window or tab >>Decentralized scalable fairshare scheduling
2013 (English)In: Future generations computer systems, ISSN 0167-739X, Vol. 29, no 1, 130-143 p.Article in journal (Refereed) Published
Abstract [en]

This work addresses Grid fairshare allocation policy enforcement and presents Aequus, a decentralized system for Grid-wide fairshare job prioritization. The main idea of fairshare scheduling is to prioritize users with regard to predefined resource allocation quotas. The presented system builds on three contributions: a flexible tree-based policy model that allows delegation of policy definition, a job prioritization algorithm based on local enforcement of distributed fairshare policies, and a decentralized architecture for non-intrusive integration with existing scheduling systems. The system supports organization of users in virtual organizations and divides usage policies into local and global policy components that are defined by resource owners and virtual organizations. The architecture realization is presented in detail along with an evaluation of the system behavior in an emulated environment. In the evaluation, convergence noise types (mechanisms counteracting policy allocation convergence) are characterized and quantified, and the system is demonstrated to meet scheduling objectives and perform scalably under realistic operating conditions.

Place, publisher, year, edition, pages
Elsevier, 2013
Keyword
Grid scheduling, Fairshare scheduling, Grid allocation policy enforcement
National Category
Computer Science
Identifiers
urn:nbn:se:umu:diva-40492 (URN)10.1016/j.future.2012.06.001 (DOI)
Available from: 2011-02-24 Created: 2011-02-24 Last updated: 2013-09-19Bibliographically approved
8. Increasing flexibility and abstracting complexity in service-based Grid and cloud software
Open this publication in new window or tab >>Increasing flexibility and abstracting complexity in service-based Grid and cloud software
2011 (English)In: Proceedings of CLOSER 2011 - International Conference on Cloud Computing and Services Science / [ed] F. Leyman, I Ivanov, M. van Sinderen and B. Shishkov, SciTePress , 2011, 240-249 p.Conference paper (Refereed)
Place, publisher, year, edition, pages
SciTePress, 2011
National Category
Computer Science
Identifiers
urn:nbn:se:umu:diva-40369 (URN)
Conference
CLOSER 2011
Available from: 2011-02-22 Created: 2011-02-22 Last updated: 2012-01-04

Open Access in DiVA

fulltext(854 kB)838 downloads
File information
File name FULLTEXT02.pdfFile size 854 kBChecksum SHA-512
d052ee1972912152fb62bf291c204e07fd7a19773d344337d748a19cd6d2de0368e80272b0cebc398921cac99b729567a10eebbd2dc6b478fe5677f6b9be3175
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Östberg, Per-Olov
By organisation
Department of Computing Science

Search outside of DiVA

GoogleGoogle Scholar
Total: 838 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

Total: 1050 hits
ReferencesLink to record
Permanent link

Direct link