Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Aspects of adaptive mesh refinement in the spectral element method
KTH, School of Engineering Sciences (SCI), Mechanics. (Linné flow centre)ORCID iD: 0000-0002-1724-0188
2019 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

This thesis deals with the improvement of the efficiency of numerical simulations in computational fluid dynamics. Some of the limitations of current algorithms on future exascale supercomputers are addressed with the main goal of using adaptive mesh refinement for the simulation of turbulent and three-dimensional flows. The comparison between two different error estimators is also investigated.The framework considered all throughout this thesis is Nek5000, a highly parallel code based on the spectral element method.

First, the strong parallel scaling of Nek5000 is studied on three petascale machines in order to assess the scalability of the code and identify the current bottlenecks. It is found that the strong scaling limit ranges between 5,000 and 220,000 gridpoints per core depending on the machine and the case. The need for synchronized and low latency communication for fast computations is confirmed. It is also shown that, on a single core, Nek5000 is memory limited rather than compute limited.

Then, a new method for the coarse grid part of the preconditioner for the pressure equation is implemented, which represents a significant improvement compared to existing solvers. We use an algebraic multigrid method from the Hypre library to perform the setup and solver parts on the fly and fully in parallel. The setup phase only amounts to a few percents of the wall clocktime for a single timestep of the solver and is therefore negligible; this is a requirement for adaptive simulations, where the setup must be performed after each adaptation. In addition, the new solver is shown to be suitable for large and complex simulations in three dimensions.

Finally, the main objective of this work is to perform simulations with adaptive mesh refinement to achieve error control and efficient use of computational resources. We develop adjoint error estimators based on the dual-weighted residuals method and also consider more traditional a posteriori error indicators for comparison. Adaptive simulations are performed on test cases of increasing complexity: the steady flow in a lid-driven cavity in 2D and 3D, the steady flow past a 2D cylinder and the turbulent flow inside a constricted periodic channel in 3D. It is concluded that adjoint error estimators are preferred for flows with a strongly convective nature, while the more straightforward spectral error indicators are sufficient otherwise. Moreover, we perform an adaptive simulation of the turbulent flow past a NACA4412 profile at Reynolds number Re = 850,000, using the spectral error indicators. In comparison to cases with a fixed mesh, the aspect ratio of grid elements in the far field remains low, the convergence of the pressure solver is significantly sped up and the much larger computational domain allows true free-stream boundary conditions and thus makes the results less dependent on the boundary conditions.

Abstract [sv]

Den här avhandlingen syftar till att utveckla metoder för att öka effektiviteten av strömningsmekaniska simuleringar. Arbetet visar också vissa begränsningarna hos nuvarande algoritmer om de ska användas på framtida superdatorer i exaskala. Adaptiv nätförfining används i den här avhandlingen för att simulera tredimensionella turbulenta strömningar. Dessutom jämförs två stycken olika metoder för att uppskatta felet av lösningen till Navier–Stokes ekvationer. Hela arbetet baseras på koden Nek5000, som använder den spektrala elementmetoden. Inledningsvis undersöker vi den starka parallela skalbarheten av Nek5000 på tre superdatorer i petaskala för att bedöma kodens skalbarhet och identifiera befintliga flaskhalsar. Det visas att den starka skalningen per kärna för koden är begränsad till mellan 5, 000 och 220, 000 frihetsgrader, beroende på superdator och fall. Det är även bekräftat att synkroniserad kommunikation medliten fördröjning är nödvändigt för att uppnå effektiva strömningsmekaniska simuleringar. Nek5000 är minnesbegränsat på en enskild kärna. Sedan implementeras en ny metod som förbättrar förkonditioneringen av tryckequationen på den grova delen av nätet. Vi använder en algebraisk flernätsmetod från Hypre för att automatiskt sätta upp lösningsmatriserna och lösa algoritmen på parallella datasystemer. Tiden som krävs för att initiera ett tidsteg är försumbar jämfört med beräkningstiden; det här är ett krav för adatiptiva simuleringar som behöver initiera efter varje adaption av nätet.Dessutom bekräftas det att den nya lösaren är lämplig för att simulera stora och komplexa tredimensionella strömningsproblem.

Huvudsyftet med arbetet att utföra simuleringar med adaptiva nätförfiningar för att kunna kontrollera felet på lösningen och att effektivt använda beräkningsresurser. Vi utvecklar adjunkta feluppskattningar baserat på dubbelviktade residualmetoden och jämför också med traditionella metoder som uppskattar felet av lösningen i efterhand (eng. spectral error indicators). Adaptiva nätförfiningssimuleringar genomförs på testfall som blir mer och mer komplexa: en stationär strömning i en kavitet med en rörlig rand (eng. lid-driven cavity) både två- och tredimensionellt, stationärt flöde kring en tvådimensionell cylinder och turbulent tredimensionell kanalströmningen i en periodisk domän.Slutsatsen är att adjunkta feluppskattningar föredras för flöden där konvektion dominerar, medan traditionella metoder för att uppskatta felet är tillräckligt bra för alla andra fall. Vi genomför även en adaptiv simulering av det turbulenta flödet över en NACA4412 vingprofil vid ett Reynoldstal Re = 850, 000, där vi använder traditionella metoder för att uppskatta felet. Resultaten från simuleringen har jämförts med simuleringar på ett nät utan adaptiv förfining, slutsatsen är en snabbare konvergens för att lösa ekvationen för trycket och att resultaten blir mindre beroende av randvillkoren på grund av en större domän,som tillåter korrekta friströmsrandvillkor.

Place, publisher, year, edition, pages
KTH Royal Institute of Technology, 2019. , p. 206
Series
TRITA-MEK, ISSN 0348-467X ; 2019:28
Keywords [en]
Error estimators, mesh refinement, adaptivity, spectral element method, algebraic multigrid method, NACA4412, periodic hill case.
National Category
Fluid Mechanics and Acoustics
Research subject
Engineering Mechanics
Identifiers
URN: urn:nbn:se:kth:diva-251638ISBN: 978-91-7873-223-4 (print)OAI: oai:DiVA.org:kth-251638DiVA, id: diva2:1316171
Public defence
2019-06-12, F3, Kungliga Tekniska Högskolan, Lindstedtsvägen 26, Stockholm, 10:15 (English)
Opponent
Supervisors
Note

QC20190517

Available from: 2019-05-16 Created: 2019-05-16 Last updated: 2019-05-17Bibliographically approved
List of papers
1. On the strong scaling of the spectral element solver Nek5000 on petascale systems
Open this publication in new window or tab >>On the strong scaling of the spectral element solver Nek5000 on petascale systems
Show others...
2016 (English)In: Proceedings of the 2016 Exascale Applications and Software Conference (EASC2016): April 25-29 2016, Stockholm, Sweden, Association for Computing Machinery (ACM), 2016, article id a5Conference paper, Published paper (Refereed)
Abstract [en]

The present work is targeted at performing a strong scaling study of the high-order spectral element uid dynamics solver Nek5000. Prior studies such as [5] indicated a recommendable metric for strong scalability from a theoretical viewpoint, which we test here extensively on three parallel machines with different performance characteristics and interconnect networks, namely Mira (IBM Blue Gene/Q), Beskow (Cray XC40) and Titan (Cray XK7). The test cases considered for the simulations correspond to a turbulent ow in a straight pipe at four different friction Reynolds numbers Reτ = 180, 360, 550 and 1000. Considering the linear model for parallel communication we quantify the machine characteristics in order to better assess the scaling behaviors of the code. Subsequently sampling and profiling tools are used to measure the computation and communication times over a large range of compute cores. We also study the effect of the two coarse grid solvers XXT and AMG on the computational time. Super-linear scaling due to a reduction in cache misses is observed on each computer. The strong scaling limit is attained for roughly 5000 - 10; 000 degrees of freedom per core on Mira, 30; 000 - 50; 0000 on Beskow, with only a small impact of the problem size for both machines, and ranges between 10; 000 and 220; 000 depending on the problem size on Titan. This work aims at being a reference for Nek5000 users and also serves as a basis for potential issues to address as the community heads towards exascale supercomputers.

Place, publisher, year, edition, pages
Association for Computing Machinery (ACM), 2016
Series
ACM International Conference Proceeding Series
Keywords
Benchmarking, Computational fluid dynamics, Nek5000, Scaling, Degrees of freedom (mechanics), Reynolds number, Supercomputers, Computational time, Interconnect networks, Parallel communication, Parallel machine, Performance characteristics, Spectral element, Application programs
National Category
Mechanical Engineering
Identifiers
urn:nbn:se:kth:diva-207506 (URN)10.1145/2938615.2938617 (DOI)2-s2.0-85014776002 (Scopus ID)9781450341226 (ISBN)
Conference
2016 Exascale Applications and Software Conference, EASC 2016, Stockholm, Sweden, 25 April 2016 through 29 April 2016
Funder
Swedish e‐Science Research Center
Note

QC 20170814

Available from: 2017-06-07 Created: 2017-06-07 Last updated: 2019-05-16Bibliographically approved
2. Towards adaptive mesh refinement for the spectral element solver Nek5000
Open this publication in new window or tab >>Towards adaptive mesh refinement for the spectral element solver Nek5000
Show others...
2017 (English)Report (Refereed)
Abstract [en]

Hypre, a library for linear algebra, is used to replace a Matlab code for performing the setup step of an Algebraic Multigrid Method (AMG). The AMG method is used to compute part of the preconditioner in Nek5000, a code for Computational Fluid Dynamics based on the spectral element method. However, the solution of the AMG problem is not performed via Hypre but by Nek5000’s internal solver. The new AMG setup is shown to be faster by at least one order of magnitude, while it does not significantly impact the efficiency of the AMG solver, as is shown from its application to relevant test cases.

Publisher
p. 6
National Category
Fluid Mechanics and Acoustics
Research subject
Engineering Mechanics
Identifiers
urn:nbn:se:kth:diva-217497 (URN)
Note

QC 20171123

Available from: 2017-11-13 Created: 2017-11-13 Last updated: 2019-05-16Bibliographically approved
3. Performance of preconditioners for large-scale simulations using Nek5000
Open this publication in new window or tab >>Performance of preconditioners for large-scale simulations using Nek5000
Show others...
2019 (English)Conference paper, Published paper (Refereed)
Abstract [en]

BoomerAMG, the algebraic multigrid solver from the hypre library, is used to solve a coarse grid problem which is part of the preconditioning strategy for thepressure equation arising from the numerical resolution of the Navier–Stokes equations. A set of optimal parameters for the setup phase is determined and used for selected strong scaling tests on two different supercomputers, namely Mira and Hazel Hen, on up to 131, 072 compute cores. The results are compared to an existing algebraic multigrid solver, designed specifically for the coarse gridproblem at hand. It is shown that the BoomerAMG solver is fast and scalable, and that performance depends on the computer architecture. The test cases considered are the turbulent flow past a NACA4412 airfoil and the turbulent flow inside wire-tapped pin bundles.

National Category
Fluid Mechanics and Acoustics
Research subject
Engineering Mechanics
Identifiers
urn:nbn:se:kth:diva-251632 (URN)
Conference
ICOSAHOM18 conference, July 9 - 13, 2018, London, United Kingdom
Note

QC 20190520

Available from: 2019-05-16 Created: 2019-05-16 Last updated: 2019-05-20Bibliographically approved
4. Non-conforming elements in Nek5000: Pressure preconditioning and parallel performance
Open this publication in new window or tab >>Non-conforming elements in Nek5000: Pressure preconditioning and parallel performance
Show others...
2019 (English)Conference paper, Published paper (Refereed)
Abstract [en]

Adaptive mesh refinement (AMR) is an important component of modern numerical solvers, as it allows to control the computational error during the simulation, increasing the reliability of the numerical modelling and giving the possibility to study a broad range of different phenomena even without knowing the physics a priori. In this work we present selected aspects of the implementation and parallel performance of a new h−type AMR framework developed for the high-order CFD solver Nek5000; the development was done within the ExaFLOW EU project. We utilise in this case the natural domain decomposition inherent to the spectral element method (SEM), which constitutes the main source of parallelism and provides meshing flexibility that can be exploited in AMR. We use standard libraries for parallel mesh management (p4est) and partitioning (ParMetis) and focus on developing efficient preconditioners for the pressure problem solved on non-conforming meshes. Two different approaches are considered: an additive overlapping Schwarz and a hybrid Schwarz-multigrid method.The strong scaling is shown on the example of the simulation of the turbulent flow around a NACA4412 wing section at Rec = 200, 000.

National Category
Fluid Mechanics and Acoustics
Research subject
Engineering Mechanics
Identifiers
urn:nbn:se:kth:diva-251633 (URN)
Conference
COSAHOM18 conference, July 9 - 13, 2018, London, United Kingdom
Note

QC 20190521

Available from: 2019-05-16 Created: 2019-05-16 Last updated: 2019-05-21Bibliographically approved
5. Adaptive mesh refinement for steady flows in Nek5000
Open this publication in new window or tab >>Adaptive mesh refinement for steady flows in Nek5000
(English)Manuscript (preprint) (Other academic)
Abstract [en]

Adaptive mesh refinement is performed in the framework of the spectral element method augmented by approaches to error estimation and control. The h-refinement technique is used for adapting the mesh, where selected grid elements are split by a quadtree (2D) or octree (3D) structure. Continuity between parent–child elements is enforced by high-order interpolation of the solution across the common faces. Parallel mesh partitioning and grid management respectively, are taken care of by the external libraries ParMETIS and p4est. Two methods are considered for estimating and controling the error of the solution. The first error estimate is local and based on the spectral properties of the solutionon each element. This method gives a local measure of the L2-norm of the solution over the entire computational domain. The second error estimate uses the dual-weighted residuals method — it is based on and takes into account both the local properties of the solution and the global dependence of the error in the solution via an adjoint problem. The objective of this second approach is to optimize the computation of a given functional of physical interest. The simulations are performed by using the code Nek5000 and three steady-state test cases are studied: a two-dimensional lid-driven cavity at Re = 7, 500, a two-dimensional flow past a cylinder at Re = 40, and a three-dimensional lid-driven cavity at Re = 2, 000 with a moving lid tilted by an angle of 30 degrees. The efficiency of both error estimators is compared in terms of refinement patterns and accuracy on the functional of interest. In the case of the adjoint error estimators, the trend on the error of the functional is shown to be correctly represented up to a multiplicative constant.

National Category
Fluid Mechanics and Acoustics
Identifiers
urn:nbn:se:kth:diva-251634 (URN)
Note

QC 20190520

Available from: 2019-05-16 Created: 2019-05-16 Last updated: 2019-05-20Bibliographically approved
6. Unsteady adjoint error estimators and adaptive mesh refinement in Nek5000
Open this publication in new window or tab >>Unsteady adjoint error estimators and adaptive mesh refinement in Nek5000
2019 (English)Report (Other academic)
Abstract [en]

Unsteady adjoint error estimators based on the dual-weighted residuals method are implemented for the spectral element method in Nek5000. The time-integration of the adjoint problem is performed based on the nonlinear direction solution recomputed via the revolve algorithm, which uses an optimal check-pointing strategy. Adaptive mesh refinement is performed on the flow inside a constricted periodic channel, the so-called periodic hill case, at four different Reynolds numbers, Re = 700, 1400, 2800 and 5600. This case is fully turbulent at all regimes, with significant flow separation, requires curved meshes, but yet has a number of accurate reference solutions in the literature. The chosen method to adapt the mesh is h-refinement, where selected elements are split by an oct-tree structure in three dimensions. The objective function for the adjoint estimators is the integral of the friction forces along the flat bottom wall between the hills, for which the location of the reattachment becomes crucial. The refinement process is compared between the adjoint error estimators and classical straightforward a posteriori spectral error indicators based on the local approximation properties of the solution.The turbulent simulations using mesh adaptation are stable, free of spurious numerical noise and accurate, as shown by comparing the statistical profiles of relevant flow quantities with reference data. The comparison between the error estimators shows that the adjoint error estimators tend to refine the mesh only around localized regions in the computational domain while leaving other areas under-resolved. However, only the locally refined regions are shown to have a significant impact on the value of the objective function and thus on the location of the reattachment point. Conversely, the spectral error indicatorstend to homogenize the error on the solution over the whole domain but have a lesser direct influence on the location of the reattachment point.

National Category
Fluid Mechanics and Acoustics
Research subject
Engineering Mechanics
Identifiers
urn:nbn:se:kth:diva-251636 (URN)
Note

QC 20190520

Available from: 2019-05-16 Created: 2019-05-16 Last updated: 2019-05-20Bibliographically approved
7. Using adaptive mesh refinement to simulate turbulent wings at high Reynolds numbers
Open this publication in new window or tab >>Using adaptive mesh refinement to simulate turbulent wings at high Reynolds numbers
Show others...
2019 (English)Conference paper, Published paper (Refereed)
Abstract [en]

The implementation of adaptive mesh refinement (AMR) in Nek5000 is used for the first time on the simulation of the flow over wings. This is done by simulating the flow over a NACA4412 profile with 5 degrees angle of attack at chord-based Reynolds number 200,000. The mesh is progressively refined by means of AMR which allows for high resolution near the wall whereas significantly larger elements are used in the far-field. The resultant mesh shows higher resolution than previous conformal meshes, and it allows for larger computational domains,which avoid the use of RANS to determine the boundary condition, all of this with, approximately, 3 times lower total number of grid points. The results ofthe turbulence statistics show a good agreement with the ones obtained with the conformal mesh. Finally, using AMR on wings leads to simulations at higher Reynolds numbers (i.e. Rec = 850, 000) in order to analyse the effect of adverse pressure gradients at high Reynolds numbers.

National Category
Fluid Mechanics and Acoustics
Research subject
Engineering Mechanics
Identifiers
urn:nbn:se:kth:diva-251637 (URN)
Conference
TSFP11 conference, July 30 - August 2, 2019, Southampton, United Kingdom
Note

QC 20190521

Available from: 2019-05-16 Created: 2019-05-16 Last updated: 2019-05-21Bibliographically approved

Open Access in DiVA

fulltext(919 kB)83 downloads
File information
File name FULLTEXT01.pdfFile size 919 kBChecksum SHA-512
ba3f78e47537302b10e0efbb253d9baa1e38d12e7fcb44cf354e2f5b008cd1f5e85227a39d26e46153139a4244dd140bd78ffdd8fb340d5f8986e45f31137a77
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Offermans, Nicolas
By organisation
Mechanics
Fluid Mechanics and Acoustics

Search outside of DiVA

GoogleGoogle Scholar
Total: 83 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 1441 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf