FEARLESS A new modelling approach for turbulent astrophysical flows

Orion Nebula (VLA)

Wolfram Schmidt
J. C. Niemeyer, L. Iapichino A. Maier, M. Hupp, Ch. Federrath, J. Adamek

In collaboration with

Ch. Klingenberg (Inst. f. Mathematik, Univ. Wrzburg) A. Kritsuk (Laboratory f. Comp. Astrophysics, UCSD)

Astrophysical Turbulence
Planetary and stellar convective boundary layers Heat transport by convection inside stars Turbulent thermonuclear combustion in supernovae Gas flow in accretion disks Turbulence in the interstellar medium Hot gas in galaxy clusters

DEISA Training Barcelona, Spain

Wolfram Schmidt Universitt Wrzburg, Germany

Turbulence Fundamentals
Fluid motion becomes increasingly chaotic through non-linear energy transfer Vortices (eddies) develop on a multitude of scales Turbulent dynamics is vortex stretching in three dimensions

DEISA Training Barcelona, Spain

Wolfram Schmidt Universitt Wrzburg, Germany

Turbulence Fundamentals
Fluid motion becomes increasingly chaotic through non-linear energy transfer Vortices (eddies) develop on a multitude of scales Turbulent dynamics is vortex stretching in three dimensions

DEISA Training Barcelona, Spain

Wolfram Schmidt Universitt Wrzburg, Germany

The Turbulence Cascade

So, natralists observe, a flea Hath smaller fleas that on him prey; And these have smaller yet to bite em And so proceed ad infinitum Thus every poet [fluid dynamicist], in his kind, Is bit by him that comes behind.
Lewis Fry Richardson
DEISA Training Barcelona, Spain Wolfram Schmidt Universitt Wrzburg, Germany 6

Large Eddy Simulation

# degrees of freedom:

E(k) k-5/3

Reynolds number:

K k

turb.: Re ~ 1000, terrestr.: Re ~ 107, astrophys.: Re ~ 1014

Supercomputers manage N ~ 10 , i.e. Re ~ 104

DEISA Training Barcelona, Spain Wolfram Schmidt Universitt Wrzburg, Germany 7

Large Eddy Simulation

=10-3L 0.01L
Static numerical grid of given resolution Computation of flow dynamics at scales < l < L Subgrid scale model accounts for turbulent eddies of size l < SGS turbulence stress ~ (SGS turbulent viscosity) * (rate of strain)

DEISA Training Barcelona, Spain

Wolfram Schmidt Universitt Wrzburg, Germany

Supersonic Turbulence
Mach number:

Compressibility becomes important if Ma ~ 1 For Ma > 1 , shock fronts dominate over large eddies
DEISA Training Barcelona, Spain Wolfram Schmidt Universitt Wrzburg, Germany 9

Adaptive Mesh Refinement

Hierarchy of dynamically created grids of varying resolution n = L/Nn = L/(nN0), where = 2 or 4 Grid geometry adapts to flow structure
Kritsuk et al. (2006)

DEISA Training Barcelona, Spain

Wolfram Schmidt Universitt Wrzburg, Germany

10

But Turbulence is Space-Filling, Right?

Homogeneous turbulence is space-filling from the view point of the ensemble average (Kolmogorov theory, E(k) ~ k -5/3) However, turbulence is intermittent At any instant of time, dissipative structures (turbulent eddies, shocks) are concentrated in regions of fractal dimension D less than 3 Challenge: Keep track of turbulent flow structure via appropriate refinement criteria in AMR simulations
DEISA Training Barcelona, Spain Wolfram Schmidt Universitt Wrzburg, Germany 11

AMR + SGS = FEARLESS

L
1 2 3 4
subgrid

effectively resolved

Resolve shocks and collapsing regions with AMR SGS model treats asymptotically isotropic turbulence Fluid mEchanics with Adaptively Refined Large Eddy SimulationS imulation
DEISA Training Barcelona, Spain Wolfram Schmidt Universitt Wrzburg, Germany 12

DEISA Project Goal

In our DEISA project, we investigated the applicability of AMR to forced supersonic turbulence by implementing new refinement criteria into the Enzo code and comparing it to static grid simulations. Enzo is an Open Source C++/Fortran AMR code developed by the Laboratory for Computational Astrophysics at The University of California in San Diego.
DEISA Training Barcelona, Spain Wolfram Schmidt Universitt Wrzburg, Germany 13

Big DEISA Runs

Box with periodic boundary conditions Stochastic forcing drives turbulent flow
Characteristic length scale ~ L = box size Intitial condition: const. density, gas at rest Force field stirs and compresses gas over time scale T RMS velocity asymptotically approaches ~ V = L/T

Static grid, N = 7683 = 452984832 cubic cells ~ 10000 time steps for sufficient relaxation ~ 100000 CPU-h required for one simulation!
DEISA Training Barcelona, Spain Wolfram Schmidt Universitt Wrzburg, Germany 14

DEISA Resources
DEISA Extreme Computing Initiative (DECI): Free CPU-time and porting/optimization support for challenging projects FEARLESS was assigned to the Dutch computing center SARA as an execution site, with the German Leibniz Computing Center (LRZ) managing correspondence and support Jobs were submitted to SARA using the UNICORE client SARA-ASTER machine features:
SGI Altix 3700 with 512 CPUs in four partitions Maximum #CPUs = 126 for MPI programs Linux OS
DEISA Training Barcelona, Spain Wolfram Schmidt Universitt Wrzburg, Germany 15

Technical Problems and Solutions Lessons Learned

Initial difficulties with Grid certificates, UNICORE and the SARA runtime environment Fixed in collaboration with SARA and LRZ Make sure to set aside some time for setting up

and getting used to the grid middleware this is not simply a matter of logging into a remote system via ssh.

DEISA Training Barcelona, Spain

Wolfram Schmidt Universitt Wrzburg, Germany

16

Technical Problems and Solutions Lessons Learned

Enzo did not respond well to some of the new features implemented (segfaults, library incompatibilities) and to the new hardware A number of debug iterations on a LRZ Altix machine in cooperation with the LRZ DEISA team helped to fix the problems Remote debugging is very cumbersome if unsure

whether your code works on the target platform, ask your computing center for a similar (local) machine to do the testing on.
Wolfram Schmidt Universitt Wrzburg, Germany

DEISA Training Barcelona, Spain

17

Technical Problems and Solutions Lessons Learned

FEARLESS uses a full quarter of the target machine, despite decreasing the computational domain from 10243 to 7683 Special arrangement with SARA allows high priority runs for long uninterrupted periods Flexibility on both sides is in order, as long as the virtualization of the computing resources is not realized in its entirety.

DEISA Training Barcelona, Spain

Wolfram Schmidt Universitt Wrzburg, Germany

18

Technical Problems and Solutions Lessons Learned

Total amount of primary output data: ~1 TB. After post processing and visualisation, ~4 TB have to be transferred to LRZ and archived GridFTP used (UNICORE and scp too slow, GPFS not yet available for the machines involved) takes about 20 hrs of uninterrupted transfer at the 55 MB/s network speed reached You may not have your results available locally immediately if major amounts of data are produced, think about where you need them and maybe ask your computing center for a net speed test.
DEISA Training Barcelona, Spain Wolfram Schmidt Universitt Wrzburg, Germany 19

Turbulence Energy and Vorticity

Isothermal EOS Mach number

V/c0 = 2.89

DEISA Training Barcelona, Spain

Wolfram Schmidt Universitt Wrzburg, Germany

20

Turbulence Energy and Vorticity

Isothermal EOS Mach number

V/c0 = 2.89

DEISA Training Barcelona, Spain

Wolfram Schmidt Universitt Wrzburg, Germany

21

Turbulence Energy and Vorticity

Isothermal EOS Mach number

V/c0 = 2.89

DEISA Training Barcelona, Spain

Wolfram Schmidt Universitt Wrzburg, Germany

22

Turbulence Energy and Vorticity

Isothermal EOS Mach number

V/c0 = 2.89

DEISA Training Barcelona, Spain

Wolfram Schmidt Universitt Wrzburg, Germany

23

Turbulence Energy and Vorticity

Isothermal EOS Mach number

V/c0 = 2.89

DEISA Training Barcelona, Spain

Wolfram Schmidt Universitt Wrzburg, Germany

24

Turbulence Energy and Vorticity

Isothermal EOS Mach number

V/c0 = 2.89

DEISA Training Barcelona, Spain

Wolfram Schmidt Universitt Wrzburg, Germany

25

Turbulence Energy and Vorticity

Isothermal EOS Mach number

V/c0 = 2.89

DEISA Training Barcelona, Spain

Wolfram Schmidt Universitt Wrzburg, Germany

26

Turbulence Energy and Vorticity

Isothermal EOS Mach number

V/c0 = 2.89

DEISA Training Barcelona, Spain

Wolfram Schmidt Universitt Wrzburg, Germany

27

Mass Density
Mostly compressive forcing generates compact dense regions

DEISA Training Barcelona, Spain

Wolfram Schmidt Universitt Wrzburg, Germany

28

The Quest for Turbulence Refinement Criteria

Small test runs: N0 = 963 root grid, 1 refined level + N = 1923 static grid simulation for comparison
Temporal evolution of global statistical moments Computation of probability density functions 3D visualisations

Production runs: N0 = 1923 root grid, 1-2 refined levels

Single data dump comprises up to 20000 output files (proportional to number of refined regions) Postprocessing and analysis quite laborious
DEISA Training Barcelona, Spain Wolfram Schmidt Universitt Wrzburg, Germany 29

The Quest for Turbulence Refinement Criteria

The conventional approach is to refine grids in the vicinity of steep gradients A priori specification of approriate thresholds for gradients is difficult We investigated an alternative approach:
Monitoring of vorticity and rate of compression Thresholds based on the regional variability

Only peaks of turbulence production and gas compression should trigger refinement!
DEISA Training Barcelona, Spain Wolfram Schmidt Universitt Wrzburg, Germany 30

Getting Hold of Supersonic Turbulene with AMR

vorticity squared mass density

DEISA Training Barcelona, Spain

Wolfram Schmidt Universitt Wrzburg, Germany

31

Getting Hold of Supersonic Turbulene with AMR

probability density function of mass density

AMR works
DEISA Training Barcelona, Spain Wolfram Schmidt Universitt Wrzburg, Germany

32

Astrophysical Application: Galaxy Cluster Simulations

Small clusters (subclusters) of galaxies occasionally fall into pontential wells of large clusters Infall produces bow shock and vortex tail in surrounding intercluster medium (hot dilute gas) Wide range of different scales necessitates AMR Application of different refinement criteria in three-dimensional simulations

DEISA Training Barcelona, Spain

Wolfram Schmidt Universitt Wrzburg, Germany

33

Astrophysical Application: Galaxy Cluster Simulations

Iapichino & Adamek (2007)

conventional
DEISA Training Barcelona, Spain

FEARLESS
34

Wolfram Schmidt Universitt Wrzburg, Germany

Resume
I. Since DECI policy does not allow projects to be extended, and computation resources are fixed, it has proven difficult to complete the development and implementation of novel techniques within the project. II. Ideally, the code used and the problem to be solved computationally should be well understood beforehand, thereby enabling accurate resource estimates in the project proposal. III. Practically, more flexibility in the usage of the DEISA resources for tackling problems at the frontiers of contemporary research would be desirable, i.e. completely virtualized resources. While this is the very point of distributed access to supercomputers, it seems to be quite far off.
DEISA Training Barcelona, Spain Wolfram Schmidt Universitt Wrzburg, Germany 35

Acknowledgement

We thank the DECI teams from SARA and LRZ for their extensive support!
DEISA Training Barcelona, Spain Wolfram Schmidt Universitt Wrzburg, Germany 36