Read MpCCI_Proceedings_2007_14.indd text version







COUPLED SIMULATIONS A new market segment for Computer Aided Engineering MpCCI (Mesh-based parallel Code Coupling Interface) has opened a new market segment for CAE: multi disciplinary simulation through coupling of established and best in-class simulation tools offers a variety of new disciplines and solution strategies. MpCCI has been developed at the Fraunhofer Institute SCAI in order to provide a code-independent interface for the coupling of different simulation codes. Most of the leading CAE software vendors support the MpCCI Interface. About 100 licensed users and 40 partner installations worldwide (in 2006) have demonstrated usability and benefit of the MpCCI open concept. Supported Codes (MpCCI 3.0.6): ° ABAQUS ° ANSYS ° Flowmaster (in preparation) ° FLUENT ° FLUX ° Icepak ° MSC.Marc ° PERMAS ° STAR-CD ° RadTherm ° Programmable Interface for inhouse Codes On behalf of the MpCCI Team Coupled Application Areas Fluid Structure Interaction ° Thermal Coupling and Radiation ° Thermo-Electrical Interaction ° Magneto-Hydro-Dynamics ° Coupling of 3D-CFD and 1D-Pipeline ° Fluid-Fluid Combinations ° and many more


Klaus Wolf

Klaus Wolf Fraunhofer Institute for Algorithms and Scientific Computing SCAI Schloss Birlinghoven 53754 Sankt Augustin Germany

Phone: +49 (0) 2241 / 14 - 2557 [email protected]





News around MpCCI Development and Projects Klaus Wolf ­ Fraunhofer Institute SCAI 10-17


A new CFD and thermal analysis method based on MpCCI Ernst-Peter Weidmann ­ FKFS, Walter Bauer, Martin Maihöfer ­ DaimlerChrysler and Uwe Scholl ­ Fraunhofer SCAI 18-27

Coupling FLUENT with FEM codes Mark Pelzer ­ FLUENT Germany


Solving fluid-structure interaction problems using MpCCI and the Component Template Library (CTL) J. Rang, M. Krosche, R. Niekamp, H. G. Matthies ­ Technical University of Braunschweig



How to create your own code adapter Holger Spiess and Pascal Bayrasy ­ Fraunhofer SCAI 42-45

Coupled simulations using the elasticity solver DDFEM and FLUENT via MpCCI Dimitar Stoyanov ­ Fraunhofer ITWM


Fluid-structure simulations using MpCCI 3.0.5 with ABAQUS and an in-house CFD-Code on distributed resources Felix Lippold, University of Stuttgart IHS




Simulation of the switching arc for a motor protective circuit breaker Christian Rümpler ­ Fraunhofer SCAI and Albert Zacharias ­ Moeller GmbH 62-69

MpCCI-coupled simulations for electrical-distribution equipment Ian Lyttle, Benjamin Pulido ­ Schneider Electric



Pressure moulding simulation using FLUENT and ABAQUS Jasper Kidger ­ FLUENT UK 76-83

Fluid-structure solutions based on ABAQUS Albert Kurkchubatsche ­ ABAQUS US


Coupling STAR-CD with ABAQUS to model FSI interaction of a body entering a fluid Philip Morris Jones ­ CD-adapco



R&D of nuclear power plant simulator on atomic energy grid infrastructure Noriyuki Kushida, Yoshio Suzuki, Osamu Hazama, Hitoshi Matsubara, Akemi Nishida, Fumimasa Araya, Tetsuo Aoyagi and Norihiro Kakajima ­ Japan Atomic Energy Agency JAEA 102-107



3-code coupling in flood simulation Jörg-Volker Peetz ­ Fraunhofer SCAI and Thomas Sommer ­ Dresden Groundwater Research Center



Preconditions for a fluid-structure interaction of the human heart using FLUENT, ABAQUS and MpCCI S.B.S. Krittian, H. Schmid, T. Schenkel, H. Oertel ­ University of Karlsruhe 118-125

Code adapter for arbitrary prescribed motion and deformation using FLUENT and MpCCI M.G. Perschall, H. Spiess, T. Schenkel, H. Oertel ­ University of Karlsruhe







Klaus Wolf ­ Fraunhofer Institute SCAI Solving multi-disciplinary problems by using coupled simulation environments is becoming a state-of-the-art technology for a fast growing engineering community. The MpCCI interface has been accepted as a »de-facto« standard for such code coupling environments. MpCCI 3.0.6 In spring 2007 the new version of MpCCI 3.0.6 will be released. Besides various improvements for the system internal processing there are some major enhancements for the user interface and code support: · In MpCCI 3.0.6 for most codes the »self-code-coupling« is available. Instead of coupling two different simulations tools a user may decide to set up an interaction between two instances of the same code. · In its public production version MpCCI already supports nine codes (ABAQUS, ANSYS, FLUENT, Flux3D, ICEPAK, MSC.Marc, PERMAS, STAR-CD 3 & 4, RadTherm). The new Code-Adapter-API in 3.0.6 now opens the door for further code-writers to create MpCCI interfaces for their own codes which are fully compatible with the codes listed above. · In most multi-disciplinary cases the user will take care that the coupled surfaces match to each other as far as possible. For large applications with hundreds of (independent) parts and a total size of coupling surface with 105 to 106 nodes/cells such a 100 percent geometric compatibility cannot be guaranteed due. MpCCI 3.0.6 now provides some extrapolation schemes which provide coupled values even for such regions which otherwise would be »orphaned«. · To support deforming FSI-structures when coupling STAR-CD with FEM codes MpCCI provides a separate morphing tool. This spring-based morpher has been extended to run also polyeder based models as used by STAR-CD 4. · For MpCCI 3.0.6 we put a lot of effort in a much more detailed documentation and extended tutorials. Various examples for different code combinations and comparisons of coupled simulation results should allow MpCCI beginners to run first simple application cases in a do-it-yourself mode.


Coupling two instances of the same Code Although most applications use MpCCI to couple two different codes there are also some problems where interaction between two instances of the same code is useful. Compared to a single complex model a coupled environment provides more flexibility for the model set-up. The coupling of two independent sub-models allows to specify appropriate runtime parameters (e.g. relaxation factors or discretization schemes) for each sub-model with different values. In MpCCI 3.0.6 such »self-code« couplings can easily be specified by selecting two models of same code type in the »Model Panel« of the MpCCI GUI. All following steps (coupling specs, parameters and startup) are handled in same way as for coupling two different codes.

Coupling two instances of the same Code An example of a coupled CFD-CFD application would be a Glass Tank simulation using a FLUENT-FLUENT environment. The first sub-domain is computing the combustion chamber ­ the second CFD domain is for the glass space itself. This is a thermal coupling only. Each model has its own radiation model ­ no radiation properties will be transferred. The temperature profile is send from the glass tank domain to the combustion area; heat flux is returned from combustion to glass area. First test cases with a simple 2D case show similar results in the single FLUENT solution and the coupled FLUENT-FLUENT model (example provided from Fluent Germany/ Fluent Asia Pacific).

Klaus Wolf Fraunhofer Institute for Algorithms and Scientific Computing SCAI Schloss Birlinghoven 53754 Sankt Augustin Germany

Phone: +49 (0) 2241 / 14 - 2557 [email protected]



Programming Interface to create Adapters for in-house Codes The basic idea of MpCCI is to enable the coupling between any two simulation codes ­ independent if they are commercial, in-house or research tools. MpCCI provides two levels of programming interfaces (APIs) to link the coupling software MpCCI with a code. · MpCCI SDK is a lower level subroutine interface and · MpCCI Adapter level used since MpCCI 3.0 for all commercial codes. The MpCCI SDK interface level provides various basic functions to define coupling regions, control communication between the coupled codes and to handle other MpCCI parameters. The usage of MpCCI SDK is very similar to the usage of the MPI Message Passing Interface - there is no fixed protocol when to communicate which data with which other coupled components. This MPI-like flexibility allows the implementation of different coupling algorithms. However, this flexibility has one big disadvantage ­ MpCCI itself cannot guarantee that the sequence how MpCCI procedure calls (define meshed, send and receive operations) used in code A matches to a sequence of MpCCI calls realised in code B. To avoid such protocol incompatibilities between different code integrations MpCCI 3.0 has introduced the concept of MpCCI code adapters. These adapters have internal mechanisms to control the current coupling state and actions of each integrated code. These »Coupling Control Managers« guarantee a consistent behaviour and communication protocol for all adapted codes. MpCCI 3.0.6 now provides an open programming interface (API) to implement code adapters for own in-house codes (details on how to use this new API can be found in the article from Holger Spiess). First solutions based on this open API already have been realised by Fraunhofer ITWM (in-house FEM code), University of Stuttgart (own CFD code), University of Karlsruhe (image processing code) and an industrial customer using a large research code for CFD which will be coupled to the ABAQUS code. The current MpCCI 3.0 still allows both levels of integration (SDK and Adapter level). However, it should be noted that for reasons of compatibility the MpCCI Adapter Level should be preferred for all new activities. In midterm the MpCCI SDK interface will be regarded as a pure internal MpCCI and thus might be subject to larger changes.

Simulation code

GUI run integration control



code adapter






Orphaned Regions in FEM without coupled values from CFD

Orphaned Regions in FEM with extrapolated CFD-values

The MpCCI interpolation algorithms are based on the assumption that both coupled surfaces are well defined and geometrically compatible to each other. The CFD and FEM domain can be modelled individually based on accuracy and runtime demands. MpCCI then is responsible to map the discretisations used in CFD and FEM to each other. However, in some cases it might happen that parts which are modelled in detail in the FEM domain are not represented in the coupled CFD domain. In the figure above the pipe is only part of the FEM model ­ and not of the CFD model. To handle such model mismatches a method is needed for assigning data and thus emulating a coupling for this pipe region. One way is to assign default values. This leads to a number of problems from listing all regions up to guessing values for each region. Therefore MpCCI was extended to deal automatically with non matched regions by extrapolating values from matched regions into the non matched ones. The extrapolation is based on a weighting function over the distances to the borders of the non matched region. With d being the distance between a non matched cell x and a border cell b, f(bn) is a value assigned to b, w is a weighting function such that, 1=w(d(x,bn))



the value of x is then extrapolated by: f(x)=w(d(x,bn))·f(bn)





MpCCI Grid Morpher The MpCCI grid morpher will smooth a grid, which contains displaced nodes / vertices, typically encountered in fluid-structure interactions. The MpCCI grid morpher is based on a spring-based morphing technique applying a modified radial-based function theory. 1. The MpCCI mesh morpher is started as a separate process and connected to the parallel STAR-CD program via sockets. At start-up the morpher receives the initial mesh definition of the STAR-CD. 2. During coupled simulation the FEM codes sends deformation information to the code adapter of STAR-CD. 3. These structural changes of the fluid boundary are then transferred to the grid morpher. The morpher calculates the new positions of boundary and inner vertices and does an additional smoothing step on the CFD mesh. 4. The morphed mesh model is returned back to the STAR-CD code adapters; the newxyzfunction is used to pass the adapted model to the STAR-CD solver.

Test case for morpher performance

Some basic benchmarks for the morpher may demonstrate the performance of this MpCCI add-on tool. The tests were done on a 3 GHertz Xeon machine. The model was a rectangular channel flow (STAR-CD 3.26) with a simple flap placed in the middle of the channel (ABAQUS 6.7). The MpCCI morpher shows a very good speed-up related to the number of nodes in volume: · ~ 1,000 nodes on surface / 25,000 Elements in volume 0,15 seconds · ~ 4,000 nodes on surface / 200,000 Elements in volume 0,8 seconds · ~15,000 nodes on surface / 1,500,000 Elements in volume 6,0 seconds


THE OPEN MpCCI COMMUNITY The vision and goal for MpCCI is to be the leading and most used open interface for simulation code coupling. MpCCI was and is open for any interested code writer (from research, industry, and commercial software vendors) who wants his code to interact with other simulation software. The open API in MpCCI 3.0.6 now provides a straight forward strategy to combine own codes with other »MpCCI enabled« codes in one multi-disciplinary environment. All technical information and software interfaces to adapt further simulation tools to the MpCCI concept are part of the public MpCCI product version. Fraunhofer SCAI will use its position as an independent organisation for applied research to push this vision of an »Open MpCCI Community« by motivating and supporting any organisation to adapt its simulation solution to the MpCCI interface concept. Trilateral FSI Co-operations In 2004 ABAQUS Inc. and FLUENT Inc. (now part of Ansys Inc.) announced their strategic partnership to solve fluid-structure problems with MpCCI based environments. In joint development and marketing efforts all three partners together could demonstrate the usability and quality of FSI environments. Since the first product version (MpCCI 3.0.4, ABAQUS 6.5 and Fluent 6.2.18) was launched in summer 2004 around 60 users in industry and research switched from a previous `mono-disciplinary' working environment to a now multi-disciplinary coupled solution. At the 8th MpCCI User Forum 2007 Dassault Systèmes and CD-adapco announced MpCCI as solution for analysing fluid-structure interaction with STAR-CD and ABAQUS. Dassault Systèmes and CD-adapco, world leading providers of simulation solutions, follow the strategy of delivering an open platform for multi-physics simulation. The bi-directional coupling leverages a strong partnership of both companies with each other and the independent, open solution for the coupling ­ MpCCI software ­ to enable ABAQUS and STAR-CD to work together to solve a wide range of important FSI problems. Both code combinations ­ ABAQUS / MpCCI / FLUENT as well as ABAQUS / MpCCI / STAR-CD ­ are fully supported by all the related partners. End-users who decide to start and use one of these combinations will receive best FSI support either from the CFD, or from the FEM or from the SCA as interface provider. Both tri-lateral co-operations are settled as strategic solutions for a long-term usage by the engineering community.



Bilateral Co-operations CEDRAT SA in France for the integration of the Flux3D EMAG code. Our customer Schneider Electric US is using a code combination ICEPAK ­ Flux3D to compute thermal behaviour virtual switching devices and busbar systems (see article from Ian Lyttle, Schneider US in this proceedings); Flowmaster Ltd. UK provides a system simulation code (Flowmaster) for 1D pipeline computations. Users in automotive and aerospace industry want to set-up 1D-3D (FLUENT or STAR-CD) environments to compute air condition or cooling networks in combination with detailed 3D parts. This 1D-3D environment has reached a prototype stage ­ first steady state cases tests show promising results. Public product versions are planned for summer 2007. Intes GmbH provides their Permas code (FEM) which is combined with STAR-CD and POSRAD to form a 3-code solution for the thermal management of automotive underhood (see contribution from Ernst-Peter Weidmann, DaimlerChrysler); MSC Software Corporation has enabled the MSC.Marc code for FSI applications with STARCD or FLUENT. First users from electronics industry in Japan recently have started to utilize this FSI solution for thermal problems. The radiation code RadTherm from ThermoAnalytics Inc. is used in combination with CFD codes to simulate the thermal/radiation effects in automotive applications. RESEARCH PROJECTS In its role as an organisation for applied research Fraunhofer SCAI participates in various national or European funded projects. In some of them MpCCI is used as a technical integration platform ­ in others new technical approaches are investigated for a potential further usage inside MpCCI. 3ZM-GRIMEX - Ground water simulation: The flood 2002 at the Elbe River showed that besides the apparent damages by surface waters, considerable damage was caused by groundwater and waters from the sewer system. Fast rising groundwater levels due to infiltration of overland flow or water from the sewer system can damage basements and can be a serious thread to the static of houses. The objective of the project 3ZM-GRIMEX is the development of a software system for the coupled simulation of three components: surface water, groundwater, and sewer water. The coupled system will be used in flood risk management. (More details are given in the article from Jörg-Volker Peetz, SCAI)


INGRID ­ Grid Based environments: There is an obvious trend to use distributed gridenvironments for large and complex applications. Compute resources, data bases and services from different sites may be combined to form a (temporary) solution for a specific problem case. Coupled FSI simulations could benefit from this »resource sharing« strategy in grids. The coupled components ­ CFD, FEM and MpCCI ­ could be offered as grid services (following the upcoming SOA standards); a user might then select available services and combine them in one application. A first prototype of such a grid based coupled simulation is presented in the article from Felix Lippold, University of Stuttgart. KOPFEM ­ Code API and Extrapolation Schemes: Nowadays, despite the variety of commercial solvers available on the market, the question of the design and development of effective numerical solvers is still open. It is a fact that many problems can not find their proper treatment using the existing (commercial) codes. Typical examples are large sized problems with complicated geometry, e.g. problems in micromechanics, see [1], which necessarily require a parallel execution. In this sense, a specific-purpose in-house code, specially designed and programmed for a particular parallel architecture, can often provide much better performance than the existing general-purpose tools (see article from Dimitar Stoyanov, Fraunhofer ITWM). IMAUF ­ Advanced Stamping Simulation: Simulation of manufacturing processes like metal stamping is a well established procedure in all of the automotive companies today. Dedicated stamping codes like Autoform, Indeed, Pam-Stamp and other more general codes like LS-Dyna or Abaqus Explicit can be used for these purposes. However, in all applications of stamping simulation today there is one major assumption about the behaviour of the stamping tool itself: the tool is expected to be stiff enough and no deformations due to friction of the stamped plate are considered during simulation. In a recently started project the coupling of a stamping simulation with the structural analysis of the deforming stamping tool will be realised. The deformation of the stamping tool will lead to locally different friction parameters ­ compared to a standalone stamping simulation. It is one of the challenges in this research project to evaluate influence of changing friction parameters onto the quality of the final stamped plate (see also CONCLUSION MpCCI 3.0 provides a lot of new features for the coupling of simulation codes. Together with MpCCI code adapters a complete toolbox for multidisciplinary simulation is ready for use with standard commercial simulation codes. Various solutions demonstrate the applicability of this concept and the valuable outcome for the end users. A growing number of commercial codes as well as inhouse and research codes are supported by MpCCI code adapters.




Ernst-Peter Weidmann ­ FKFS; Walter Bauer, Martin Maihöfer ­ DaimlerChrysler; Uwe Scholl ­ Fraunhofer Institute SCAI ABSTRACT Vehicle thermal protection is of vital importance for the development process of passenger cars. Tightly-packaged engine compartments and strongly increased engine power demand extensive testing and more and more numerical analysis. Underhood component temperatures are sensitive to all three modes of heat transfer, conduction, convection and radiation. Due to very high temperatures next to the exhaust system, in particular, a thermal radiation analysis is required. The different numerical properties of the three physical modes can be handled separately by sophisticated but specialized simulation codes. This paper proposes a partitioned approach for the multi physics 3-D simulation methodology based on coupling the commercial CFD code STAR-CD, the radiation code POSRAD and the finite element code PERMAS by the Coupling Software MpCCI. The applicability of this methodology is demonstrated by a simplified exhaust system. The surface temperature is computed considering both ­ internal and external flow. Air and exhaust gas temperatures are coupled with the thermal analysis. In the near future a similar methodology will be used to perform thermal analyses of the engine compartment and the entire car. The new trilateral coupling approach is supposed to be more accurate, especially in regions with a very strong coupling between temperature and velocity field. INTRODUCTION Thermal protection is increasingly important in the development process of passenger cars. Tightly-packaged engine compartments and strongly increased engine power demand extensive testing and analysis. Traditionally thermal protection is tested in climate wind tunnels and road tests. Reduced time to market and high costs for prototypes result in the need for building up the first prototype very close to its optimum design. The introduction of numerical methods allows for the optimization of cooling requirements as well as thermal analysis of temperature sensitive components in an early stage of the vehicle development process. Although road tests will also be necessary in the future to verify the numerical predictions, the use of numerical methods helps to focus such tests on the most critical points. Fluid-structure interaction covers a broad scope of problems. The interaction can be mechanical, thermal or even both. Underhood component temperatures are sensitive to


all three modes of heat transfer: conduction, convection and radiation. Due to very high temperatures next to the exhaust manifold, a thermal radiation analysis is required. There are basically two ways of approaching this problem. One way is to model the problem as one closed monolithic system. The other solution is based on a so called partitioned approach where each physical domain is modeled separately. Solving the system is to solve each domain and to exchange boundary values between them. In contrast to the monolithic approach, existing specialized simulation tools can be assigned to the different domains and modelling can be done individually with respect to accuracy and runtime demands. One drawback is the need of an additional tool for handling the exchange of boundary values between the simulation codes. For industrial application it is important to ensure that the tool is easy to use such that the overhead for setting up the coupling is small with respect to the effort for modelling and solving the different domains. It has been known for some time that there is no numerical analysis code available able to deal with the thermal problems in the complex geometry of a whole vehicle by a monolithic approach. Several examples of partitioned multi-mode heat transfer computations can be found in recently published literature (1, 2, 3). Reister and Maihöfer (4) and Bauer and Maihöfer (5) described a procedure to import heat transfer data from a CFD computation into a thermal analysis code. In the case of forced convective heat transfer the coupling between component temperature and flow field is relatively weak, although the opposite is not true. It is therefore sufficient to provide convective heat fluxes in a preliminary CFD analysis. On the other hand the strong coupling of the velocity and temperature field in buoyancy driven flows requires a two-way coupled multimode heat transfer analysis (6, 7). The methodology presented in this paper is an extension to the one presented by Maihöfer and Bauer (5) by considering the feedback between the structure and the CFD in both directions and in an iterated way. The method by Maihöfer and Bauer is based on the computational fluid dynamics code STAR-CD (8), the finite element code PERMAS (9) and the thermal radiation solver POSRAD (8). It is integrated in the development process of all new Mercedes-Benz passenger cars. To enable a coupling between the codes a software tool was needed. There are some vendor specific approaches to couple codes from different application fields via proprietary mechanisms (10). There are also some area specific coupling

Dipl.-Ing. Ernst-Peter Weidmann Forschungsinstitut für Kraftfahrwesen und Fahrzeugmotoren Stuttgart FKFS Pfaffenwaldring 12 70569 Stuttgart Germany

Phone: +49 (0) 7031 / 90 45727 [email protected]




solutions available; but their use is restricted to special application fields like aeroelastic coupling in airplane design or the coupling of ocean and atmosphere models in climate research (11). This situation motivated the development of a software environment which provides a general solution for multidisciplinary simulations. The radiative heat flux is calculated in a third program named POSRAD. POSRAD uses a fast beam tracking method based on an automatically generated »voxel mesh« for computing viewfactors for the surface to surface radiation. POSRAD and PERMAS are sequentially coupled using a script-based methodology. Radiative heat transfer is considered as a thermal load and modifies in each coupling step the right side of equation 2. The CFD mesh and the PERMAS grid are based upon the same CAD data but are generated with different meshing tools with different accuracy for the representation of the surface. STAR-CD V3.2x works most accurately with a trimmed mesh consisting of hexahedral volume cells in almost all parts of the computational domain. For PERMAS a tetrahedral grid, which is in most cases automatically created, is adequate. The common interface of both codes is the surface of both meshes. Therefore a mesh mapping tool is needed to map the physical properties from one mesh to the other (Figure 2). MpCCI (Mesh based parallel Code Coupling Interface) (12) has been developed at the Fraunhofer Institute SCAI in order to provide an application independent interface for the coupling of different simulation codes. MpCCI is a software environment which enables the exchange of data between the meshes of two simulation codes in the coupling region. The coupling is setup in 4 steps with the help of a graphical user interface. There are no restrictions on the type of transferred quantities; different discretisations at the coupling regions (interface) are handled by interpolation and the communication is based on common network protocols such that the simulation codes can run on different machines without requiring a shared file system. Despite this features MpCCI needed to be extended to meet the requirements for coupling three domains. STEADY-STATE UNDERHOOD LOAD CASES Maximum component temperatures are amongst others reviewed considering two steadystate load cases: 40 km/h trailer towing uphill and driving at maximum speed. For these load cases it can be assumed that the energy equation does not significantly feed back into the momentum equations. Therefore the flow through the engine compartment and the convective heat transfer can be computed separately in a first preliminary CFD computation. Only relevant heat sources such as the engine, the gearbox and the exhaust pipes are considered and experimental results are provided as thermal boundary conditions (5). Afterwards the convective heat transfer data is transferred to the thermal solver and a thermal analysis is carried out.


For the thermal analysis the finite-element code PERMAS is used. Equation (1) describes the energy equation solved in PERMAS.

__ · cp · T · t __ with T = 0 in steady-state computations. t

T ___ ___ (___ + yT + zT)= q x

2 2 2 2 2 2




Figure 1: Heat transfer at the interface boundary of fluid and solid

The thermal equilibrium at the interface boundary (Figure 1) of fluid and solid parts yields nt · · grad (T) qs + · (T -T




The radiative heat flux qs is calculated in a third program named POSRAD. POSRAD uses a fast beam tracking method based on an automatically generated »voxel mesh« for computing viewfactors for the surface to surface radiation. POSRAD and PERMAS are sequentially coupled using a script-based methodology. Radiative heat transfer is considered as a thermal load and modifies in each coupling step the right side of equation 2. The CFD mesh and the PERMAS grid are based upon the same CAD data but are generated with different meshing tools with different accuracy for the representation of the surface. STAR-CD V3.2x works most accurately with a trimmed mesh consisting of hexahedral volume cells in almost all parts of the computational domain. For PERMAS a tetrahedral grid, which is in most cases automatically created, is adequate. The common interface of both codes is the surface of both meshes. Therefore a mesh mapping tool is needed to map the physical properties from one mesh to the other (Figure 2). This methodology performs quite well in problems involving forced convection which is the dominating convective heat transfer mode when the vehicle is driving, either at high speed



Figure 2: Mesh mapping on different meshes for STAR-CD and PERMAS or with an operating fan. However buoyancy driven flows, as the heat soak problem after stopping and shutting off the engine, cannot be computed by this approach. In this case the flow in the engine compartment is only driven by local density gradients caused by different air and component temperature. Flow field and component temperatures are tightly linked to each other. Besides buoyancy driven flows a methodology is required that is able to deal with two fluid streams determining the surface temperature of a component. This is in particular important for exhaust pipes with an inner exhaust flow and an outer air flow, e.g. the underbody flow. As mentioned before the temperatures of most heat sources in the engine compartment were set to known temperatures. As accurate experimental values for the outer surface of an exhaust system are often not available in an early stage of the development process, a closed simulation solution is necessary. COUPLED SYSTEM Both applications demand feedback between the systems. In the case of underhood simulation the problem is reducible to two bilateral couplings. One is the reaction of the structure on radiation and vice versa, and the other is the heat exchange between the fluid and the structure. This can be modelled by a so called staggered loop as illustrated in Figure 3, based on an iterative coupling of the three software tools. In the steady-state simulation the heat transfer coefficient and film temperature are sent from the fluid and the heat flux from the radiation code to the structures code, starting a new simulation of the structural problem. The resulting wall temperature is sent back to the radiation solver and to the fluid code, which start their simulation (Figure 4). The system is converged if the results of all codes keep equal related to their proceeding results. To improve convergence a relaxation can be applied to the values at the boundary. This methodology can be extended such that the staggered system can solve transient problems. Further investigations are necessary to prove if staggered loops between the


domains for each time step are necessary to get a reliable solution (Figure 3 (right)) or if simple predictor based systems may give accurate solutions as well (Figure 3 (left)).

t t











Figure 3: (left) solution of B is used to predict the boundary values P(y) of next time step for A. (right) Inner loops are used to find the accurate solution for each timestep.

TEST CASE EXHAUST SYSTEM As an example a part of a simplified exhaust system and surrounding components are considered. The mass flux and the temperature of the exhaust gas are set as boundary conditions.

Figure 4: Coupling Methodology Additionally the contact surface of the manifold and cylinder head is set to a fixed temperature. For confidential reasons all boundary conditions presented here are arbitrarily chosen. In a first step the inner surface of the exhaust system is considered as adiabatic and the preliminary STAR-CD run for the script-based approach is done. The resulting near wall temperatures and heat transfer coefficients are mapped to the PERMAS grid as described earlier. Afterwards the wall temperature is computed with fixed boundary conditions for the convective heat flux. In a second step the same problem is run with the MpCCI based trilateral coupling approach. The inner surface of the exhaust system is computed by coupling STAR-CD, PERMAS and POSRAD iteratively. For the outer surface PERMAS and POSRAD are coupled iteratively while the convective heat transfer is read from include files analogue to the script-based approach. The first computation neglects the cooling effect on the exhaust flow at the outer side of the exhaust pipes. Therefore the computation tends to result in temperatures of the exhaust system which are much too high (Figure 5 (right)). To overcome this problem, the temperature



distribution of the first computation with the »old« approach is used as boundary condition in a new STAR-CD computation and the resulting convective heat fluxes are once again transferred to PERMAS. After three loops of the »old« approach similar results compared to the new one can be observed. It is important to emphasize that in the »old« approach data is only manually transferred to the other code after fully converged runs while the MpCCI based method performs an automatic iterative exchange.

steering column

element 1

adiabatic wall treatment in STAR-CD

MpCCI approach

Figure 5: Surface temperatures of the simplified exhaust system and surrounding components (left) ­ temperature of the exhaust gas system for adiabatic wall treatment in STAR-CD and the MpCCI based approach (right) Figure 6 shows the results for »element 1« which is located on the exhaust system for the »old« approach with different iterations and the result with the new approach. The steering column (Figure 5 (left)) is chosen for plotting temperature values (Figure 6). It is strongly affected by the radiation from the exhaust pipes. The overestimated temperatures in case of an adiabatic wall treatment in STAR-CD can also be found in the component temperatures. The computational effort comparing both approaches is given in Figure 7. The »old numerical« approach is set to the value of one as a reference value. The proposed MpCCI

Figure 6: Results for the exhaust system (left) and the steering column (right)


approach has a little more computational effort then a single loop with the old methodology, but is much more efficient, especially compared with two or more loops for the old one. The runtime differences result from the iterative coupling, the leaner communication architecture of MpCCI compared to the file based old approach and from the reuse of calculated neighbourhoods for each coupling iteration step. These numbers are strongly affected by the model size and may differ for other examples.

Figure 7: Scaled computational effort for both methodologies

CONCLUSION The trilateral partitioned approach presented in this paper demonstrates its capability to deliver accurate numerical results, with minor effort with respect to runtime and set up overhead. It is a straight forward extension to the already established method presented in (4,5) and can be used for buoyancy driven flows or for computing surface temperatures of an exhaust system considering both the inner exhaust flow as well as the air flow at the exterior. It also allows ­ as introduced above ­ for extension to transient problems in the near future. The new approach fulfils accuracy and runtime demands within a development process for passenger cars. In the near future this approach will be used to compute component temperatures in an entire car underhood, e.g. in case of »thermal soak«. ACKNOWLEDGEMENTS This research was funded by DaimlerChrysler AG, department EP/SAE. The authors would like to thank Dr.-Ing. Raimund Siegert for supporting this work and for permission to publish this paper.



NOMENCLATURE AND ABBREVIATIONS cp d g h q t P T x y [K] [J/KgK] [m] [m/s2] [W/m2K] [W/m2] [s] specific heat Distance acceleration due to gravity convective heat transfer coefficient heat flux Time Predicator Temperature Boundary values of code A Boundary values of code B

Greek symbols

[W/mK] [Kg/m3] Conductivity Density


CFD con S MpCCI computational fluid dynamics Vonvection Radiation Mesh-based parallel Code Coupling Interface


REFERENCES [1] Bendell, E. »Investigation of a Coupled CFD and Thermal Modelling Methodology for Prediction of Vehicle Underbody Temperatures« VTMS 7, 2005-01-2044, Toronto 2005 [2] Srinivasan, K., Woronowycz G., Zabat M., Tripp J. »An Efficient Procedure for Vehicle Thermal Protection Development«, SAE 2005-01-1904, Detroit 2005 [3] Fortunato, F., Damiano, F., Di Matteo, L., Oliva P. »Underhood Cooling Simulation for Development of New Vehicles«, VTMS 7, SAE 2005-01-2046, Toronto 2005 [4] Reister, H., Maihöfer, M. »Underhood Component Temperature Analysis for Passenger Cars« VTMS 6 SAE Conference, 2003. C599/028/2003. [5] Bauer W., Maihöfer M. »Numerische Simulation der Bauteiltemperaturen eines Gesamtfahrzeuges«, VDI Tagung Berechnung und Simulation im Fahrzeugbau, Würzburg, 2004 [6] Binner T., Reister H., Weidmann E.P., Wiedemann J. »Underhood Temperature Analysis in Case of Natural Convection«, VTMS 7, SAE 2005-01-2045, Toronto 2005 [7] Binner T., Reister H., Weidmann E.P., Wiedemann J. »Aspects of Underhood Thermal Analyses«, in Hucho/ Wiedemann (Editors), Progress in Vehicle Aerodynamics IV, Numerical Methods, expert Verlag, Renningen 2006 [8] STAR-CD User's manual version 3.15/ Methodology,Computational Dynamics Limited, London, 2001. [9] INTES, PERMAS user guide, Version 10.00, INTES GmbH, Stuttgart, 2004 [10] CFDLink, [11] Piperno, S., Farhat, C, »Design and evaluation of staggered partitioned procedures for fluidstructure interaction simulations.«, INRIA, Rapport de recherche no. 3241, 1997 [12] Fraunhofer Gesellschaft SCAI, MpCCI user guide, St. Augustin, 2004




Mark Pelzer ­ FLUENT Germany ABSTRACT The computational fluid dynamics code FLUENT provides best-in-class simulation solutions in its respective application area. With the MpCCI technology provided by Fraunhofer SCAI, the code can be used to simulate multi-physics like fluid-structure interaction (FSI) or thermal electro magnetism phenomena. This enables the user to simulate both sequential and fully coupled multi-physics applications with high accuracy. The communication of results between FLUENT and the finite element codes is handled by Fraunhofer SCAI technology MpCCI, making the FSI coupling between FEM codes and FLUENT accurate and user friendly. In the presentation the technical background of the multi-physics approach will be presented and several examples will be shown. INTRODUCTION So called »multi-physics« problems have been of high interest for recent years. Fields of combination of multi-physics are structural analysis, fluid dynamics, aeroelastics, heat transfer, radiation, electrodynamics, magnetodynamics, stamping, crash. Simulations have been able to address some of these combination within single tools. For example fluid dynamics, combustion, multiphase, conjugated heat transfer and radiation within FLUENT or structural dynamic, heat transfer and electrodynamics in FE tools are already feasible. For some cases one could also use CFD and FE tools' programmable interfaces to implement additional multi-physics. This is currently very often done in CFD codes since CFD tools often provides an option to account for moving geometry. If such on-board means are not sufficient the user can couple software tools manually. A prerequisite then is identical computational meshes in both tools. A one-way coupling can be established by exporting mesh and results from one tool in the appropriate format and import the result into the other tool. In case of a loose physical coupling of the problem even a two-way coupling can be achieved this way. A closely coupled problem or a problem that requires different computational meshes in both tools requires the use of a coupling software. This software has to handle data exchange, mapping of data from one mesh to the other and the synchronization of physical time in both tools. For some application also the deformation of the geometry is handled by the coupling software. Following we will discuss the various level of complexities by examples between finite element programs and the CFD software FLUENT using the coupling software MpCCI [1].


FLUENT FLUENT provides comprehensive modeling capabilities for a wide range of incompressible and compressible, laminar and turbulent fluid flow problems. Steady-state or transient analyses can be performed. In FLUENT, a broad range of mathematical models for transport phenomena (like heat transfer and chemical reactions) is combined with the ability to model complex geometries. Examples of FLUENT applications include laminar non-Newtonian flows in process equipment; conjugate heat transfer in turbomachinery and automotive engine components; pulverized coal combustion in utility boilers; external aerodynamics; flow through compressors, pumps, and fans; and multiphase flows in bubble columns and fluidized beds. To permit modeling of fluid flow and related transport phenomena in industrial equipment and processes, various useful features are provided. These include porous media, lumped parameter (fan and heat exchanger), streamwise-periodic flow and heat transfer, swirl, and moving reference frame models. The moving reference frame family of models includes the ability to model single or multiple reference frames. A time-accurate sliding mesh method, useful for modeling multiple stages in turbomachinery applications, for example, is also provided, along with the mixing plane model for computing time-averaged flow fields. Another very useful group of models in FLUENT is the set of free surface and multiphase flow models. These can be used for analysis of gas-liquid, gas-solid, liquid-solid, and gasliquid-solid flows. For these types of problems, FLUENT provides the volume-of-fluid (VOF), mixture, and Eulerian models, as well as the discrete phase model (DPM). The DPM performs Lagrangian trajectory calculations for dispersed phases (particles, droplets, or bubbles), including coupling with the continuous phase. Examples of multiphase flows include channel flows, sprays, sedimentation, separation, and cavitation. Robust and accurate turbulence models are a vital component of the FLUENT suite of models. The turbulence models provided have a broad range of applicability, and they include the effects of other physical phenomena, such as buoyancy and compressibility. Particular care has been devoted to addressing issues of near-wall accuracy via the use of extended wall functions and zonal models.

Dr.-Ing. Dipl.-Phys. Mark Pelzer Fluent Deutschland Birkenweg 14a 64295 Darmstadt Germany

Phone: +49 (0) 6151 / 3644 - 120 [email protected]




Various modes of heat transfer can be modeled, including natural, forced, and mixed convection with or without conjugate heat transfer, porous media, etc. The set of radiation models and related submodels for modeling participating media are general and can take into account the complications of combustion. A particular strength of FLUENT is its ability to model combustion phenomena using a variety of models, including eddy dissipation and probability density function models. A host of other models that are very useful for reacting flow applications are also available, including coal and droplet combustion, surface reaction, and pollutant formation models [2]. SOLVING MULTI-PHYSICS PROBLEMS WITH A COUPLING SOFTWARE A closely coupled problem or a problem that requires different computational meshes in both tools requires the use of a coupling software. This software has to handle data exchange, mapping of data from one mesh to the other and the synchronization of physical time in both tools. For some application also the deformation of the geometry is handled by the coupling software. MpCCI (Mesh-based parallel Code Coupling Interface) has been developed at the Fraunhofer Institute SCAI as a general software platform for coupling different simulation codes. MpCCI is a software environment which enables the exchange of data between the meshes of two or more simulation codes in their coupling region. Since the meshes belonging to different simulation codes are not compatible in general, MpCCI performs an flux conservation interpolation. In case of parallel codes MpCCI keeps track on the distribution of the domains onto different processes. MpCCI allows the exchange of nearly any kind of data between the coupled codes; e.g. energy and momentum sources, material properties, mesh definitions or global quantities. The intricate details of the data exchange are hidden behind the concise interface of MpCCI. Most of the commercial CFD/FEM applications allow users to add additional features, physical models, or boundary conditions via a programming interface. Within these user routines access to internal data structures is possible, either through subroutine parameters and global variables, or via internal modules for reading and storing data. MpCCI uses these capabilities for code adaptation. A user-subroutine called after each iteration or time step works as a hook to MpCCI [3].


Figure 1: MpCCI uses a client-server architecture. The »MpCCI Code-Adapter« links the FEM code to FLUENT through the MpCCI Coupling Server.

COUPLING EXAMPLES FEM-FLUENT (FSI) The coupling capabilities between FLUENT and FEM codes includes both sequential and coupled simulations. The following list of quantities gives an overview about the possible data exchange.


NPosition OverPressure RealWallForce Temperature WallForce WallHTCoeff WallHeatFlux WallTemp Nodal positions Relative pressure Relative wall force Ambient (fluid) temperature Absolute wall force





Wall heat transfer coefficient (film coefficient) FLUENT Wall normal heat flux density Wall temperature (nodal temperature) FLUENT FEM



Plotted in EnSight Gold from CEI. Courtesy of Deutz AG Figure 2: Prediction of temperature on the surface of a cylinder head. Red shows the hottest regions and blue the coldest regions.

Cylinder head thermal-stress prediction The prediction of thermal stresses in an cylinder head depends strongly on the influence of the fluid. The heated air enters the flow domain and heats up the material. The temperature gradient in the material causes thermal stresses. Via one way code coupling it is possible to send the fluid temperature and heat transfer coefficient from FLUENT to ABAQUS/Standard. With these values ABAQUS can calculate the heat flux and so the temperature distribution in the material. Afterwards ABAQUS calculates the thermal stresses. Red shows the hottest regions and blue the coldest regions (Figure 2). A comparison between an ABAQUS stand alone simulation and an ABAQUS-FLUENT coupled simulation shows a significant difference in the temperature distribution on the surface of the cylinder head. Vernay-VernaFlo® Fluid Flow Valve In this application a liquid flow through a valve is examined. The pressure inlet boundary condition changes from 0-40 psi and deforms the valve more and more (Figure 3). A steady state analysis was done in both FLUENT and ABAQUS. FLUENT started with a low pressure inlet boundary condition (Figure 4) and sent the relative forces after getting the converged flow field. ABAQUS calculates the new shape (Figure 4a) of the geometry and send back this information via MpCCI. After this the inlet bc. will be increased and the complete procedure is repeated until it reached the maximum value of 40psi at the inlet (Figure 4b). A comparison between experimental data and simulation results shows a good agreement (Figure 5).


Under low pressure Under high pressure Flow control value cross-section

Figure 3: Illustration of the undeformed and deformed shape of the fluid flow valve.

Figure 4a

Figure 4b

Plotted in EnSight Gold from CEI Figure 4: Contours of path lines of the flow field and von Misses stresses in the valve. Picture a) shows the starting point, picture b) shows the end shape of the valve.

Figure 5: Comparison between experimental and simulation results. The values are in good agreement.



SIMULATION OF A HARD DISK USING ANSYS AND FLUENT A hard disk is very complex flow system. The high speed of rotation of the disc causes a very turbulent flow system. This high-grade unsteady flow interacts with the reading head. The reading head starts to vibrate if the turbulence vortices hit a resonance frequency of System. To avoid this situation it is necessary to start a lot of difficult tests or the other option is to simulate this process with a coupled transient simulation between ANSYS and FLUENT. Figure 6 shows the complete FLUENT model. For this kind of simulations it is necessary to resolve the reading head geometrically. In general a very fine mesh (ca. 10 mill. cells) is required because of the turbulence effects in the system. For a realistic simulation one should use the large eddy simulation (LES) model for turbulence description. Just for simplification we used a much more coarse mesh (ca. 250.000 cells) and the RNG k-E model with enhanced wall treatment. The flow medium is air with a constant density. Figure 8 shows the ANSYS model. It was used the sparse solver with ca. 3000 linear elastic (solid 45) elements. The exchanged quantities at the surface of the reading head are the forces (from FLUENT to ANSYS) and the nodal displacements (from ANSYS to FLUENT). The resulting history plot of displacement for a special node on the tip of the reading head shows very small values (Figure 9). The behavior of the reading head is more or less similar to a damped harmonic oscillator. For a realistic simulation we would expect a different behavior. The reason for this damped answer from the structural part is the coarse mesh and the simple turbulence model. CONCLUSION With the capability of automatic data exchange via MpCCI it is possible to describe different complex multi physics systems. The interpolation method guaranteed a correct data exchange between FLUENT and different FEM codes. This avoids time consuming file based exchange between the codes and gives the flexibility to use different meshes. REFERENCES [1] Sitzki L., Kaufmann F.H., Dehning C.: »Fluid-Structure Interaction: Status and Strategy«, 2004 Conference »Virtual Product Development (VPD) in Automotive Engineering« [2] FLUENT Inc.: »FLUENT 6.3 User Guide Vol 2A«, 2005, 9-2 [3] Fraunhofer SCAI: »MpCCI 3.0 Overview and Release Notes«, 2004, 4


Figure 6: Complete FLUENT model for the hard disc

Figure 7: ANSYS model for the reading head

Figure 8: History plot of displacements for a special node on the tip of the reading head




Joachim Rang, M. Krosche, R. Niekamp, H. G. Matthies ­ Technical University of Braunschweig INTRODUCTION Problems of fluid-structure interaction contain of a fluid and structure part. Both problems are coupled by the boundary conditions on a common interface [MO95]. In the simulation of fluid-structure interaction often so-called partitioned methods are used, i.e. both problems are solved by different codes. In this talk the programs are controlled by the help of the component template library (CTL) [KNM03]. To use the CTL, component-based software systems are needed since the codes are controlled by the help of the CTL. Component-based software systems have the advantage that the implementations have a longer lifetime and that linkage of may be incompatible libraries is avoided. Since the fluid- and the structurecode use different meshes and different space and time discretizations the nodes on the common interface have to be modified with the help of MpCCI. In this talk algorithmic aspects of the interaction of the fluid-, the structure-code, MpCCI, and the CTL are explained. The CTL has the main task to distribute the subproblems and to control the loops for solving the linear systems of equations and the loop for the time discretization. THE PROBLEM In our case we have a coupling of fluid and structure (see [MW01]). The domain is given in Figure 1. The fluid part is modelled with the help of incompressible Navier-Stokes equations, i.e. f ( + (( - ) · )v) - · + p = rf 2 = (() ) · = 0. Moreover some boundary and initial conditions are given. We use the following notations: · f ... density of the fluid · v ... velocity · ... position of the referece ALE-coordinate system


Figure 1: Domain of the problem

· ... velocity of the referece ALE-coordinate system · p ... pressure · rf ... body force The structure is modelled by the following equations s¨u - · (FS) = rs, F = u, S = (tr E)I + 2E,

E = F F - I.

Furthermore we some boundary and initial conditions and the variables have the following meanings. · s ... density of the structure · u ... displacement · F ... gradient of the displacement · S ... second Piola-Kirchhoff stress · , ... Lamé moduli, E ... Lagrange-Green strain · rs ... body load

Dr. Joachim Rang Technische Universität Braunschweig Institut für Wissenschaftliches Rechnen Hans-Sommer-Str. 65 38106 Braunschweig Germany

Phone: +49 (0) 531 / 391-3007 [email protected]




SOLUTION TECHNIQUES There are two classes of solution techniques known in the literature. On the one hand-side we have direct or the monolithical approach. In this case we solve the whole problem with one software package. On the other hand a modular approach is possible, i.e. we solve the subproblems with different codes. Therefore this methods are often called partitioned methods. In our case we use for the fluid-part the Finite-Volume code OpenFOAM and for the structure part the Finite-Element code ParaFep. Since OpenFOAM and ParaFep use different meshes the nodes of the interface have to be approximated. Therefore we use the software-package MpCCI. Partitioned methods have several advantages if we compare them with direct methods. · existing software can be used · the best codes for the subproblems can be used (i.e. finite volumes for the fluid, finite elements for the structure) · the discretized problem is smaller than the whole problem Of course partioned methods have some disadvantages. The numerical results depends strongly on the iterative solution of the subsystems. Therefore in the literature is a difference between weak and strong coupling. We use strong coupling methods [MS02, MNS06] as · block-Jacobi · block-Gauß-Seidel · block-Newton THE CODES As we mentioned before the fluid part is solved with software package OpenFOAM. This code is open source and can be downloaded from the internet. The space discretization is done with help of finite volumes. The main application field of OpenFOAM is CFD. It supports mesh motion and topology changes. The structural part is solved with code ParaFep, developed at the University of Hannover. The code is open source, too, and uses finite elements. The main applications are structural mechanics problems. It performs adaptivity and algorithms can be solved in parallel. SOFTWARE COMPONENTS Clemens Szyperski and David Messerschmitt define software components as follows. Definition A piece of software offering (via an interface) a predefined service and which is able to communicate with other components. This is a rather general definition and so we mention some criteria for software components. · Multiple-use parallel execution · Non-context-specific exchangeable · Composable with other components · Encapsulated i.e., non-investigable through its interfaces


· A unit of independent deployment and versioning It is clear that we need a further code which controls our software-components. This is done by the so-called Component Template Library (CTL) [KNM03] which has the following features. · first partially realised as part of the parallel FE-code ParaFep at the Institute of Structural and Numerical Mechanics in Hannover, 1995 · originally thought as an enhancement of C++ a template (header) library like the STL = no further libraries or tools are needed · an implementation of the component concept with an RMI semantic similar to CORBA or Java-RMI. · a tool for component building from existing libraries · suitable for High Performance computing on parallel hardware · also usable by C or FORTRAN programs · compared with CORBA very easy to use It may a little bit surprising but the idea of software components is rather old as we see in the following. 1960 so-called software crisis 1968 The idea to componentize prefabricated software first published at the NATO conference on software engineering in Garmisch, Germany, titled Mass Produced Software Components by Douglas McIlroy · Subsequent inclusion of pipes and filters into the Unix operating system · The modern concept of a software component largely defined by Brad Cox of Stepstone, Objective-C programming language 1990 IBM: System Object Model Microsoft: OLE and COM today several successful software component models exist There are several reasons why we should use software components. · Growing number of (freely) available libraries and programs worth to be reused · Exchangeable software units · Support of distributed parallel run time systems · Avoids linkage of may be incompatible libraries · Longer lifetime of implementations Next we have to specify how the codes can interact. In this talk we present the Explicit Message Passing and Remote Method Invocation (RMI). First we consider the Explicit Message Passing, which is used by MPI and PVM. · Coupling programs by explicit message passing needs inserting of communication points into source code source and expertise for each program needed · No separation of communication and algorithm difficult maintenance of code



· Each new pair of coupling produces amount of programming The Remote Method Invocation (RMI) is used by the CTL has following properties. · Needs component framework · Keeps functional programming style · Exchangeability of coupled components · Type safe communication The CTL supports the following communication protocols/linkage types. · MPI (Message Passing Interface) · PVM (Parallel Virtual Machine) · TCP/IP (directly via sockets) · dynamic linkage · threads · pipes (to get through a firewall via ssh) · daemon (connect to a running process) · file (reasonable for dump of data to disc) In our case the interaction of the software components, i.e. of OpenFOAM, ParaFep and MpCCI, is done in the following way (Figure 2). The CTL controls the whole process. First some forces are given to the solver ParaFep which solves his problem and give some displacements values back. The CTL transforms these values into velocities and send them to OpenFOAM. This code solves the fluid part and gives some pressure values back. The corresponding nodes of displacement and pressure are send to MpCCI which give back some modified quantities.

Solver CTL vs pf rf us



Fluid OpenFOAM

Interface MpCCI

Structure ParaFep

solve (vf , pf )

translator (





solve (us , rs )

Figure 2: Control of the whole problem

Figure 3 shows the control of OpenFOAM. The code obtains some displacement values from the CTL. Then a mesh is generated. Then OpenFOAM solves the fluid problem using the function icoFoamAutoMotionMod. Finally some pressure values are given to the CTL.


OpenFOAM blockMesh displacements write mesh motion O CTL protocol f o a m pressures read

time t

mesh case motionU

call U

pBoundary read icoFoamAutoMotionMod

Figure 3: Usage of OpenFOAM

REFERENCES [KNM03] M. Krosche, R. Niekamp, and Hermann G. Matthies. A component based architecture for coupling optimisation and simulation software in a distributed environment. In Walter Dosch and Roger Y. Lee, editors, Proceedings of the ACIS Fourth International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD'03), 16­18 October, 2003, Lübeck, Germany. ACIS, 2003. [MNS06] H. G. Matthies, R. Niekamp, and J. Steindorf. Strong coupling methods. Computer Methods in Applied Mechanics and Engineering, 195:2028­ 2049, 2006. [MO95] H. Morand and R. Ohayon. Fluid-Structure Interaction. John Wiley & Sohn, Chichester, 1995. [MS02] H. G. Matthies and J. Steindorf. Partitioned but strongly coupled iteration schemes for nonlinear fluid-structure interaction. Comput. Struct., 80:1991­1999, 2002. [MW01] D. P. Mok andW. A.Wall. Partitioned analysis schemes for the transient interaction of incompressible flows and nonlinear flexible structures. In K. Schweizerhof W. A. Wall, K.-U. Bletzinger, editor, Trends in Computational Structural Mechanics, pages 689­698. CIMNE, Barcelona, 2001.




Holger Spiess and Pascal Bayrasy ­ Fraunhofer Institute SCAI THE MPCCI 3.0.6 CODE API The MpCCI Code API is an interface which allows to create code adapters for inhouse or research codes. These codes are then treated by MpCCI like any other code in the list of supported codes. It is thus possible to couple the code with any other supported code or even couple two inhouse codes using MpCCI. It is recommended to have at least a basic knowledge of the structure and procedures of MpCCI before starting to work on the code interface. The first step towards a code adapter is to define which goals shall be achieved: Which quantities shall be transferred, which kind of simulation ­ transient or stationary - shall be carried through, and whether the partner code can send and/or receive the selected quantities. Access to the source code of the inhouse code is a prerequisite, the mesh definitions must be read and the coupling manager functions of the adapter library must be called during execution. Of course, the quantities which shall be sent have to be written and the quantities which are received must be stored at appropriate places in your code. The MpCCI Code API consists of two basic parts, which are needed for a coupled simulation. The first is the integration into the MpCCI GUI, which allows the user to choose options and start your code. The second part is the actual code adapter which is responsible for data exchange with the MpCCI server.

inhouse code

GUI run integration control



code adapter




Figure 1: The two basic parts of the MpCCI Code API


An MpCCI adapter license for your own code is required to run a coupled simulation. SCAI also offeres the APIKit for easy code integration, which will be discussed below. It is recommended to proceed as follows for code integration: · Get the MpCCI Code Integration Kit »APIKit« with templates for GUI integration and adapter. · Create GUI integration. · Test GUI integration alone. · Write code adapter. · Test coupling with sample problem and analyze log files. A more detailed description of procedure and further information is given in the MpCCI manuals. GUI INTEGRATION All configuration files for integrating a code into the MpCCI GUI must be placed in a configuration directory, which is located in the subdirectory »<MPCCI_HOME>/codes«. This contains the files: gui.xcf...defines the GUI entries for the code. script for scanning the input file. script for starting the code. script for stopping the code. basic information on the code. an MpCCI subcommand. A template with all of the above files is part of the APIKit. As soon as a configuration directory is present, the code appears in the list of codes. The file »gui.xcf« is the central configuration file for the MpCCI GUI, which is written in XML format. It contains basic information like name and type of the code, the entries for Models and Go Step, a list of quantities which can be handled by the code and which information is passed to Scanner, Starter and Stopper.

Dr.-Ing. Holger Spiess Fraunhofer Institute for Algorithms and Scientific Computing SCAI Schloss Birlinghoven 53754 Sankt Augustin Germany

Phone: +49 (0) 2241 / 14 - 2647 [email protected]




The Scanner is responsible to retrieve basic information which is needed for coupling from the input file of your code. This includes space dimension, coordinate system, solution type, load cases, unit system, floating point precision and further code-specific information. Furthermore the possible coupling components must be identified. The mesh definitions are not needed, they are directly sent during code execution. All information gained by the Scanner is stored in a »*.scan« file. The Starter script is called when the »Start« button is pressed in the MpCCI GUI, it needs to know how to start the code and should pass command line options as defined in the MpCCI GUI. The Stopper script is called to stop a code when the »Stop« button is pressed and should cause the code to end the simulation as soon as possible. The Info script is an optional additional script to gather information for later use in Starter and Stopper. CODE ADAPTER The code adapter is a plug-in into a simulation code, which can also be based on a userdefined function interface. The adapter uses the MpCCI adapter library (mpcci.h, libmpcci. a) to communicate with the MpCCI server. This library is written in C and can be linked with C/C++ or FORTRAN code.

Figure 2: Code adapter structure

The code coupling manager functions must be called at specific times during the simulation. Before any other function, MpCCI_Message_init should be called to specify a list of output functions to be used by the adapter. Before the time loop is started, MpCCI_Init must be called to define the basic communication settings, the list of driver functions and initialize the connection with the MpCCI server. During the execution of MpCCI_Init the first driver function MPCCI_Driver_defineGrid is called to obtain node coordinates and mesh topology.


During the simulation the code must call MpCCI_Transfer for data exchange. The adapter then calls driver functions to either read data for sending or write data which was received from the partner code via the MpCCI server. Finally, MpCCI_Exit should be called at the end of the simulation to terminate the connection to the MpCCI server. MpCCI_Exit should also be used to abort the simulation in case of errors. MPCCI CODE INTEGRATION KIT »APIKIT« The APIKit shall simplify code integration. For this purpose it contains commented templates for all files in the configuration directory as well as for the code adapter. In addition a simple example code is included with source code, Makefiles and a sample coupling problem. All steps of code integration are described in detail in the Programmers Guide which is part VIII of the MpCCI 3.0.6 documentation. A description of the APIKit can be found there as well. Please note that the APIKit is not included in the standard MpCCI distribution but must be ordered separately together with an appropriate license.




Dimitar Stoyanov ­ Fraunhofer Institute ITWM ABSTRACT DDFEM is a parallel finite element solver for 3D linear elasticity. The solver possesses significantly better performance than other universal FE-solvers, especially for certain specific types of problems, e.g. in the field of micro mechanics. This higher performance is mainly due to the domain decomposition approach used to parallelize the finite elements method. The solver is highly portable over parallel computer architectures supporting the MPI-standard. DDFEM has been recently extended to handle heat problems, being able in this way to take into account the temperature effects appearing in the elasticity models: the thermal stress and strain. Another new feature in the recent development of DDFEM is the MpCCI-adapter implemented in a collaboration with the MpCCI group at Fraunhofer SCAI. This enables our elasticity code to be used in the numerical investigation of FSI-models. Here we present two test cases of coupling DDFEM with FLUENT using MpCCI: · 3D elastic flap in a flow channel; · heat transfer within an exhausting pipe, including the thermo-elastic effects. INTRODUCTION Nowadays, despite the variety of commercial solvers available on the market, the question of the design and development of effective numerical solvers is still open. It is a fact that many problems can not find their proper treatment using the existing (commercial) codes. Typical examples are large sized problems with complicated geometry, e.g. problems in micro mechanics, see [1], which necessarily require a parallel execution. In this sense, a specific-purpose in-house code, specially designed and programmed for a particular parallel architecture, can often provide much better performance than the existing general-purpose tools. The parallel 3D-elasticity solver DDFEM is an example for such an in-house development, undertaken to meet the needs of certain macro- and micromechanical problems. »DD« in the name stays for »domain decomposition« ­ this is the method for parallelizing the numerics. First and second order hierarchical local approximations are employed ([1]) within tetrahedral finite elements ­ because the latter are appropriate for automatic mesh generation, providing at the same time a good approximation for complicated geometries. Recently DDFEM has been extended to tackle heat problems as well.


The MpCCI driver for DDFEM is a new development also, which makes possible to use DDFEM in FSI simulations employing the so called weak coupling. Note, that in some observable future it is rather not very probable some general-purpose (commercial) package with strong coupling between the flow- and solid solvers to appear. Therefore, tolls like MpCCI, technically facilitating and intermediating the weak-coupling approaches, are indispensable. The present paper consists of two main parts: first we will briefly present the elasticity solver DDFEM, its basic applications and performance. Then two test examples of coupled FLUENTDDFEM simulations via MpCCI will be discussed. DDFEM: BRIEF DESCRIPTION, BASIC APPLICATIONS, PERFORMANCE DDFEM has been designed as a reliable, portable and high-performance application for parallel computers. Its C++ implementation follows the object oriented approach. The solver references two external libraries: · the mesh partitioning in subdomains uses ParMETIS, see [2], and · the (parallel) iterative solution of the assembled linear system is performed via PETSc [3]. The iterative linear solver, employed in DDFEM, is the conjugate gradients method with (block-)Jacobi preconditioning for the elasticity and GMRES for the heat equation. Both methods are provided by PETSc as routines. The higher performance of DDFEM compared to the one of many other FE-solvers, including commercial ones, is achieved via effective parallelization: with exception of the input, all stages of the numerics are performed in parallel. The serial input in many cases can not be avoided because most of the mesh generators produce a global mesh description ­ i.e. for the whole domain. After reading it and partitioning the mesh into subdomains, the local discretization and further the linear system solver work in parallel. This standard approach of a posteriori mesh partitioning, mentioned so far, is not always appropriate. In some cases, e.g. in micro mechanics, an a priori partitioning would be more suitable and DDFEM provides it. Another advantage of the solver is the higher order approximation with hierarchical shape functions we use. It allows a basis of a degree k+1 to be obtained as a correction to that of degree k. The advantage in this case, contrary to the lagrangian approach, is that the higher degree basis should not be entirely reconstructed (recomputed), but only the necessary

Dr. Dimitar Stoyanov Fraunhofer Institute for Industrial Mathematics ITWM Fraunhofer-Platz 1 67663 Kaiserslautern Germany

Phone: +49 (0) 631 / 31600 - 42 68 [email protected]




corrections should be found. Further details can be found in [1]. A special input language for DDFEM has been designed, where the mesh description is a list of nodes and elements, and the boundary condition specification is always local, i.e. related to a certain finite element (correspondingly to its edge, face) or to a node. DDFEM has been used at ITWM as both · a standalone elasticity solver for complex large-sized problems in macro- and micromechanics (Figure 1), and as a · FE-solver, iteratively referenced to perform structural analysis in a shape-optimization loop (Figure 2).

Figure 1: Micro mechanics: a fibre structure

Now, extending it with a heat solver, one can resolve models including temperature effects. Both solvers are independent and can be separately referenced. If the coupled problem is solved, then the heat solver is first referenced and after obtaining the temperature field one could resolve the elasticity model, taking into account the thermal strain effects also. Typical geometries resolved by DDFEM can be seen on the figures. Figure 1 represents a complex structure, containing fibres of three different materials. DDFEM is used to calculate the effective elastic properties of the sample: i.e. to »find« an energetically equivalent »reference« material. In fact this means to solve six boundary value problems for the domain. Such problems are computationally expensive, require parallel execution because of their large size (usually ~5-6 millions unknowns), and also need effective ways to handle the complicated geometry. To meet this requirements DDFEM first reads the so called voxel description of the domain (voxel is a »small« cube, located at a point). Note, that such a description naturally arises from the computer tomography images, which are usually the source for the digital reconstruction of such real materials. An a priori domain partitioning, e.g. using cutting planes, would be more suitable for such problem and DDFEM performs it. Then in parallel the numerical mesh is generated within the subdomains by subdividing each voxel into five tetrahedra and finally the solution starts to obtain the effective elastic tensor. Figures 2 and 3a represent macroscopic geometries. Particularly the one on Figure 3a is an outcome of a shape optimisation loop, where the optimisation tool ­ after reshaping the


Figure 2: Macroscopic geometry: closing plate

Figure 3a: Macroscopic geometry subject of a shape optimization

domain on each new iteration, e.g. by »drilling« a new hole - refers to DDFEM as a structure mechanics solver. These examples use global tetrahedral meshes with further a posteriori partitioning. Coupling heat and elasticity gives the possibility to model a large set of physical phenomena. Figure 4 presents the solution of a simplified model of a casting process, where the solidification heat is introduced via the source term in the heat equation. Then, after obtaining the temperature field, one can calculate the (elastic) deformations in the metal cast. The geometry of the problem can be seen on Figures 4a and 4c a metal alloy cast surrounded by a mold of sand. A source of a (latent) heat is associated with the middle rib of the metal cast and on the outer surfaces of the sand mold a convective cooling via the surrounding air is supposed.

PERMAS interface (*.uci, *.dat)


PERMAS interface (*.post)



DDFEM input language


Figure 3b: Shape optimization loop with TOSCA using DDFEM as elasticity solver



Figures 4a and 4c represent the temperature distribution on the mold surface and inside it correspondingly, while on Figure 4b the displacements of the metal alloy cast are given.

Figure 4a: Temperature distribution at the mold surface

Figure 4b: Displacements in the metal alloy cast

Figure 4c: Temperature distribution in the cast and in the cross section of the mold


The performance of the numerical procedures is extremely important in the practical simulations. For example, the repeating references to DDFEM in the shape optimization loop (see Figure 3b) increase significantly the performance requirements before the solver. We may say that DDFEM successively meets them and it replaced practically entirely some other commercial solvers used before in the shape optimisation iterations at ITWM on our parallel distributed architectures. In fact the real speed up is up to 15-20 times in favour of DDFEM, see [1]. Here by »real« we mean the acceleration obtained in a normal everyday working environment for real problems of middle size. The core of the DDFEM numerics has been shown to scale very well up to 1000 CPUs, see Figure 5. This has been tested by a special bench-mark version of the solver, see [1], where one considers pre-defined geometries of a simple form ­ a combination of spheres, cylinders, etc. Such geometries could be easily partitioned a priory and therefore the mesh generation within each subdomain could be also performed in parallel, providing in this way very good scalability for the whole combination of the grid generator and the FE-solver. The scalability in the real applications is also good enough, but the serial input of the mesh puts some upper limit and is a kind of performance bottle-neck.

4 x 10




3 Total Flops/sec (reported by PETSc)












500 Num PEs






Figure 5: DDFEM scalability ­ up to 1000 CPUs (IBM RS6000 Power 3)



PROGRAMMING A MPCCI ADAPTER FOR DDFEM We will skip the technical details here. Instead it is better to say that to create an own adapter is quite achievable task, providing that a detailed documentation of the API-functions of MpCCI is available (which soon will be the case). The necessary extensions of the source code of DDFEM, caused by the adapter, were rather auxiliary and certainly did not affect the overall design of the solver. FSI SIMULATIONS WITH DDFEM AND FLUENT The following test models are a kind of »classics« for MpCCI. Although the geometry is not that complicated, the complexity of a FSI-simulation is valid for the both examples. 3D elastic flap This is the example from the MpCCI coupling cases tutorial. The geometry of the channel, the position of the flap inside it, and the flow mesh can be seen on Figure 6a. The calculated pressure by the flow solver (FLUENT) at the flap walls is further considered as distributed load over the flap surface, and then the solid solver (DDFEM) calculates the displacements, i.e. the new position of the flap in the channel. The transfer quantities ­ flow pressure and flap nodal coordinates ­ are transmitted by MpCCI on each time iteration. The size of the solid problem is about 90 000 unknowns. The flap positions on iterations 0 (initial position), 2, and 9 could be seen on Figures 6b, 6c, and 6d correspondingly. The residuals of the FLUENT solution are presented on Figure 6e.

Figure 6a: Configuration of the flap within the channel

Figure 6b: Initial displacements of the flap on iteration

Figure 6c: Flap displacements, iteration 2

Figure 6d: Flap displacements, iteration 9


Scaled Residuals (Time= 9,2750e-02)

FLUENT 6.3 (3d, pbns, dynamesh, ske, unsteady) Feb 07,2007 Figure 6e: Fluent residuals, 3D elastic flap Exhausting pipe As a physical model this is the same example as the well known »exhaust manifold« of the MpCCI meetings, but in our case the geometry is more simplified. We consider an inflow of air with a temperature of 300º[C] in a metal tube ­ a kind of exhausting pipe, see Figures 7. The outer surface of the pipe is a subject of convective cooling by the surrounding air, the inner surface of the pipe is the interface between the flow- and the solid domains. The inlet is on the right-hand side of the pictures, the outlet ­ on their left-top part. The goal is to investigate both the fluid and solid for a steady-state temperature distribution and the resulting thermal strain and stress in the metal pipe. As a flow solver we use FLUENT, as a heat and elasticity solver in the solid ­ DDFEM. On each iteration the flow solver calculates the heat flux normal to the walls inside the pipe and this flux is then transmitted by MpCCI as a boundary condition for the heat equation solver in the solid. Solving for both the heat and the elasticity equations in the solid, one finds the temperature distribution on the internal wall of the pipe and the new nodal positions at the interface, which are transferred back to the flow solver by MpCCI. The iterations continue until steady-state for the flow is reached. Figures 7a, 7b, and 7c represent the final flow solution ­ the velocity, temperature, and pressure fields correspondingly, on the Figure 7d one can see the residual history. Figure 7e represents the final temperature distribution in the solid, while Figure 7f give the displacements.



CONCLUSION This paper presents two examples for coupled fluid-structure simulations between FLUENT and the elasticity solver DDFEM via MpCCI. The heat- and elasticity solver DDFEM is shortly introduced and the implementation of a MpCCI driver for the solver is briefly discussed. Finally two test cases for FSI are considered: 3D elastic flap in a flow channel and a simple model of an exhausting pipe.

Figure 7a: Velocity field within the pipe

Figure 7b: Temperature distribution of the flow within the pipe

Figure 7c: Pressure distribution of the flow within the pipe

Figure 7d: Fluent resudual history

Figure 7e: Temperature distribution within the solid pipe

Figure 7f: Displacements of the solid pipe


ACKNOWLEDGEMENTS For Figure 5 this research used resources of the National Energy Research Scientific Computing Center, which is supported by the Office of Science of the U.S. Department of Energy under Contact No. DE-AC03-76SF00098. Special thanks also to Inga Shklyar, Fraunhofer ITWM, who helped to automatically produce the descriptions of the fluid-solid interfaces in the coupled simulations. Many thanks to the MpCCI team at Fraunhofer SCAI, particularly to Dr. Holger Spiess, for the help and assistance to program the MpCCI code adapter for DDFEM. REFERENCES [1] Andrä, Stoyanov, Error indicators in the parallel finite element solver DDFEM, (Berichte des Fraunhofer ITWM, No 83, 2006). [2] METIS: Multilevel Partitioning Algorithms, ( [3] Portable, Extensible Toolkit for Scientific Computation, ( petsc/petsc-as/).




Felix Lippold ­ University of Stuttgart IHS ABSTRACT Current trends in the development of new technologies for renewable energy systems show the importance of tidal and ocean current exploitation. But this also means to enter new fields of application and to develop new types of turbines. Latest measurements at economically interesting sites show strong fluctuations in flow and attack angles towards the turbine. In order to examine the dynamical behaviour of the long and thin structure of the turbine blades, coupled simulations considering fluid flow and structural behaviour need to be performed. For this purpose the CFD code FENFLOSS, developed at the IHS, is coupled with the commercial FEM-Code ABAQUS. MpCCI is used for steering of the simulation and managing the data transfer between the two computational fields. Since the CFD domain has to be modelled in a certain way, the grid size tends to be in the range of about one million grid points. Hence, to solve the coupled problem in an acceptable time frame the CFD calculations have to run on more than one CPU. Whereas, the structural grid is quite compact and, does not request that much computational power. The implementation of the code driver for the inhouse CFD code FENFLOSS for the current version MpCCI 3.0.5 is presented. Further issues concerning programmer's interfaces and moving grid algorithms for CFD codes are mentioned. Furthermore, the setup of the technical and computational problem as well as the results obtained on a distributed computational grid will be discussed. BASIC EQUATIONS In order to simulate the flow of an incompressible fluid the momentum equations and mass conservation equation are derived in an infinitesimal control volume. Including the turbulent fluctuations yields the incompressible Reynolds-averaged Navier-Stokes equations, see Ferziger et al.[3]. Considering the velocity of the grid nodes UG due to the mesh deformation results in the Arbitrary-Langrange-Euler (ALE) formulation, see Hughes [5].

Ui ___ = 0 xi Ui Ui p ___ + (U U ) ___ = - 1 ___ + __ _ x j G x t xi j i ___ (U + ___)­ x x

i j i



ú jú i



Reynolds Stresses


The Reynolds Stresses are usually modelled following Boussinesq's vortex viscosity principle, see Boussinesq [1]. To model the resulting turbulent viscosity, for most engineering problems k- and k-w-models combined with logarithmic wall functions or Low-Reynolds formulations are applied. The discretization of the momentum equations using a Petrov-Galerkin Finite Element approach, see Zienkiewicz [10] and Gresho et al. [4], yields a non-linear system of equations. In FENFLOSS a fixed point iteration is used to solve this problem numerically. For each iteration the equations are linearized and then smoothed by an ILU-preconditioned iterative BICGStab(2) solver, see van der Vorst [9]. The three components can be solved coupled or decoupled, see Ruprecht [8]. Working on parallel architectures MPI is applied in the smoother to exchange data in the matrix-vector and scalar products, see Maihoefer [7]. The discretised structural equations with mass, damping and stiffness matrices M, D, and K, load vector f, and displacments u can be written as Mü + Du + Ku = f , see Zienkiewicz [10]. (3)

Fluid (CFD)

Fluid (CFD)


pressure loads deformation

pressure loads

Mesh (CMD)


Structure (CSM)

Structure (CSM)

Figure 1: Physical and numerical coupling model

Dipl.-Ing. Felix Lippold University of Stuttgart IHS Pfaffenwaldring 10 70550 Stuttgart Germany

Phone: +49 (0) 711 / 685 63267 [email protected]




COUPLING FLUID AND STRUCTURE Seen from the physical point of view a fluid-structure interaction is a two field problem. But numerically, the problem has three fields, where the third field is the fluid grid that has to be updated after every deformation step of the structure, to propagate the movement of the fluid boundaries, the wetted structure, into the flow domain, see Figure 1: threefield. The solution of the numerical problem can be arranged in a monolithic scheme, which solves the structural and flow equations in one step. This method is applied for strongly coupled problems, e.g. for modal analyses. Another scheme, suitable for loosely and intermediately coupled simulations, is the separated solution of flow and structural equations with two independent codes. In the separated scheme well proven codes and models can be used.

n Fluid 1 Structure 3



1. Deformations for timestep n 2. Update grid and integrate fluid field to timestep n+1 4. Put fluid loads to structure 5. Advance structural solution to timestep n+1



n Fluid 3 5 1

Fluid n-1/2 2 1 Structure n 4 n+1/2 3

n+1 2 4

· Weak coupling · First order (or less!) " " Subcycling with predictor for structure Time shifting


1. Deformed grid at time n+1/2 r n+1/2 = dn +



2.-4. see above Second order (almost) for structure middle point intergration



Figure 2: Exchange schemes for loosely coupled problems

However, a data exchange between the codes has to be arranged, including the interpolation between the two computational meshes. This is the reason why for engineering problems usually the second approach is employed. In order to account for the coupling and to avoid unstable simulations, some coupling schemes have been developed, e.g. see Farhat et. al. [2]. Figure 2 shows three of the most used schemes.


CFD-Start Matrix assembly ui Solve for ui Update p Solve turbulence

pressure loads (interpolation) subiteration


Picard iteration

Solve for y

deformations y

Update Mesh END

Figure 3: Coupling scheme

Using the simple coupling scheme results in the flow chart shown in Figure 3. The left part shows the solution of the flow problem as it is implemented in FENFLOSS. On the right side the structural solver and the grid deformation is sketched. The moving grid algorithm may be implemented either directly in FENFLOSS or as a stand alone software called during the coupling step. Here, an extended interpolation method, which is implemented in FENFLOSS, see Lippold [6], is used. COUPLING FENFLOSS AND ABAQUS WITH MPCCI In order to couple the in-house code FENFLOSS with the commercial software ABAQUS to perform fluid-structure simulations, an interface between FENFLOSS and MpCCI has to be available. The old MpCCI 2.0 SDK interface in FENFLOSS is replaced with a new one based on the adapter-driver connection required by the current MpCCI versions. In this case it means mixing the MpCCI-API written in C with classical FORTRAN code. The implementation of some special function calls in FENFLOSS is possible, creating an unflexible interface introducing new code into the program. Furthermore, the code then can only be used for special analysis types, i.e. fluid-structure simulations. Since, it is recommended to link the adapter and driver via an external library, the best and most flexible way is by using the shared object interface implemented in FENFLOSS. Once written, all algorithms needed for coupling the two codes, updating the grid, as well as coupling schemes can be implemented by user-subroutines without changing the simulation code itself. This yields a better code



quality and a faster development of the interface. The set-up and configuration for the graphical MpCCI interface is done with simple perl scripts. Figure 4 shows the architecture of the coupled codes. Because not all MPI releases provide the export option of the environment on the master node to the slave nodes, the MpCCI environment variables are exported via files written and set in the MpCCI code adapter.

MpCCI Initialisation Driver (API) Adapter

· CFD · API · User functions (MpCCI-Adapter)

ABAQUS v6.5 Solver Driver Adapter

· Initialisation · Data transfer · Fixed transfer points


· Scan · Start · Stop


· Grid-update · CFD-solution · API-calls (Data transfer)


· Data transfer · Interpolation · Log files


· Store data

COVISE Visualisiation

Figure 4: Architecture for coupling FENFLOSS with the shared object adapterdriver library and ABAQUS

CONCLUSION For most engineering applications it is found that loosely coupled schemes are appropriate to model fluid-structure interaction problems. Therefore, proven and established codes and models can be used. In order to couple the in-house CFD code FENFLOSS with the commercial structural analysis code ABAQUS MpCCI is used as an interface. Since the MpCCI SDK implementation cannot be reused, a new programmer's inter face is developed for FENFLOSS and the adapter-driver unit can be linked as shared object library. It turns out that this approach brings for effort in the beginning but saves time and problems in the end.



REFERENCES [1] J.Boussinesq, Théorie de l'écoulement tourbillonant et tumultueux des liquides dans les its rectilignes à grande section, Tome I - II, Gautier-Villard (1897). [2] C. Farhat, M. Lesoinne, P. le Tallec, Load and motion transfer algorithms for fluid/ structure interaction problems with non-matching interfaces, Computer Methods in Applied Mechanics and Engineering, No. 157, 95 (1998). [3] J. H. Ferziger, M. Peric, Computational Methods for Fluid Dynamics, Springer (third Ed. 2002). [4] P.M. Gresho, R.L.Sani, Incompressible Flow and the Finite Element Method, Vol. I, John Wiley & Sons (1999). [5] T. J. R. Hughes, W.K. Liu, T.K. Zimmermann, Lagrangian-Eulerian Finite Element Formulation for Viscous Flows, Computer Methods in Applied Mechanics and Engineering, 29, 329-349 (1981). [6] F. Lippold, Fluid-Structure-Interaction in an Axial Fan, HPC-Europa report (2006). [7] M. Maihoefer, Effiziente Verfahren zur Berechnung dreidimensionaler Strömungen mit nichtpassenden Gittern, PhD-Thesis, University of Stuttgart (2002). [8] A. Ruprecht, Finite Elemente zur Berechnung dreidimensionaler turbulenter Strömungen in komplexen Geometrien, PhD-Thesis, University of Stuttgart (1989). [9] H. A. van der Vorst, BI-CGSTAB: A fast and smoothly converging variant of BICG for the solution of nonsymmetric linear systems., SIAM J. Sci. Stat. Comp., 13, 631-644 (1992) [10] Zienkiewicz, O. C., Taylor, R. L., The Finite Element Method, Vol. I, McGraw-Hill (1989)




Christian Rümpler ­ Fraunhofer Institute SCAI and Albert Zacharias ­ Moeller GmbH ABSTRACT The electric current is omnipresent and indispensable in our life. But there are always threatening dangers connected with it. One of them is the electric short circuit and we have to protect people and facilities from its effects. This protection is often provided by power circuit breakers or motor protective circuit breakers. These devices are able to detect the short circuit, limit the current, switch it off and provide save insulation before any harm can occur. During the switching process of short circuits an electrically conductive plasma, the electric arc, exists between the metal contacts. The thermal, aerodynamic and magnetic interactions of this electric arc within the switching device are decisive for a safe current switch-off. These processes are simulated using the finite element code ANSYS to solve the magnetic field problem and the finite volume code Fluent to solve the fluid dynamics problem. Both codes are coupled using the coupling interface MpCCI. This work shows numerical results of an electric arc simulation using the example of a motor protective circuit breaker from Moeller series production. The simulation results show good agreement with experimental results. INTRODUCTION The electric current is omnipresent and indispensable in our life. It is delivering energy and allows us to operate and control a lot of applications. The simplest case of controlling is the switch-on and switch-off operation where electrical switches are required. But there are always threatening dangers connected with the electrical current. Current consumers can be overloaded, faulty equipment can be overheated and become sources of fire. An other danger is the electric short circuit and we have to protect people and facilities from these effects. This protection is often provided by power circuit breakers or motor protective circuit breakers (Figure 1). These devices are able to detect the short circuit, limit the current, switch it off and provide save insulation before any harm can occur.

Figure 1: Motor protective circuit breaker PKZM0


Electro-mechanical switches ­ like the motor protective circuit breaker presented here ­ mechanically open or close the electric circuit. They offer advantages compared to semiconductor devices; specifically with low power losses in current carrying operation (closed contacts) and save insulation in the open position. The presented motor protective circuit breaker operates electro-mechanically as well. During the switch-off operation an electrical arc occurs within the quenching chamber. This discharge is responsible for the current limiting, for a smooth decay of the current to zero and for the safe switch off. A special challenge is the interruption of short circuit currents because the small circuit breaker is loaded by currents of several kA within a short time. It is very important to know the behaviour of the electric arc inside the switch during the short circuit and to influence it properly. ELECTRIC ARC SIMULATION The electric arc After opening of the electrical contacts the current is not interrupted immediately. Because of the inductance L in the electric network that is always present (motor winding, cables) only a continuous change of current i is possible. Otherwise in case of discontinuous current changes an infinite voltage would be induced:

__ uL = L di dt


To continue the current after the contact separation a conductive media, the electric arc ­ which is a plasma discharge ­ is created between the separating contact pieces. Because of the high temperatures the gas is ionized and becomes conductive. The current can be carried by the plasma which leads to ohmic power losses and thus the temperature in the plasma is kept on a high level and the continuity of the discharge is ensured. In case of low voltage switchgear the medium used is air. Depending on temperature and pressure the oxygen and nitrogen of the air is dissociated and ionized and the thermodynamic and electric properties of the arc are strongly dependent on this. For instance the electrical conductivity (Figure 2, we used data from [1], further data can be found in [2],[3]) is very small below temperatures of 5000 K because no appreciable ionization occurred and thus not enough free charge carriers are available. Until about 25000 K the conductivity is increasing due to the increasing ionization.

Dipl.-Ing. Christian Rümpler Fraunhofer Institute for Algorithms and Scientific Computing SCAI Schloss Birlinghoven 53754 Sankt Augustin Germany

Phone: +49 (0) 2241 / 14 - 2135 [email protected]




Figure 2: Electrical conductivity of air at environmental pressure, comparison of different literature sources

Due to the difference of electric potentials and the electrical conductive path between the contacts an electrical current flows within the arc which is generating a magnetic field (self magnetic field). With the design and configuration of fixed and movable contact (these carry the current as well) also an magnetic field is generated (called magnetic blow field). The resulting force acts on the arc and drives it towards the metal splitter plates (Figure 4) where the arc is cooled, elongated and separated by the metal sheets. This results in a fast increase of the voltage drop. Now this quickly increased arc voltage ub is acting against the driving line voltage uN and is therefore influencing the current (see equation 2 with electrical resistance R ).

__ L di = uN ub R · i dt


In order to extinguish the arc quickly and a to effectively limit the current it is necessary that the arc voltage ub exceed the line voltage uN noticeably. SIMULATION The dynamic flow processes of the plasma are described by Navier­Stokes equations (mass balance, momentum balance, energy balance) combined with additional equations for the description of the electric and magnetic field. In the literature similar approaches can be found ([4]­[6]) for the simulation of low voltage switchgear. The solution of the flow problem is done by the commercial finite volume code Fluent [8] and the electro-magnetic problem is solved by the commercial finite element code ANSYS [9]. The equations for electric and magnetic field are coupled with the flow equations through the momentum source term Lorentz force density, the material property electrical conductivity and the energy source term power loss density. Depending on pressure and temperature the electrical conductivity is calculated via Fluent user defined function (UDF) and sent to


the partner code ANSYS. Using this material property the electric and magnetic fields are calculated offering the results Lorentz force density and power loss density. These values are sent back to Fluent for the next solution step. More details of underlying equations and assumptions can be found in [7]. The data exchange of the volume quantities between the simulation codes is done by the software MpCCI (Mesh based parallel Code Coupling Interface, [10]). Using this method it is possible to use specialized codes for each discipline with full functionality (e.g. nonlinear finite element methods for ferromagnetic materials) as well as suitable discretisation of the geometry within each code. Furthermore due to the separation of the whole task into different tasks the computation is accelerated. Thus the high numerical effort of complex three dimensional simulations can be handled better. Parallel to the modelling process experiments with a simple geometry (parallel runners) have been done for verification purposes. Experimental data (current, voltage, pressure, magnetic field) have been compared with simulation results to obtain inferences for the modelling process.

Figure 3 a:Calculated temperature in the middle plane compared to Figure 3 b:optical exposures, current 1000 A, time 300 s, 350 s, 400 s, 450 s

Figure 3 shows the simulated temperature compared with optical exposures of the parallel runner geometry. Further results can be found in [7], [11] and [12].

APPLICATION Motor protective circuit breaker PKZM0 Circuit breakers are often known from the fuse boxes at home. They switch off when lines are overloaded or a faulty device causes a short circuit. Manual switching is possible as well and provides for safe insulation. The functional principle of the motor protective circuit breaker is the same as that of the miniature circuit breaker, but all features and device characteristics are tuned to fulfil the special demands of motor protection and industrial application. Actuators are detecting the current flow continuously and in case of an overload current the contacts are separated depending on the amount and duration of this overload. For this contact separation a pre­tensioned latch is triggered. Furthermore in case of short circuit special



mechanisms enable a faster opening. The process of current switch off follows after contact separation within the quenching chamber (Figure 4). This chamber is specially designed for the fast switch off of high short circuit currents with prospective currents up to 50 kA. The switch is limiting the current to an effective value of 10 kA. The processes during the switch-off operation is as follows (see also Figure 4): · The contacts open, an initial arc is formed. · The special design of the current path leads to electrodynamic forces driving the arc towards the splitter plates. Due to the ferromagnetic effects of the iron material of the splitter plates the forces are amplified. · Between the metal sheets the arc is cooled, elongated and separated. The arc voltage is increased and will act against the current, limit the current and switch off the current (see equation 2).

Figure 4: Quenching chamber of the motor protective circuit breaker PKZM0, consisting of a connection in series of two separate quenching chambers Within some milliseconds this switch off operation is done where power in the order of 106 Watt is dissipated. Because of the temperature increase an over pressure of several bar inside the chamber is produced, which is released through special exhaust openings. The quality of a short circuit switch-off operation is basically influenced by the arc movement behaviour and the penetration of the splitter plates by the arc. Again the arc movement can be controlled by thermal, magnetic and aerodynamic manipulation of the arc. These processes are under examination in case of electric arc simulations. Simulation of the quenching chamber The simulations are done using ANSYS and Fluent simultaneously as described before. The Fluent model represents the quenching chamber, as shown in Figure 4. Effectively modelled is one half of a single chamber, a quarter of the geometry shown in the picture. To keep the computation time short the model consists of only 50000 cells. We used an explicit time discretisation scheme.


On the one hand the ANSYS finite element model represents the quenching chamber, which is geometrically identical to the Fluent model. On the other hand the surrounding air has to be modelled for the magnetic field calculation. Compared to the Fluent model the ANSYS model is build up coarser with about 35000 elements. Because of the nonlinear magnetic material behaviour of the iron parts (e.g. splitter plates) a nonlinear solution has to be obtained for a series of stationary steps. A fixed geometry is assumed. This means that the opening process of the movable contact is not included. The calculation starts (t = 0 ms) assuming completely open contacts. Here we present results from a calculation of 2 ms arcing time. The results we obtain are temperature, pressure and velocity distribution depending on time from Fluent as well as the distribution of current density, voltage, magnetic field from ANSYS. From the temperature distribution, as shown exemplary in Figure 5, one can evaluate the position and shape of the arc at different time points.

Figure 5: Calculated temperature distribution inside the quenching chamber. The isosurfaces of temperature representing the arc.

Figure 6: Prepared motor protective circuit breaker for arc observation

Verification Beside the arc simulation experimental investigations have been performed. To get optical records of the arcing process the optical access was needed. Therefore the sidewall has been perforated like a breadboard as shown in figure 6. The holes are covered by a transparent plate which is additionally sealed by an o-ring. The prepared switch was used in a short circuit interruption, where the arc was filmed with a high­speed video camera. Some pictures of this movie are presented together with the corresponding simulation results in figure 7 exemplarily.



The simulation started at t = 0 ms with fully open contacts patching a hot zone into the fluid model representing the initial arc, whereas the arc could spread out a little bit during contact opening phase in the experiment. Afterwards the arc moves towards the splitter plates. Beginning at 1.0 ms the arc penetrates the splitter plates almost, where a short reignition in front of the splitter plates can be seen on the left side.

Figure 7: Comparison of simulation results and experiment

SUMMARY The short circuit interruption is a special challenge in the development process of switching devices such as motor protective circuit breaker, power circuit breaker or miniature circuit breaker, particularly because the tests using real devices can only be done in a late stage of the development process. Here the arc simulation was introduced into the development process providing hints about the arc behaviour.


Even though a lot of effects are included in the simulation, the arc simulation model itself is still under development. The model is improved continuously and tested by means of adapted experiments. The application of the simulation model to series devices and experimental investigations enables a realistic estimation of the significance of the simulations for the development process. Additionally this short link to the application pushes the modelling process ahead.

REFERENCES [1] S. Selle, U. Riedel: Transport Coefficients of Reacting Air at High Temperatures. In: 38th Aerospace Sciences Meeting & Exhibit, American Institute of Aeronautics and Astronautics, Reno, NV, USA, 10.­13. January 2000. [2] J. Yos: Revised Transport Properties for High Temperature Air and its Components, AVCO Corporation, Technical Release (1967). [3] M. Capitelli; G. Colonna, A. D'Angola: Thermodynamic properties and transport coefficients of high-temperature air plasma. In: 28th IEEE International Conference on Plasma Science and 13th IEEE International Pulsed Power Conference, digest of papers, S. 694-7, Las Vegas, Nevada, USA, 2001. [4] Frank Karetta: Dreidimensionale Simulation wandernder Schaltlichtbögen. VDI Verlag Düsseldorf, Fortschritt- Berichte Band 21, Nr. 205, 1998. [5] Manfred Lindmayer, Erik Marzahn, Alexandra Mutzke, Matthias Springstubbe: Low voltage switching arcs ­ experiments and modeling. In: 15th Symposium on Physics of Switching Arc Brno, Czech Republic, 2003. [6] Hartwig Stammberger, Thomas Daube, Carsten Dehning, Michael Anheuser: Arc simulations in realistic low­ voltage arcing chambers. In: 21st Intern. Conf. on Electrical Contacts, Zürich, Switzerland, 9.­12. Sept. 2002. [7] Christian Rümpler, Frank Reichert, Hartwig Stammberger, Peter Terhoeven, Frank Berger Experimentelle und numerische Untersuchung des Lichtbogenlaufverhaltens In: Kontaktverhalten und Schalten, 18. Fachtagung Albert-Keil-Kontaktseminar, Karlsruhe, 5.-7. Okt, 2005 [8] Fluent 6: Fluent Inc., 10 Cavendish Court, Centerra Resource Park, Lebanon, USA ( [9] ANSYS, Rev. 10.0, ANSYS Inc., Canonsburg, PA, USA, ( [10] Mesh-based parallel Code Coupling Interface, ( [11] Christian Rümpler, Frank Reichert, Hartwig Stammberger, Peter Terhoeven, Frank Berger Numerical study of the electrical arc movement supported by experiments In: 23st Intern. Conf. on Electrical Contacts, Sendai, Japan, 6.­9. June 2006. [12] Frank Reichert, Frank Berger, Christian Rümpler, Hartwig Stammberger, Peter Terhoeven: Experimental studies of the arc behaviour in low voltage arc rail arrangements supporting numerical simulations In: 23st Intern. Conf. on Electrical Contacts, Sendai, Japan, 6.­9. June 2006.




Ian Lyttle, Benjamin Pulido ­ Schneider Electric MOTIVATION MpCCI has become an indispensable tool for multi-physics simulations, allowing for co-simulation using software tools native to each set of physics. Furthermore, the open architecture of MpCCI offers considerable possibilities for customization, permitting optimized solutions for specific classes of problems. At present, MpCCI is used principally by specialists who have training and experience with MpCCI. Greater use of the tool can be made, for example, if an everyday user of thermalanalysis software can work with an everyday user of electromagnetics-analysis software to make a co-simulation, without being an everyday user of MpCCI. Much of the co-simulation community is motivated by fluid-structure interaction (FSI) modeling, where surface regions are coupled between a fluid simulation and a structural simulation. However, at Schneider Electric we are concerned with problems where volume regions are coupled to each other. In this work, we consider problems where an electromagnetics simulation is coupled to a thermal simulation. For electrical-distribution equipment, the electromagnetics simulation calculates volume-based Joule (resistive) heating, based on temperature-dependent electrical resistivity. Accordingly, the thermal simulation calculates the volume-based temperature, based on the Joule heating. For this work, we use FLUX (offered by Cedrat) as our electromagnetics simulation tool; we use ICEPAK or FLUENT (offered by ANSYS) as our thermal simulation tool; we use MpCCI as our coupling tool. SOLUTION The first step in our efforts to automate the coupling procedure was to develop a naming system (within Schneider Electric) for co-simulation. It was decided that two levels of naming are required: the class level and the scheme level. We define a coupling class by the types of physics used. For example, the coupling class »magneto-thermal« refers to the coupling of an electromagnetics simulation with a thermal simulation. We define a coupling scheme by the broad assumptions we make within each of the physics. For the magneto-thermal example, the coupling scheme »harmonic-steady-state« identifies the electromagnetics problem as harmonic, and the thermal problem as steadystate. This class-scheme combination is abbreviated to MT-HSS.


Figure 1: Flow of information for the magneto-thermal coupling class




send mesh receive temperature solve emag problem send power losses

interpolate mesh interpolate data

send mesh send temperature

interpolate data

receive power losses solve thermal problem





Figure 2: Simplified flowchart for the harmonic ­ steadystate coupling scheme

Ian Lyttle Schneider Electric 3700 6th Street SW Cedar Rapids, IA 52404 USA

Phone: +1 (0) 319 368 - 3033 [email protected]




Our next step was to define the scheme further: All co-simulations of a given scheme have the same exchange of volume data and scalar data; processes follow the same flowchart. The data flow for MT-HSS is shown graphically in Figure 1, and as a flowchart in Figure 2. From the perspective of the MpCCI project, having defined the coupling class and scheme, the only items particular to a co-simulation are the names of the simulation files and the names of the regions to be coupled. Having established these definitions for the MT-HSS scheme, we wrote a set of scripts for FLUX and ICEPAK. To automate the construction of the MpCCI project file, we developed a simplified GUI in which the user specifies the following information: · Coupling class, e.g. magneto-thermal · Software, e.g. FLUX and ICEPAK · Coupling scheme, e.g. harmonic ­ steady-state · Release of simulation software, e.g. FLUX 9.3.2, ICEPAK 4.2.8 · Names of simulation files, e.g. FLUX problem, ICEPAK case · Runtime information, e.g. amount of memory for FLUX solver Once the coupling class and software are chosen, a tabbed window appears where the remaining choices are specified. By following a specific naming convention, a script is used to match the regions automatically. Upon the choice of coupling scheme, specific scripts used to control FLUX and FLUENT (the ICEPAK solver) are put into place. The simplified GUI constructs and saves MpCCI project file and launches the MpCCI GUI; the co-simulation is then launched from the MpCCI GUI. By providing a simplified GUI based on the coupling class and scheme, we are able to provide everyday thermal and electromagnetics users with a reduced set of choices as compared with the MpCCI GUI. As well, we are able to automate the matching of the regions, which can save time and possibility of error. For a well-defined coupling class and scheme, this lowers considerably the threshold of MpCCI experience needed to make an effective co-simulation. TEST CASES Shown in Figure 3 are the results for a simple problem, where a copper wire is subjected to an alternating current. On the left of this figure are shown the electromagnetic skin-effects; on the right are shown the temperatures for the wire and the surrounding air. We use this approach to solve problems with more-complicated geometries such as: · Busbar systems · Circuit breakers · Switchboard systems


Figure 3: Results for a simple MT-HSS co-simulation One of the features of this automation method is that the MpCCI setup for a switchboard system is no more time-consuming than for the copper-wire example. This may be notable, considering that a switchboard system may have hundreds of coupled regions, while the copper-wire has only one. POTENTIAL IMPROVEMENTS The developments previewed by Fraunhofer SCAI for MpCCI version 4 offer the possibilities to make more-sophisticated coupled analyses. One of these possibilities is »stronger« coupling for transient analysis; however, this depends on the partner software as well as MpCCI. Consider the transient FSI problem using, for example, FLUENT and ANSYS. Ideally, one could choose a time-step size for FLUENT, tF, according to the fluid problem and a time-step size for ANSYS, tA, according to the structural problem. One could also choose an MpCCI time-step size, tC, denoting the physical times for making exchanges. Presumably, the MpCCI time-step would be an integer multiple of both the FLUENT time-step and the ANSYS time-step. A flowchart of how such a stronger coupling might be implemented is shown in Figure 4. In this arrangement, there would need to be an iterative agreement between ANSYS and FLUENT at the end of every MpCCI time-step. Within the ANSYS cycle, the loading would vary linearly with time for the MpCCI time-step. Correspondingly, within the FLUENT cycle, the displacement would vary linearly with time for the MpCCI time-step.



Preserve loading at t=t0+ tC ANSYS t=t0 t=t0+ tA ... t=t0+ tC








t=t0+ tC



t=t0+ tF


t=t0+ tC Preserve displacement at t=t0+ tC

Figure 4: Outline for how »stronger« coupling might be implemented To implement such an algorithm, it will be necessary for the partner software (FLUENT, ANSYS, etc.) to be able to save and recover their solver states at arbitrary time-steps, while maintaining their connections to MpCCI. It may be important to be able to save the solver state for an arbitrary time-step, and not just the previous time-step. This is because the user may wish for the MpCCI time-step to be a multiple of an individual software's time-step. For those software for which this step-back procedure is not currently possible, we encourage their management and developers to work with Fraunhofer SCAI to find ways to implement this idea. ACKNOWLEDGEMENTS The authors would like to thank the following people and organizations for their ongoing technical support: · Pascal Bayrasy, Fraunhofer SCAI (MpCCI) · Rémy Perrin-Bit, Cedrat (FLUX) · Bill Wangard, Ansys Inc. (FLUENT) · Vincent Leconte, Schneider Electric ANSYS is a registered trademark of Ansys Inc. FLUENT is a registered trademark of Ansys Inc. FLUX is a registered trademark of Cedrat ICEPAK is a registered trademark of Ansys Inc. MpCCI is a registered trademark of Fraunhofer SCAI





Jasper Kidger ­ FLUENT UK ABSTRACT This paper presents the methodology and results of a co-simulation using MpCCI (v3.0.5), FLUENT (v6.3) and ABAQUS (v6.6-2). The simulation concerns a pressure-moulding application, where the interest lies in the movement of a flexible plastic membrane, forced into a mould by a high external pressure. Whereas the basic moulding process could be simulated using the FE code alone, in this case it is important also to account for the movement of the air within the mould cavity. The loading is applied as a uniform external pressure in the ABAQUS model, on the top surface of the flexible membrane. The FLUENT model simulates the air region inside the mould, which deforms as the membrane moves. INTRODUCTION This work concerns the prediction of the moulding process as a flexible membrane is formed into its final shape by the action of an external pressure. The cavity of the mould (into which the membrane is forced) has three ventilation holes of a very small diameter through which the trapped air must pass as the flexible part is formed into shape. This paper presents work performed by Fluent Europe for Amcor Flexibles. Amcor are one of Europe's largest suppliers and market leaders in flexible packaging, supplying to a wide range of food, beverage and health care markets. Amcor Flexibles are existing users of both FLUENT CFD software and the ABAQUS structural-finite element (FE) software. Traditionally Amcor have simulated this process by FE alone. However in order to gain a more detailed understanding, and improve their computational prediction capability, Amcor wished to explore the coupling of FLUENT and ABAQUS (using MpCCI) so that the movement of the air could be accounted for when predicting the structural response. A sketch of the practical arrangement is shown in Figures 1 and 2. The membrane is held across the top of a rigid mould. The cavity of this mould originally contains air at atmospheric conditions. A high external pressure is applied to the top surface of the membrane to drive the process. As the membrane takes up the shape of the mould, the trapped air is ejected through three small holes on the base of the mould. .


Applied Force (Uniform Pressure) Flexible Membrane


Rigid Mould

BEFORE Vent to Atmosphere

Figure 1: Sketch of Start of Moulding Process

Membrane PressureMoulded to Shape


Figure 2: Sketch of End of Moulding Process

ABAQUS MODEL If the pressure inside the mould cavity could be assumed to be constant (or indeed known as a function of time), then the pressure moulding simulation could be performed using a structural-FE code on its own, without the need for co-simulation with a CFD code. Simulations have been performed in this manner with ABAQUS-EXPLICIT (v6.6-2). Figure 3 shows the mesh on the top membrane ­ this is a square surface of 11,796 M34DR elements, held encastre around the perimeter of the square. Figure 4 shows the model from below, in which the rigid mould can be seen, along with the three vents from the mould cavity. This rigid surface is built of 11,724 elements.

Jasper Kidger Fluent Europe Ltd. Sheffield Business Park 6 Europa View Sheffield S9 1XH United Kingdom

Phone: +44 (0) 0114 2818888 [email protected]




Figure 3: Top View of the ABAQUS Model

Figure 4: Bottom View of the ABAQUS Model

Applying a constant pressure of 6 bar gauge on the top face (the same pressure was used for the co-simulation discussed later), and running the ABAQUS model on its own (hence assuming a constant, atmospheric, pressure on the lower face of the membrane) results in the deformation behaviour shown in Figure 5. This plot shows the movement, with time, of the node at the centre of the membrane. There is a significant amount of initial bouncing, but of decreasing magnitude. When the tip of the membrane reaches its final location on the floor of the mould, no further displacement of this node occurs, however the simulation continues whilst the membrane expands into the outer corners of the mould. The final deformed shape of the membrane, coloured with von-Mises stresses, can be seen in Figure 6.

Figure 5: Transient Profile of the Displacement of the Centre Node of the Flexible Membrane (ABAQUS Only)


Figure 6: Deformed Shape of Membrane from ABAQUS-only simulation (colour contours show von-Mises Stresses)

FLUENT MODEL In order to solve for the fluid flow within the mould cavity, it is necessary to use a CFD code capable not only of compressing its mesh as the domain changes shape, but also one capable of remeshing (adding or deleting cells) at very frequent intervals, as a result of the extreme deformation present here. Hence the unique dynamic mesh capability of the FLUENT CFD software (v6.3) was used for this work. At every timestep, as the new location of the deforming boundary is received from ABAQUS, FLUENT first performs a spring-smoothing action to compress/stretch the existing mesh to fit the new shape of the domain. Then, the quality of the resulting mesh is checked. If any cells are found either to exceed a prescribed skewness, or whose lengthscales exceed set minimum or maximum values, then these cells are deleted, and a new mesh grown in just that region of the fluid domain. Since the remeshing is purely a local issue close to poor-quality cells, and not a global remeshing, this makes for a very efficient process that is affordable to perform at every timestep during the simulation. The compute time for the remeshing work at each timestep is of approximately the same magnitude as the compute time for solving the fluidflow equations at that timestep. Figures 7a-d show cross-sections of the FLUENT mesh during the solution process. A small minimum length for the mesh is needed in order to handle the region close to the crater lip, hence comparing Figures 7a and 7b shows that as the deforming membrane moves downwards, a zone of compressed cells is evident below the membrane (the depth of this region is a function of the spring-smoothing constant applied to the dynamic mesh). As the deformation then becomes more severe (Figures 7c and 7d), it becomes clearer the extent to which cells have been deleted as the volume of the fluid region shrinks.



The solutions here were made on a full 360° representation of the mould. Although two symmetry planes do exist in the model, which would allow a 90° sector to be simulated, this was found to create problems because of the added constraints on the mesh along the common edge between the two symmetry planes. It is a requirement of FLUENT`s dynamic mesh model that the topology of the fluid domain cannot change during the simulation. As the flexible membrane takes the shape of the mould, consideration needs to be given to the way of representing the region where the two surfaces (membrane and mould) have come into contact. There are two ways in which this could be accommodated. The nodes along the two boundaries could be slid along as the contact point moves. However, the approach adopted here has been to shrink the dimensions of the rigid mould by 1 % in the ABAQUS model only. When the position of this membrane is transmitted to FLUENT, this preserves a small, finite, gap between the membrane and mould wall, in which a mesh can remain (as can be seen on Figure 7d).

Figure 7a: Initial FLUENT mesh on mid-plane

Figure 7b: FLUENT mesh on mid-plane

Figure 7c: FLUENT mesh on mid-plane

Figure 7d: FLUENT mesh on mid-plane


Other key details of the FLUENT model are as follows: · The air was modelled as an ideal gas · The Pressure Based Coupled Solver was applied. This simultaneously solves the momentum and pressure correction equations, and is ideally suited to applications (like this) where compressible effects are important on the resulting flow. · Turbulence was modelled using the standard, k-epsilon turbulence model. · The mesh was originally 139k cells, which reduced to 76k cells at the end of the simulation. · Spring-smoothing was applied to the dynamic mesh model with a spring constant of 0.7. Any cells with a skewness of greater than 0.9 were automatically remeshed, as were any cells above or below a set lengthscale threshold. · Timesteps of 10-6s were used, with the data exchange with ABAQUS, and potential remeshing, occurring at every timestep. MPCCI SETTINGS The co-simulation of ABAQUS and FLUENT was made with the use of MpCCI, version 3.0.5. This was set for a gauge-pressure based FSI simulation, with loads being transmitted across the deforming boundary between ABAQUS and FLUENT. Both codes were run in parallel (job run on a dual-CPU computer), by taking the explicit transfer option. ABAQUS (itself timestepping at 10-7s) was set to exchange data with FLUENT every 10-6s in a loose manner. All other options within MpCCI were left at their defaults. RESULTS Figure 8 shows the displacement of the centre-node of the membrane (in a similar manner to Figure 5), but comparing the results from the ABAQUS-standalone simulation (blue line) and the co-simulation with FLUENT (magenta line). The co-simulation ran until the centre node first met the base of the mould, at which point the topology of the problem changes and the middle vent became blocked. It can be seen that the duration of the event (from

Figure 8: Transient Profile of the Displacement of the Centre Node of the Flexible Membrane, comparing Standalone and Coupled Simulations



start, to centre node reaching base of mould) is about 50 percent greater in the co-simulation case than in the standalone case, and that the oscillatory behaviour of the membrane has continued throughout this time period. Figure 9 presents a snap-shot of the moulding process at a time of 0.002s. The two data-sets have been combined in this image, with the contour plotting of the membrane representing sheet thickness, and the centre-plane of the fluid region coloured by pressure. The ability of FLUENT's dynamic mesh technology to continuously deform and rebuild the mesh as the domain changes shape is clearly evident. Figure 10 shows the final point of the co-simulation, as the membrane meets the mould base, in which the final shape of the fluid mesh, and the thicknesses of the membrane, can be seen. The computations were performed on a 3GHz dual-CPU machine (with FLUENT and ABAQUS each running simultaneously, with one processor per code). The total compute time was 5 days wall-clock time. Over this period FLUENT performed 3,400 timesteps, ABAQUS 34,000 timesteps, and there were 3,400 2-way communications through MpCCI between the two codes.

Figure 9: Thickness of the Membrane, CentrePlane Fluid Pressure, and the CFD Mesh at a time of 0.002 seconds

Figure 10: As Figure 9, but at the end of the Transient Co-Simulation

CONCLUSIONS A fluid-structure interaction co-simulation has been performed for a pressuremoulding application in which a very large change in shape of the fluid region has been accommodated. In order to perform this simulation, ABAQUS FE software was used to simulate the response of a flexible membrane to an applied force which was constant on one side, but which varied on the other side as a result of the varying fluid conditions in the mould cavity. Simultaneously, FLUENT CFD software was used to simulate the fluid flow in the mould as the air was ejected, taking account of the moving boundary from the membrane,


and the very small exit apertures through which the trapped air could exit. The coupling of these two engineering simulation packages (FLUENT and ABAQUS) was made using MpCCI software. This enabled, at every time-step, the position of the structure to be transmitted from ABAQUS to FLUENT, and the forces on the moving surface to be transmitted from FLUENT to ABAQUS. FLUENT's unique dynamic mesh capability is ideally suited to this application, since arbitrarily large changes in shape of the fluid region can be resolved. At every timestep, FLUENT applied spring-smoothing to the mesh to accommodate the change in shape, as well as deleting and rebuilding the mesh in regions where cell skewness was too great, or cell dimensions exceeded prescribed thresholds. As a result of this co-simulation, it was demonstrated that the presence of the air in the mould cavity does have a significant influence on the moulding process, and therefore there are clear reasons why one should consider adopting a coupled fluid-structure interaction technique. ACKNOWLEDGEMENTS We are very grateful to the following people for their assistance in this work: · Amcor Flexibles Ltd, Bristol, UK, for provision of original ABAQUS input model. · ABAQUS UK, for technical support on use of the ABAQUS software. · Fraunhoffer-SCAI for technical support on issues relating to the ABAQUS-FLUENT coupling.




Albert Kurkchubatsche ­ ABAQUS US ABSTRACT As fluid-structure interaction (FSI) gains momentum in the industry, the use of established numerical solution techniques to enable coupled simulations is becoming relevant. We outline various solution techniques for solving fluid-structure interaction problems using ABAQUS and provide industrial examples from various industry sectors. ABAQUS provides significant build-in capabilities for modeling fluid. This includes ALE, equation of state, hydrostatic fluid elements, a capability to compute current and wave loading on submerged and partially submerged structures, and acoustic-structural capability. In cases where the ABAQUS fluid capability is limited, you may use one of the approaches described below to obtain a solution in conjunction with a computational fluid dynamic (CFD) code. Linear FSI includes the class of application where the structural deformations are small and the linearized displacements and fluid flow computation may be solved entirely within the CFD code. In such cases, ABAQUS can export structural stiffness, mass and damping matrices to the CFD solver. These approaches are CFD centric, that is the flow results and structural displacements may be readily obtained, however the structural response (e.g. stresses and strains) are not available. For cases where structural response is important you need to use a partitioned approach described below. Non-linear FSI includes the class of applications where it is important to consider the effect of structural non-linearity using finite element formulations. For non-linear structural applications we resort to a partitioned treatment of the coupled system in which the interaction effects are communicated in an explicit manner via Mesh-based Parallel Code-Coupling Interface, MpCCI. With the upcoming release of ABAQUS V6.7, ABAQUS supports a link with FLUENT and STAR-CD.



Unified FEA Realistic Simulation

Open simulation platform V5 Simulation Products and CAA V5 Partners »The World is Non Linear«

Albert Kurkchubatsche ABAQUS US 39221 Paseo Padre Parkway, Suite F Fremont, California 94538-1611 USA

Phone: +1 (0) 510 / 794 5891 - 103 [email protected]




SIMULIA: The DS Simulation Brand Vision and Mission The SIMULIA® vision is to make simulation an integral business practice that · enables engineers and scientists to reduce dependence on physical testing · supports multidisciplinary collaboration and · drives innovation in research and development The SIMULIA® mission is to be the leading provider of simulation solutions for engineering and scientific simulation · powered by an open, simulation centric framework that facilitates · collaborative decision making, · manages the simulation lifecycle and · achieves cross-disciplinary efficiencies FSI FROM AN ABAQUS PERSPECTIVE

Automotive Crash Simulation and Airbag Deployment using ABAQUS/Explicit

6-DOF solver Linear FSI Compliance matrix/eigen value approach to solving the structural problem inside a fluids code Nonlinear FSI Staggered Approach Structure and fluid equations solved separately with code coupling and mapping at the interface Suited for weak to moderately strong coupling physics problems. Implicit coupling well suited for tackling unstable FSI problems Specialized techniques 1. SPH: Meshless method 2. Immersed Boundary Techniques Suitable for problems where structural modelling is too complex or deformations are significant Monolithic approach Single set of equations for the fluid and structural domains Suited for most all coupling physics problems Examples: All

Structure represented in the fluids code as a 6 DOF entity

Suitable for rigid body motions in a fluid. Examples: IC engines, rigid valve movement

Suitable for linear structural problems Examples: Sloshing, vortex-induced vibrations

Examples: pulsatile blood Examples: Airflow, tire hydroplaning, bag/parachute dispensing deployment


FSI Technical Approaches Sequential Coupling: Heat Transfer File-based, sequential approach commonly used in industry via ODB API Interface to · FLUENT · STAR-CD · third-party codes · In-house codes Pro-Star (CD-Adapco) interface for ABAQUS

Rigid-body Displacements FSI problems involving rigid-body displacement are most effectively handled by the CFD solver · Most CFD solvers include 6-dof solver capability The ABAQUS co-simulation approach using MpCCI can be used for problems involving complex mechanisms using connector elements

Analysis of Landing Gear using ABAQUS



Linear Structural Response ABAQUS co-simulation approach using MpCCI can be used for problems with linear structural response Various approaches are offered by CFD partners · Limited to structural deformation only (no stress recovery)

A typical application: SWAGELOK pressure regulator Nonlinear Structural Response ABAQUS provides an open system approach to couple with third-party CFD software using MpCCI · any third-party CFD software compatible with MpCCI can couple to ABAQUS · Fraunhofer SCAI will support an adapter toolkit for in-house codes with the release of MpCCI 3.0.6 ABAQUS, Inc. fully supports and qualifies the following products · ABAQUS and ANSYS / FLUENT · ABAQUS and CD-Adapco / STAR-CD

Courtesy Fraunhofer SCAI


ABAQUS FSI PRODUCT UPDATE Status with FLUENT Next FSI product release in Q2, 2007 · ABAQUS 6.7-1 with FLUENT 6.3.26 and MpCCI 3.0.6 ABAQUS will jointly support the FSI solution with FLUENT using MpCCI · Releases to be qualified · ABAQUS to provide frontline support for joint customers ABAQUS/CAE FSI plugin module to be available to customers with Version 6.7-1 Status with STAR-CD First jointly qualified product will be available with · ABAQUS 6.7-1 with STAR-CD 4.02 and MpCCI 3.0.6 ABAQUS will jointly support the FSI solution with CD-Adapco using MpCCI · Releases to be qualified · ABAQUS to provide frontline support for joint customers STAR-CD Version 3.26 is compatible with ABAQUS version 6.5 / 6.6 via MpCCI 3.0.5 · Although we have not qualified this product, we will support mutual customers who are using these products INDUSTRIAL FSI EXAMPLES Industrial Use Case Overview Highlight use cases from industry segments using ABAQUS for FSI · Automotive · Biomedical · Manufacturing · Aerospace · Consumer goods · Offshore/Energy

UDEX simulation performed with ABAQUS acoustic / structure capability



Review how ABAQUS is currently being used to solve engineering problems involving FSI · Each industry segment will highlight use cases with different approaches to solve the fluid structure interactions. Automotive: Coupled Heat Transfer Coupled heat transfer analysis · File based sequential coupling approach · Open system approach via MpCCI

Pro-Star Interface to ABAQUS

MpCCI Interface with FLUENT

BIOMEDICAL: DRUG-ELUTING STENTS · Drug-eluting stents deliver medication to the surrounding tissue after implantation to help keep the passageway from re-closing


»Computational Modeling of Drug-Eluting Stents: Modeling Device and Drug Delivery«, ABAQUS Inc., FLUENT, US FDA researchers to be presented at 5th World Congress of Biomechanics, Munich, Germany, July / August, 2006 and BMES, Chicago, October 2006 The mention of commercial products, their source, or their use in connection with material reported herein is not to be construed as either an actual or implied endorsement of such products by the Department of Health and Human Services Biomedical: Heart Valves Analysis of heart valve motion · Unsteady pulsatile flow · Hyperelastic material properties for artery wall · 1/6th symmetry model · MpCCI interface with STAR-CD

Courtesy CD-Adapco Manufacturing: Flow devices Elastomeric components in fluid flow: · Minimize impact of inlet pressure variation for reliable and consistent operation · Nonlinear, large deformation structural analysis from ABAQUS · Turbulent flow, cavitation effects, remeshing from FLUENT · MpCCI interface with FLUENT

Model Courtesy: Vernay Labs



Aerospace: ZAERO Interface to ABAQUS Critical aerodynamic loads lead to instabilities: · Flutter analysis · Flight loads (trim) analysis

Example Courtesy: General Atomics Aeronautical Systems, Inc. Consumer Products: Packaging Drop test failure analysis of water bottle ABAQUS provides build-in capabilities for modeling fluid · ALE · Equation of State · Hydrostatic Fluid Elements


Courtesy: Bayer Material Science, LLC Offshore/Energy: Oil platform ABAQUS/Aqua · Current and wave loading on submerged and partially submerged structures

ABAQUS CAE PLUG-IN FOR FSI Complementary product to MpCCI GUI · MpCCI GUI facilities execution of FSI simulation between a CFD and structural analysis group · Models may be prepared by different groups with different software · Groups may be located at remote facilities The aim of the ABAQUS CAE plug-in to allow structural users to work in the familiar ABAQUS CAE environment to define · the domain geometry and mesh, · loads/boundary condition, · run the coupled simulation, and · post process results all within ABAQUS CAE. Plugin is available to ABAQUS users via an ABAQUS Answer · Plug-in currently supports FLUENT and in the future STAR-CD



Animation: ABAQUS CAE Plug-in for FSI

SUMMARY ABAQUS is being used for FSI across all major industry segments · We showed examples from various segments Most engineering analysis involve FSI in one form or the other · Different solution techniques are being used in industry to solve these FSI problems · MpCCI is being employed to solve the most complex FSI simulation As part of Dassault Systèmes, ABAQUS will be the foundation for multi-physics simulation and the SIMULIA vision · We are looking forward to continue our efforts with Fraunhofer SCAI





Philip Morris Jones ­ CD-adapco INTRODUCTION CD-adapco have been solving fluid flow problems for over 25 years. The software developed, STAR-CD and STAR-CCM+, have a wide range of physical models that span a wide range of flows, from supersonic flows and space craft re-entry to the mixing of foodstuffs such as yoghurt. The latest versions even go beyond computational fluid dynamics (CFD) to computational continuum mechanics (CCM) and so model heat transfer in solids, stress analysis, melting and solidification for casting etc. As is often found, despite having such a range of models applicable to different fluids and flow regimes, the majority of the flow simulations are concerned with simple flows of air or water. These are very common everyday fluids but are very different. Air at standard temperature and pressure has a density of 1.292 kg/m3 whereas water is much more dense at 999.8 kg/m3. The whole atmosphere, a depth of some kilometers, exerts a pressure of about 100,000 Pa at sea level but just 10m of water produces the same pressure. Many designs are aimed at containing or working with one of these fluids, but a real problem comes when an object is submitted to changing conditions that go from air to water and back. This can lead to failure through a number of mechanisms, but commonly it can be through peak loads exceeding the design limits or cyclic loads leading to failure by fatigue. This paper looks at such flows where a body and a free surface (the interface between a light and heavy immiscible fluids) interact and coupling these forces with a structural model of the body. FLUID STRUCTURE INTERACTION WITH A FREE SURFACE To correctly model fluid structure interaction with a free surface, you first need a number of building blocks in place: · Model free surface flows in a stationary domain · Model free surface flows in a moving domain · Couple the flow with a structural model MODEL FREE SURFACE FLOWS IN A STATIONARY DOMAIN One modelling approach used by CD-adapco to model free surface flows is called the volume of fluid (VOF) approach. In this technique we solve transport equations for the volume fractions of component fluids:

V i = __i V



The properties of the effective fluid (e.g. density) vary in space according to the volume fraction of each component i.e.




i i


Extensive validation studies have been performed to check the VOF implementation. A classic experiment is blunt body flow.

Figure 1: Blunt body in a flow tank

Figure 2: CFD mesh and free surface

Figure 3: Average water level in symmetry plane and along hull, and average velocities in symmetry plane in water and air

Figure 4: Comparison of wave profile along hull computed at TUHH with experimental data obtained at Ship Research Institute, Tokyo

Philip Morris Jones CD-adapco 200 Shepherds Bush Road London W6 7NL United Kingdom

Phone: +44 (0) 20 7471 6200 [email protected]




MODEL FREE SURFACE FLOWS IN A MOVING DOMAIN CD-adapco can model flows with moving meshes. In the most general case, a mesh can be made to translate, rotate or distort in any prescribed way, by specifying time-varying positions for some or all of the cell vertices. This type of mesh movement, which is sometimes referred to as Arbitrary Lagrangian-Eulerian, can accommodate a wide range of moving-boundary problems, such as the flow in a reciprocating-engine combustion chamber with moving piston and valves. For this type of problem, an additional equation called the space conservation law is solved for the moving coordinate velocity components. This relates the change in cell volume to the cell-face velocity. The simultaneous satisfaction of the space conservation law and all other equations of fluid motion facilitates the general moving mesh operations performed. Effectively this means that in addition to a velocity term caused by the fluid moving through the mesh, you get a contribution from the mesh moving through the fluid. The following shows an example of a domain showing rigid body motion. Here the domain moves and causes the fluid to slosh within it.

Figure 5: Computation and experiment of sloshing in a tank.


Figure 6: Geometry and locations of pressure taps

Figure 7: Comparison of simulation and SRI experiment for sway motion, 20 % full, 60 mm amplitude, 1.74 s period, at location P1

Figure 8: Comparison of simulation and SRI experiment for sway motion, 20 % full, 60 mm amplitude, 1.74 s period, at location P3

The next example is of a light wedge (nose angle 140°) in free fall from two positions enter into water (from PhD thesis of R. Azcueta, TU Hamburg-Harburg, 2001). This is a more complex case because the motion is not pre-determined. Obviously the wedge initially falls with gravity (so has an acceleration of ­1g) but as soon as it starts to interact with the water it has a strong resisting force. This type of calculation needs very strong coupling and needs to be done within the inner iterations of the CFD solver as the problem is highly non-linear and linearisation would produce gross overshoots.

Figure 9: Water entry of a wedge at 5° inclination



Figure 10: Comparison of vertical acceleration in simulation and experiment

Figure 11: Comparison of angular acceleration in simulation and experiment

COUPLE THE FLOW WITH A STRUCTURAL MODEL There are many options for the coupling, and many methods for each option. · The coupling can be very loose or one way with data exchange happening only once the simulation is complete · The coupling can be two way with data exchange at a number of points within the simulation · The coupling can be fully coupled: needed for highly non-linear problems. The following example shows a coupling of type (2) above. The wedge case shown previously is fully coupled as (3) above but for this case the wedge is modelled as a rigid body so its motion is defined by Newton's laws of motion. The structural model is build in ABAQUS and is a reinforced box meshed with hexahedra using the ABAQUS mesh tools. The STAR-CD model is of the surrounding fluid. The mesh shows where the box starts above the water and is initially surrounded by air. The meshes do not match.

Figure 12: ABAQUS model

Figure 13: ABAQUS Mesh


Figure 14: STAR-CD model of the fluid domain MpCCI is used to couple the two codes, with STAR-CD sending pressure to ABAQUS and ABAQUS sending displacements back to STAR-CD. The deformation of the box is very small compared to the size of the near wall cell so the whole deformation is accounted for in this cell without the need to spread the deformation further. Results are shown for displacement using a modified material to exaggerate the deformation and time vs. pressure history for a number of points in the flow domain which show that the maximum loading is around 14 or 15 bars.

Figure 15: Displacements within ABAQUS due to hitting the water

Figure 16: Pressure vs. Time for a number of points in the flow domain

CONCLUSIONS AND FUTURE WORK It has been shown that STAR-CD has extensively validated models for free surface flows with moving meshes. This capability has been coupled to the ABAQUS structural code using MpCCI and a coupled result has been shown. The next step would be to look at a much tighter coupling to further address the non-linear aspects of the coupling.




Noriyuki Kushida, Yoshio Suzuki, Osamu Hazama, Hitoshi Matsubara, Akemi Nishida, Fumimasa Araya, Tetsuo Aoyagi and Norihiro Kakajima ­ Japan Atomic Energy Agency JAEA ABSTRACT Center for Computational Science and E-systems of Japan Atomic Energy Agency (hereafter referred to as CCSE/JAEA) conducts the R&D of the virtual nuclear power plant simulator to improve security and safety. To realize such simulator, we are constructing complex structural analysis simulation program considering the influence from fluid or thermal. At the current state, we have established the structural analysis method for the extremely large-scale structure called assembly structural analysis (hereinafter referred to as ASA), which enables to treat a complex structure as an assembly of parts. ASA enabled us to analyze the whole outer shell of the High Temperature Gas Cooling Reactor named as High Temperature engineering Test Reactor (hereinafter referred to as HTTR). Concurrently with HTTR structural analysis, we are trying to perform structural-thermal coupling analysis of a steam generator of Fast Breeder Reactor (hereinafter referred to as FBR). CCSE/JAEA also conducts the R&D the information infrastructure for atomic energy research named Atomic Energy GRID InfraStructure (hereinafter referred to as AEGIS) including some applications works on it. One of the most important roles of GRID is to perform large-scale computing beyond one supercomputer. However, its slow network provides extremely long computation for usual parallel computing method. In ASA, each part could be solved almost independently, thus ASA could well work in spite of slow network. In this presentation, we will present the current state of our developed simulator and international collaboration related to AEGIS with Fraunhofer SCAI, HLRS (in Germany), IRIT and ORNL in United States. INTRODUCTION Japan Atomic Energy Agency, who is the only comprehensive laboratory in Japan, desired to ensure the long-term energy security and countermeasures to environmental problems. To respond to such expectation, we are currently constructing entire nuclear power plant simulator in recent days. In line with such background, entire nuclear power plant simulator must have following features: · It can simulate seismic response of quite complex structure such as nuclear power plant · It can simulate multi-physical system that is observed in nuclear power plant · It can run on world wide GRID environment, because entire nuclear power plant simulator requires huge computational resources


Deformation simulation that was related to nuclear power plant was sometimes curried out (for example, static[1] or dynamic[2] simulation of pressure vessel were presented by Shioya and Ogino, respectively), however, the target of them were limited to a part of entire nuclear power plant. According to the recent report, flaws that appeared in nuclear power plant are not limited in pressure vessel but also the outer part such as piping of cooling system. Because of complex structure of nuclear power plant, no one can predict the effect of any nugacious flaws. Therefore, we are trying to understand the behavior of entire nuclear power plant including building. Additionally, we must analyze multi-physical effect for more accurate simulation of nuclear power plant. As a result, simulating entire nuclear power plant requires quite large computer resource which is far beyond a current supercomputer. To provide such huge computer resource, we are developing Atomic Energy Grid Infrastructure (hereinafter referred to as AEGIS) as a hub of world wide GRID environment. In the following chapter, the features of entire nuclear power plant simulator are introduced: · Assembly Structure Analysis which is the framework to analyze complex structure · Weak coupling multi-physical analysis on GRID environment · Interoperability among different GRID systems ASSEMBLY STRUCTURE ANALYSIS As described in previous chapter, nuclear power plant consists of considerably complex structure and typical it has 100 thousand to 10 million parts. The finite element method (FEM) [3] has mainly been used in structural analyses with success. However, from our past experiences with the method in nuclear power plant simulations, seismic response simulations of such a huge structure in its entirety will require resources of memory and disk capacity of the order of 100 TB to 100 PB, which far exceed the hardware limitation of modern supercomputers. In line with this context, we hold following two issues for conventional FEMbased structural analyses: Difficulty in generating a mesh model of a huge structure in its entirety The mesh generation process usually requires interactive operations on a PC by a human. The preparation of models as illustrated in Figure 1 through the conventional approach requires four to six months because of geometric complexity. With regard to hardware,

Noriyuki Kushida Japan Atomic Energy Agency JAEA 4-49 Muramatsu, Tokai-mura, Naka-gun, Ibaraki 319-1184, Japan

Phone: +81 (0) 29 - 282 - 1111 [email protected]




Figure 1a: A CAD model Figure 1b: Finite element mesh data Figure 1: Mesh generation of a geometrically complex reactor model. modeling an entire nuclear power plant will require tera- to peta-byte memory. Constructing a model of an entire nuclear power plant would require too much manpower and too many computational resources to be practicable; this is a serious bottleneck in large-scale simulations. Difficulty in processing gigantic data sets of a huge structure in its entirety As illustrated by the red line in Figure 2, CPU time and memory storage consumed by the conventional approaches increase by the order of n2, where n is the model size. Therefore, they would rapidly exceed the hardware limitation of current super computing resources as the model size n increases [4,5]. To overcome these difficulties, we developed a numerical simulation framework using an approach called »assembly structure analysis« (ASA). The concept of this approach is somewhat analogous to »structured programming« proposed by Dijkstra [6], which helps to reduce the complexity of large programs. ASA allows: · Scalable generation of a mesh model To overcome the first difficulty, we developed a strategy called »from parts to the whole«. This strategy allows construction of the necessary finite element mesh data of an entire nuclear power plant in a part-wise manner. Each part is bonded together using existing mortar schemes to form an entire structure. · Hierarchical processing of a seismic response simulation To overcome the second difficulty, we developed a hierarchical simulation strategy. This approach reduces the amount of CPU time and memory storage from n2 down to n log n (blue line in Figure 2) because the entire simulation of a nuclear power plant can be considered as a linear combination of component-wise simulations: a component is considered to be an assembly of small parts.


Figure 2: Effectiveness of »from parts to the whole« strategy in ASA against conventional approach with respect to the required resources.

We present our numerical simulation framework based on the ASA approach and its application to the seismic response simulation of major components composing an actual nuclear power plant. MESH GLUING The mesh generation procedure encounters bottlenecks in the flow of finite element simulations. Conventional analysis must generate a mesh of an entire model all at once. This makes mesh generation very difficult as the geometric complexity and size of the model grow. We developed a strategy called »from parts to the whole« in ASA that resolves the bottleneck during the mesh generation process. This was realized by enabling the generation of mesh data of an entire nuclear power plant part by part. To enable part-bypart mesh generation, we incorporated penalty schemes into our strategy to mesh gluing. The nonconforming interfaces of each part are bonded together using penalty schemes. The »from parts to the whole« strategy in the ASA approach offers two promising features in the preparation of a huge mesh: · Part-wise meshing reduces the difficulties in meshing a complicated global geometry · Part-wise meshing removes the limitation to the global model size Simulation Results In this chapter, we show the example of our ASA simulation. In this simulation, the 20-second real data taken from the El Centro earthquake [7] was applied as input to the primary level. The results of analysis of the major components of a nuclear power plant were accomplished by the developed ASA framework. The 2.3 TB result data was visualized as shown in Figure 3.



Figure 3b Figure 3a Figure 3: Global views of seismic responses of (a) all major components and (b) just the reactor complex. WEAK COUPLING MULTI-PHYSICAL SIMULATION ON GRID As mentioned above, nuclear power plant must be considered as multi-physical system for accurate simulation. For example, as you can easily imagine, heat-flow-structure coupling system exists in pressure vessel. In present work, we developed two coupling codes as first step using MpCCI, heat-structure and fluid-structure coupling. Since these two coupling analysis are popular, we do not explain the detail of codes, but the way to build the GRIDenabled MpCCI is briefly introduced. In present work, older version of MpCCI was used and we must developed GRID-enable MpCCI using our GRID-enabled MPI implementation called »STAMPI«. In Figure 4, the illustration of developed system is shown. In our developed system, both fluid and structure analysis code were parallelized using vender-MPI in one computer and each codes were set for different computers. When value exchange period was coming, MpCCI was called with STAMPI, thus GRID-enabled coupling simulation was achieved. GRID INTEROPERABILITY In recent years, a number of high performance computing centers have recognized that GRID can play a great role to provide secure and huge computing environment. GRID is originally developed to provide barrier-free access among the institutes or universities with strong security. However, such barrier still remains so far, because GRID middle-wares are developed without any standard. In line with such background, we recognized that enabling the global GRID environment is the next issue. Up to now, JAEA is working with some foreign institutes to provide or enhance interoperability between their GRID and ours. Especially, interoperation with UNICORE of high performance computing center Stuttgart is the first experience for us. In that work, we employed »connection sever style« to provide interoperability which requires no modification to GRID middle-ware. Connection server was set between two GRIDs


Figure 4: Illustration of MpCCI-STAMPI system for fluid-structure analysis and translated the messages of them. The details of connection server should be referred in reference [8]. Following this successful result, we are currently developing interoperability with DIET at Institut de Recherche en Informatique de Toulouse. Concluding remarks In this presentation, our current state of entire nuclear power plant simulator was introduced. For entire nuclear power plant simulation, we set roughly three issues: · How to simulate large scale problem · How to simulate multi physical problem · How to obtain computational resources which satisfy above two issues To overcome these issues, our newly developed simulation method which was named ASA, multi-physical simulation code with MpCCI and computational environment which was build with GRID interoperability were introduced. REFERENCES [1] R. Shioya, G. Yagawa; 100 million DOF analysis using hierarchical domain decomposition method, Transactions of JSCES, No. 20010024, 2001. (written in Japanese) [2] M. Ogino, R. Shioya, H. Kawai and S. Yoshimura; Seismic Response Analysis of Nuclear Pressure Vessel Model with ADVENTURE System on the Earth Simulator, Journal of the Earth Simulator, Vol. 2, pp. 41 - 54, 2005 [3] O. C. Zienkiewicz, and R. L. Taylor; »The Finite Element Method (Fifth edition)«, Betterworth-Heinemann, 2000 [4] ADVENTURE project. [5] GeoFEM project. [6] E. W. Dijkstra,; »Structured programming«, In J. N. Bixton and B. Randell editors, Software Engineering Techniques. Pages 84-87, NATA Scientific Affairs Division, 1970 [7] Vibrationdata, [8] Yoshio Suzuki, Takahiro Minami, Masayuki Tani, Norihiro Nakajima,Rainer Keller, Thomas Beisel; Interoperability between UNICORE and ITBL, Workshop on Computational Grids and Clusters (WCGC`2006), at The 7th Int. Meeting on HPC for Computational Science, Rio de Janeiro, Brazil, July 10.-13., 2006




Jörg-Volker Peetz and B. Steckel ­ Fraunhofer SCAI; Thomas Sommer and K. Eulitz ­ Dresden Groundwater Research Center; N. Ettrich ­ Fraunhofer Institute for Industrial Mathematics ITWM; M. Müller ­ Ingenieurbüro für Grundwasser GmbH, Leipzig ABSTRACT The flood 2002 at the Elbe River showed that besides the apparent damages by surface waters, considerable damage was caused by groundwater and waters from the sewer system. Fast rising groundwater levels due to infiltration of overland flow or water from the sewer system can damage basements and can be a serious thread to the static of houses. 3ZM-GRIMEX logo The objective of the project 3ZM-GRIMEX is the development of a software system for the coupled simulation of three components: surface water, groundwater, and sewer water. The coupled system will be used in flood risk management. The implementation of the software system is based on MpCCI. Exchanged quantities are water heights and water fluxes between the components. The paper describes the coupled software system and the involved application codes with emphasis on the requirements for MpCCI. Challenges include that all application codes are interacting with each other, the codes are 1-, 2- as well as 3-dimensional, and the time-scales in the codes are very different. Furthermore, the paper outlines the use of MpCCI, including the development of suitable code adaptors with a coupling driver, the time step negotiation, the coupling algorithm, and a dummy code used for tests and as substitute. Finally, numerical results for a generic test model will be given. Simulations of the flood 2002 at Dresden are planned for 2007. THE 3ZM-GRIMEX PROJECT, GOAL, AND MOTIVATION The following text gives a report about the project »Development of a 3-Zone Model for Groundwater and Infrastructure Management After Extreme Flood Events in Urban Areas« or abbreviated 3ZM-GRIMEX [1]. This project has two main goals: (1) The simulation of flood events in the urban area of the city Dresden by coupling the calculation of groundwater, sewerage, and overland flow. (2) The development of a holistic flood risk management. Six institutional partners are working together in this project: · DGFZ Dresdner Grundwasserforschungszentrum e.V. · TUD Technische Universität Dresden · Fraunhofer-Institut für Techno- und Wirtschaftsmathematik ITWM · Fraunhofer-Institut für Algorithmen und Wissenschaftliches Rechnen SCAI · UFZ Umweltforschungszentrum Leipzig-Halle GmbH · Umweltamt Landeshauptstadt Dresden


The project started in March 2005 and is expected to finish in February 2008. It is funded by the Federal Ministry of Education and Research (BMBF) within the research focus RIMAX. The conception of the project was initiated after the catastrophic Elbe flood event in Saxony in the year 2002.

Figure 1: Causes of damage for the flood August 2002 in Saxony (after [2]) It was shown that serious damages were not only caused by the surface water but also by the groundwater and the water from the sewer system. Nearly half of the flood damages are attributed to groundwater and sewage. While the surface water reaches normal levels within days or weeks after the peak of the flood, the groundwater can have significant raised levels as long as one year after the flood event. Figure 2 shows a cross-section of the underground through the Elbe valley in Dresden. In the middle the river in its bed is depicted in light blue with the water-gauge of the flood in August 2002. The light blue line indicates the level of the groundwater for this flood event. The dark blue line marks the groundwater level a few weeks later. This illustrates that during a flood event there occurs a flood wave in the groundwater although with a slower dynamic. Likewise, the level of the groundwater rises due to infiltration of overland flow or water from the sewer system. Conversely, groundwater can infiltrate into the sewer system if the groundwater level is above the sewage level. Therefore, an integrated flood risk management in urban areas requires the consideration of all relevant flow processes including surface runoff, flow through sewer systems, and groundwater flow.

Dr. Jörg-Volker Peetz Fraunhofer Institute for Algorithms and Scientific Computing SCAI Schloss Birlinghoven 53754 Sankt Augustin Germany

Phone: +49 (0) 2241 / 14 - 2945 [email protected]




Figure 2: Flood and groundwater (after [3])

Figure 3: Participating codes

INTRODUCTION TO THE MpCCI CONNECTION IN 3ZM-GRIMEX The strategy of this project is to use established simulation programs for each of the three domains. These programs have been approved for their particular area of application. They will be coupled with each other with the MpCCI software [4] (see Figure 3). MpCCI manages the communication between the individual programs, i.e., the mapping between the different model geometries, the time synchronization, and the data exchange. Depending on the task, two different codes will be used for each, the overland flow (RisoSurf [5], TrimR2D [6]) and the sewerage system (Hamoka [5], Hystem-Extran [7]). Because the groundwater code (PCGEOFIM [8]) can calculate selected regions with finer spatial resolution, it can be used for large scale simulations, coupling with TrimR2D and Hystem-Extran, as well as for small scale simulations, coupling with Risosurf and Hamoka.

Figure 4: Surface waters, computation areas (Map: Dresden, Office for Environment)


For the urban region of Dresden the simulated area is divided into a global model and a few local models. The global model extends parallel to the river Elbe on each side of it (the area with the yellow background color in Figure 4) where buildings and streets are only roughly taken into account and the sewer system is modeled without details. Here the combination of the programs TrimR2D, Hystem-Extran, and PCGEOFIM is applied. The local models (illustrated by colored rectangles in Figure 4) are computed with details on the surface, such as buildings and curbs, and with more manholes in the sewerage system. In this case the combination of the programs RisoSurf, Hamoka, and PCGEOFIM is used. The identification and definition of the relevant water flows between the individual components of the flow system was an important step of the project, compare Figure 5.

Figure 5: Water paths

Figure 6: Example of water path

The flow system includes the diverse states depending on the expansion of the flood. Each of the individual simulation domains works with its own boundary conditions and sends one quantity to and receives one quantity in return from each of the other component. As an impressive example for one of the water paths in the flow system, the picture in Figure 6 shows a fountain-like outflow of water from a manhole in an already awash region of Dresden in August 2002. In order to set up the communication in the coupled computation for each bilateral exchange the geometrical part of the simulation model on each side and the quantities to be exchanged have to be specified. For example, for the coupling between the surface water program and the groundwater code the geometrical model parts are given by the potentially flooded elements (triangles or rectangles) of the surface model and the top side of the cells nearest to the surface of the groundwater model. The quantity sent by the surface program is the water height above ground at cell center in meters. The quantity sent in return by the groundwater code is the water velocity (or water flux per area) measured in meters per second. The same procedure is applied to each of the other two combinations: surface code with sewerage program and groundwater code with sewerage simulation.



In order to set up a scenario simulation of the 2002 flood, the simulation periods of the coupled codes have to be defined and synchronized. Some of the codes need a certain pre-coupling run-time. Also, since the surface code needs much more CPU-time for a certain simulation time interval, it will disconnect from the coupled computation soon after the peak of the river flood wave. On the contrary, the groundwater code has to calculate a simulation time interval of several months after the flood peak coupled with the sewerage program in order to observe the normalization of the underground water level. Furthermore, if a second flood occurs within a few months from the first, as it was the case in January 2003, the groundwater calculation has to include both flood peaks to get a realistic starting condition for this situation. (Compare Figure 7)

Figure 7: Coupling period

INTERFACE DESCRIPTION MpCCI DRIVER API AND CODE API In a coupled computation the coupled codes and the MpCCI coupling server processes run independently and (quasi) in parallel on one or several computers. For this project a driver for MpCCI was developed, which provides a simple interface as software library for the codes to be coupled. Nevertheless, each code has to be extended for the coupling. Such an extension is mainly a loop around the underlying simulation, which comprises the negotiation of the coupling time step and the exchange of the coupling quantities. Furthermore, an API for the access on the code internal data needs to be added to each code. This can be done either by re-programming, if the source code is available, or by a wrapper technology. The newly developed MpCCI driver offers a specialized API for accessing MpCCI for the simulation codes. This API includes initialization of the coupling, negotiation of the coupling time step, data exchange, as well as disconnection from the coupled computation (compare Figure 8). For the communication with the simulation program the driver uses the code API to be provided by the code owners. The driver also implements the algorithms for the coupling time step negotiation. For the development and testing of the driver and the code extensions a so called »dummy« code was developed. It serves three purposes: it provides an exemplary implementation of a code API and the computation flow in the main simulation loop; it is used as a testing


Figure 8: MpCCI-GRIMEX-driver tool for the MpCCI driver and its API; it can be used as a substitute (dummy) for any of the participating simulation programs by reading its geometrical model and the data to be sent from files. The code API of each coupling simulation program has to provide functions for describing the part of the geometrical model which takes part in the coupling. It has to identify the coupling region or part, to supply the number of nodes and elements, their identifiers, the coordinates of the nodes and the definition of the elements. The API further needs a function to check the negotiated coupling time step and finally functions to get and set the exchange quantities (see Figure 8). Blockade situations need to be avoided when more than two codes are coupled with MpCCI. For example, code A wants to communicate with code B, B is asking C, and C tries to contact code A. This can happen since the functions for the coupling time step negotiation and the data exchange wait for a reply from the partner code. Therefore, we introduced a »rule of way«. It states that when a code wants to negotiate the coupling time step or exchange data with two other codes at the same time the code has to do it in a prescribed order of the coupling parts. For the same reason each data exchange has to be preceded by a coupling time step negotiation. Since the three coupling regions in this project have different simulation dynamics, a customized coupling algorithm had to be invented. The programs for the surface runoff and the sewerage system with the faster dynamics have to couple more frequently with each other before they both couple with the groundwater. For their coupling a parallel coupling scheme is used, while they both utilize a sequential scheme for the coupling with the groundwater (compare Figure 9). As already mentioned, one method to connect a simulation code to MpCCI is the wrapper technology. For example, PCGEOFIM, the code for the groundwater, is adapted to the MpCCI driver in this way. PCGEOFIM is wrapped as a Python module with the help of the tool »f2py« [9].



Figure 9: Exemplary Computation Flow for 3-Code Coupling This allows for a fast wrapping because most work is done automatically. The main loop of PCGEOFIM remains; it needs only small program modifications. By using a »callback« -function as argument, a call back into Python is possible from any place in the code. PCGEOFIM runs in a thread, whose state is observed from the Python-wrapper. he data exchange is based on a FORTRAN 90 structure that holds the data to be exchanged. This principle has proven its usefulness also in other coupling projects. It makes it possible to build an object-oriented interface for a program that has been developed for more than 35 years. A Python interpreter was embedded into a C program by using Pythons C-API because there exists no native Python interface for MpCCI. The C-calls are handed over to Python and, if necessary, the change of data formats is handled in Python. VISUALIZATION OF RESULTS FOR A GENERIC MODEL Test and analysis of the results of the coupled simulation have their special difficulties. There may be visualization tools for each of the simulation programs but none for the integrated results of the coupled system. Additionally, no single person knows how to operate all of the coupled codes, let alone how to utilize the analyzing tools for each. Therefore, MpCCI contains a visualizer for the geometry of the coupling regions and the exchanged quantities. Unfortunately, it can not present not-exchanged results of one of the coupled codes such as the height of the water in the manholes of the sewer system. Hence, we decided to extract the results of interest and convert them into a format understood by a general purpose visualization program. We used the program called »paraview«, which is built upon the visualization toolkit »vtk«. The results presented here are calculated for a generic test model consisting of a plane with a swale and two hills behind and separated from it by a river. In the pictures shown here only the part of the plane with the swale is considered. The first picture in Figure 10 is a top view. It contains the dry surface of the ground in shades of brown color. The parts of the surface with water on it are shown in shades of blue color, depending on the height of the water:


the darker the blue, the higher is the water. Additionally, the height of the groundwater is indicated with semi-transparent tiles according to the top of the groundwater cells in the height of the groundwater level. The color of the tiles represents the water flux from the sewerage system to the groundwater (»q_KzuGW«).

Figure 10: Top view of the generic test model

Figure 11: The generic test model from below

Figure 11 shows a second view on the landscape of the generic test model now from below. This allows for a look on the sewerage system. The water-height in the pipes perpendicular to the surface (the manholes) is indicated by the blue color. The dry part is shown in brown. In this scenario the sewerage system is divided into two parts by a bottleneck between the third and the fourth manhole from the left. Due to the height of the underground water level, the water from the underground is infiltrating into the sewerage system. Therefore, in the right part of the sewerage system the water is pushed out through the manholes and flows onto the surface. In the Figures 12 to 14 the development in time of this test scenario is shown. The groundwater is transferred from the underground through the sewerage system to the surface. On the surface the water is converging at the swale and partly running away again through the part of the sewerage system on the left. Up to now, we have implemented a customized MpCCI driver for the coupling in 3ZMGRIMEX along with a »dummy« code as programming example and as testing replacement

Figure 12: Generic test scenario after 1800 s

Figure 13: Generic test scenario after 5400 s

Figure 14: Generic test scenario after 18000 s



for each of the application programs. Three of the five codes to be coupled in this project are already adapted to the MpCCI-GRIMEX-driver. With this coupled system first scenarios for a generic test model have been calculated and the results were analyzed with the help of 3D visualization by means of adaptation to a standard tool. After the coupled simulation system has been completed with the last two codes, tested, and investigated elaborately with further scenarios, computations with scenarios of the area of Dresden have to be done in order to develop a holistic flood risk management. REFERENCES [1] Web site of the project: [2] Huber, G.; Hiller, G. u. Braune, A.: »Konzepte des Hochwasserschutzes für die Bauten des Freistaates Sachsen im Historischen Stadtkern von Dresden«. In: LH DD / DGFZ (Hrsg.) (2003): »Hochwassernachsorge Grundwasser Dresden«. Tagungsband zum Statusseminar am 8. Oktober 2003, S. 57-61. [3] »Hochwasser August 2002: Auswirkungen auf das Grundwasser ­ Gesamtkonzeption zur Grundwasserbeobachtung/ -überwachung für den Großraum Dresden, 3. Etappe«. Dresdner Grundwasser Consulting GmbH, AG: StUFA Radebeul, Ref. Grundwasser, Dresden, 01.11.2003 (unveröff. Gutachten). [4] Web site: [5] Ettrich, N. et al.: »Risikomanagement für urbane Entwässerungssysteme ­ Simulation und Optimierung (RisUrSim) ­ Teilprojekt 1: Oberflächenwasser und Informationsaufbereitung ­ Abschlussbericht«. Fraunhofer-Institut für Techno- und Wirtschaftsmathematik ITWM, 2003. Schmitt, T. G. and Thomas, M.: »Risikomanagement für urbane Entwässerungssysteme (RisUrSim) -- Teilprojekt 2: Kanalnetz«. TU Kaiserslautern, 2003. [6] Fulford, J. M.: »Computational Technique and Performance of Transient Inundation Model for Rivers ­ 2 Dimensional (TRIMR2D): A Depth-Averaged Two-Dimensional Flow Model«. Technischer Bericht, U.S. Geological Survey, 2003. [7] Fuchs , L., Scheffer, C. and Verworn, H.-R.: »Modellbeschreibung: Hystem-Extran 6«. Institut für technisch wissenschaftliche Hydrologie GmbH (ITWH), Hannover, 2004. [8] M¸ller, M., Sames, D., Mansel, H. (2003), »PCGEOFIM ­ A Finite Volume Model for More?«, MODFLOW and More 2003: Understanding through Modeling. Conference in Golden, CO, USA, September 16. -- 19.09.2003. [9] Peterson, P. (2006): F2PY: Fortran to Python interface generator. Open-Source-Software, part of Numpy, available from





Sebastian Krittian, T. Schenkel , H. Oertel ­ University of Karlsruhe H. Schmid ­ University of Auckland, NZ ABSTRACT Statistically, heart disease has been the major cause of death in Germany in the recent past. In this context surgical treatment mostly focuses on a clinical or structural point of view. However, in order to characterise current cardiac conditions, the flow pattern inside the heart can no longer be neglected. To emphasise this statement, the patient specific Karlsruhe Heart Model (»KaHMo« ) has been developed at the Institute for Fluid Mechanics, University of Karlsruhe (TH), Germany. The present development stage of this numerical CFD-approach (STAR-CD) describes the inner ventricular wall movement by segmentation of Magnetic Resonance Images (MRI). This allows the transient flow pattern as well as the total losses to be precisely analysed. However, the internal flow pattern is also influenced by numerous external biomechanical processes. In order to gain a deeper understanding of the entire system, the prescribed geometry within the KaHMo will now be replaced by simulating the myocardial deformation using FLUENT, ABAQUS and MpCCI. Starting with general information about the KaHMo, this article summarises the identified requirements towards a coupled patient specific heart simulation. Like in other biomechanical models it turns out that performing fluid structure interaction goes along with the need for stronger coupling schemes. INTRODUCTION Many fluid structure interaction approaches exist around the world analyzing cardiac conditions [2,7,12,13]. However, due to the lack of structural in vivo data, they are mostly based on either generic data or in vitro measurements of animal hearts. Consequently, a transformation on a patient-specific human heart model cannot be undertaken.

Figure 1: KaHMo CAD model [9]


For this reason, a stand-alone CFD model (STAR-CD) based on time dependent movement of the inner ventricular wall was developed based on patient-specific MRI images. The main objective of the so called Karlsruhe Heart Model (»KaHMo«) is the ambition to analyse the transient inner ventricular flow pattern. Both, healthy and diseased hearts, can be investigated from a fluid mechanics point of view and important clinical indication criteria can be determined.

Figure 2: Comparison of MRI flux measurements (left) and simulation results (right) [6]

A description of the heart anatomy itself, the segmentation procedures as well as information about geometry and numerical model are given in Oertel et al. 2006 [9]. Figure 2 shows the latest evaluation process comparing simulation results and MRI flux measurements [6]. Observing the continuous progress of MRI techniques, future generations of KaHMo will have to take the structural calculation into account again. Desperately needed in vivo data will become more and more available. The new KaHMo FSI will extend the fluid calculation of KaHMo by replacing the time dependent movement with heart wall models. However, the main focus will still be put on fluid mechanics.

Dipl.-Ing. Sebastian Krittian University of Karlsruhe Kaiserstraße 12 76131 Karlsruhe Germany

Phone: +49 (0) 721 / 608 - 2765 [email protected]




FIRST SIMULATIONS The cardiac cycle can be divided into a passive filling and an active contraction phase whereas the focus of this work is put on the filling phase only. To judge the ability of the chosen software packages to perform the cardiac fluid structure interaction, simplified ABAQUS and FLUENT models have been generated. Geometry In order to gain a symmetrical model geometry for first testing purposes, Chase Medical [1] kindly provided information about so-called shaper devices which are currently used for selected surgery methods after myocardial infarctions. The special geometrical dimensions of this body ensure the reconstructed ventricle keeps a certain maximum volume and at least an approximate prolate shape. This information builds the starting point of the developed endsystolic reference geometry shown in Figure 3.

Figure 3:. Shaper geometry Settings Starting from the reference geometry mentioned above, both a structural and a fluid stand alone model have been developed using ABAQUS and FLUENT, respectively. The finite element model consists of about 1000 shell elements; triangle elements have been used to mesh the base ( fixed boundary conditions) and quad elements to mesh the symmetrical apex (coupling surface). The structural density is s =980 kg/m3. Generic nonlinear hyperelastic material properties and damping values have been chosen so that a stable simulation could be performed with physiological load conditions. The number of 10.000 tetrahedron cells for the fluid domain is also kept small in light of the testing purpose of the simulation. Due to large deformation of the structural domain, the mesh dynamics ability of Fluent is strongly needed to provide an adequate mesh quality. As boundary condition an atrial pressure of p = 1000 Pa has to perform a volume change from V0 = 50 ml up to V1 = 150 ml. The fluid is treated as laminar and Newtonian with F = 1008 kg/m3 and = 0.0055 Pa·s and solver settings are chosen as segregated and implicit. The total filling time is set to T = 0.5 s whereas the coupling time step size is t = 0.005 s.


RESULTS As far as the finite element model is concerned, the chosen generic non-linear hyperelastic material parameters allowed to achieve the demanded end volume with the physiological loading conditions. In order to check the stand alone fluid model, a one-way coupling was performed using the described inlet boundary conditions. Loads were not applied by the coupling procedure but directly to the structural model. Therefore, the FE model was responsible to deform the fluid domain to the end volume configuration. Consequently, both, the structural and the fluid stand alone model proved their ability to perform either simulations. However, the one-way coupling is not reasonable from a fluid mechanical point of view. The fluid structure interaction process definitely needs to be applied to get into the transient flow pattern. Performing the fluid structure interaction, stability issues occurred. Reasons for diverging simulation results are mainly caused by not achieving equilibrium conditions after each time step due to the weak coupling scheme. Like in other biomechanical fluid structure interaction processes, stability issues occur if the density of fluid and structure are of the same order and thus inertia effects come into play [8]. Within the actual test case, stable end configurations could be achieved by either using higher structural stiffness accompanied by higher inlet pressure or using strongly increased numerical damping. However, both configurations are not recommendable for analysing the patient-specific flow pattern inside the human heart. Figure 4 shows a qualitative pressure distribution during the filling phase as well as midplane projected streamlines.

Figure 4: left: Qualitative pressure distribution during filling phase; right: Streamlines during filling phase Simulation: Krittian & Steinhilber (2006)



Finally, the test cases pointed out that further basic research has to be taken into account. First of all, advanced solid mechanical modelling methods as well as more realistic constitutive parameters have to be applied. The executable stand alone models have to be replaced by more complete numerical approaches. Therefore, a finer mesh discretisation has to be used for the fluid domain; the FE model needs to be represented by continuum elements. Figure 5 shows the future coupling surface in Fluent with inlet (blue) and outlet (red) boundary positions as well as the corresponding FE model.

Figure 5: left: inlet (blue), outlet (red) and coupling surface of CFD model; right: structural model However, the main issue seems to lie in the weak coupling scheme itself. Therefore, possibilities for a long-term realisation of a stronger coupling for biomechanical applications need to be taken into account. Please note that future models of KaHMo FSI have to consider patient-specific reference geometries out of MRI data together with »quasi fibre directions« for the filling phase.

Figure 6: Schematic of left ventricular matrix structure and fibre reinforcement

Figure 7: Schematic of left ventricular matrix structure and fibre reinforcement


ADVANCED METHODS Composite approach The myocardial deformation within KaHMo FSI shall be considered from a macroscopic point of view to satisfy available patient-specific information. The decision whether an orthotropic, a transversely isotropic or even an isotropic nonlinear constitutive behaviour is chosen for a special application depends strongly on the application itself. Although the heart muscle is usually considered to be nonlinearly orthotropic [5], KaHMo FSI will follow a quasi transversely isotropic modelling approach. Effectively existing dominant directions ­ called fibre, sheet and sheet normal direction (Figure 6) ­ are represented by an isotropic »matrix« and »fibre« assembly (Figure 7). Schmid et al. [11] suggested, that even selected isotropic laws may build a good first approximation for whole heart FE models, when phenomena like wall thickening do play a secondary role. A major advantage in choosing the composite approach is the possibility to include so called »quasi fibre directions« out of MRI tissue phase mapping procedures [4] in the future. The correlation between these directions derived from acceleration measurements and the effect of distributed muscle fibres need to be further analysed within the scope of those analyses.

1 W = __ (e(c ·(I 2c2





3 5 1) + __ (e(c ·(I -3) ) 1) + __ 2c4 2c6






(e (c6 · (I4 - 1)2)



iso aniso Constitutive laws One can see that the isotropic part is represented in terms of the principal invariants of the isochoric right Cauchy-Green stretch tensor (I1, I2) whereas the anisotropy is considered in terms of a modified invariant (I4) which includes the fibre direction (see also Holzapfel et al. [3]). A note able alternative approach is the usage of ABAQUS` Marlow Law which is used for existing uniaxial stress strain test data for nearly incompressible behaviour. In order to capture the highly nonlinear hyperelastic myocardial behaviour a simple shear fitting procedure is applied to define the needed material parameters c1 -c6 [11] based on a modified least square approach. However, the simple shear test data used is still based on animal tissue tests. Please note that for an optimal fitting result less restricted laws like exponential Fung-type or polyconvex laws need to be considered [10]. CONCLUSION First simulations for KaHMo FSI showed the ability of FLUENT, ABAQUS and MpCCI to simulate cardiac fluid structure interactions. However, the need for stronger coupling possibilities for biomechanical applications had to be brought into context. Future work includes the described further development of both stand-alone models towards a more complete patient-specific description. Therefore, more information out of MRI phase mapping



procedures will be taken into account. As long as no in vivo stress strain relationships for human hearts are available, parameter estimation will be used but converted to human tissue requirements. REFERENCES [1] [2] Cheng, Y., Oertel, H. and Schenkel, T.: Fluid-Structure Coupled CFD Simulation of the Left Ventricular Flow during Filling Phase. In: Annals of Biomedical Engineering 33 (2005), S. 567 776 [3] Holzapfel, G.A., Gasser, T.C. and Ogden, R.W.: A New Constitutive Framework for Arterial Wall Mechanics and a Comparitive Study of Material Models. In: Journal of Elasticity 61 (2000), S. 1 48 [4] Jung, B.A., Kreher, B.W., Markl, M. and Henning, J.: Visualization of Tissue Velocity Data from Cardiac Wall Motion Measurements with Myocardial Fibre Tracking: Principles and Implications for Cardiac Fiber Structure. In: European Journal of Cardio-thoracic Surgery 29 (2006), S. 158 164 [5] LeGrice, I.J., Smaill, B.H., Chai, L.Z., Edgar, S.G., Gavon, J.B. and Hunter, P.J.: Laminar Structure of the Heart. In: American Journal of Physiology 269 (1995), S. H571 H582 [6] Malve, M.: Weiterentwicklung des Modells KaHMo, Institut für Strömungslehre, Universität Karlsruhe (TH), Diss., 2006 [7] McQueen, D.M. and Peskin, C.S.: A Three-Dimensional Computer Model of the Human Heart for Studying Cardiac Fluid Dynamics. In: Computer Graphics (2000), S. 56 60 [8] Nobile, F.: Numerical Approximation of Fluid-Structure Interaction Problems with Application to Haemodynamics, Ecole Polytechnique Federale de Lausanne, Diss., 2001 [9] Oertel, H., Spiegel, K. and Donisi, S.: Modelling the Human Cardiac Fluid Mechanics - 2nd edition. Universitätsverlag Karlsruhe, 2006 [10] Schmid, H., Nash, M.P., Young, A.A. and Hunter, P.J.: Myocardial Material Parameter Estimation - A comparitive study for simple shear. In: Journal of Biomechanical Engineering 128, 5 (2006), S. 742 750 [11] Schmid, H.,Wang, Y.K., Ashton, J., Ehret, A.E., Krittian, S.B.S., Nash, M. and Hunter, P.J.: Myocardial Material Parameter Estimation II ­ A comprehensive comparison of orthotropic constitutive equations. In: Journal of Biomechanical Engineering (at review) (2007) [12] Vierendeels, J.A., Riemslagh, K. and Dick, E.: Computer Simulation of Interventricular Flow and Pressure Gradients During Diastole. In: Journal of Biomechanical Engineering 122 (2000), S. 667 674 [13] Watanabe, H., S., Sugiura, H., Kafuku and T., Hisada: Multiphysics Simulation of Left Ventricular Filling Dynamics using Fluid-Structure Interaction Finite Element Method. In: Biophysical Journal 87 (2004), S. 2074 2085





Markus Perschall, H. Spiess, T. Schenkel, H. Oertel ­ University of Karlsruhe ABSTRACT The purpose of this paper is to show an approach to the general problem of handling large motions and deformations of fluid regions such as they occur in the wide field of biomechanics applications. A MpCCI code adapter has been developed for the general purpose of prescribing arbitrary geometry deformations based on time discrete MRI or CAD data. MpCCI is supposed to transfer nodal coordinates from motion prescribing, structured surface meshes to an unstructured surface mesh in FLUENT. Taking profit from MpCCI`s neighbourhood search and interpolation techniques the full dynamic mesh capabilities of the receiver FLUENT can be used to generate high quality volume meshes for each stage of motion. In future work this approach may be used as a standard method for CFD simulations of either flow bounded by moving biological tissues or the fluids engineering design process of flexible pumping chambers. Those cavities are frequently used in sac-type blood pumps and cardiac assist devices. Using this method one is able to prescribe any geometry motion as long as structured surface meshes for the corresponding geometries are available. In contrast to structured volume meshes, these surface meshes can easily be generated by meshing software packages such as FLUENT Gambit even for complex geometric shapes and deformations. MOTIVATION As severe heart disease is the major cause of death in Germany different surgical techniques are applied to treat heart failure. Ventricular hemodynamics play an important role for the success of any surgical treatment. Therefore, both an evaluation and a prediction of the effect of surgical options on fluid mechanical and ventricular sufficiency is necessary. In cooperation with surgeons from the University Hospital of Bonn and Freiburg the Institute for Fluid Mechanics developed KaHMo (Karlsruhe Heart Model), a patient specific CFD-Model of the human heart. Like in almost every biomechanical kind of problem the simulation geometry is moving as well as deforming. Geometry data is obtained from the segmentation of patient specific MR images and is therefore available at discrete points of time only. Similar issues arise on the topic of animal locomotion in fluids, such as swimming and flying as well as in the fluids engineering design of the sac-type ventricular assist devices. The latter case will be considered as an application example in the following section.


METHOD AND APPLICATION EXAMPLE Before describing the applied method a short review of the previous treatment and model setup relevant for the simulation of deforming zones is given. In the past the usual approach at ISL (German: Institut für Strömungslehre) to numerically simulate the fluid flow in a deforming region consisted of the following steps: · Generation of the geometry surfaces for discrete stages of the motion · Generation of structured volume meshes of identical topology for each of the geometries with the same node numbering order. · Application of an approximation/ interpolation technique on each volume mesh vertex to account for stages of motion in between the ones provided by the given geometries · Prescription of each volume node position in time Due to the restriction of topological identity the sketched method allows limited deformations only because large overall deformation leads to large cell deformation and thus decreasing accuracy. To overcome these limitations FLUENT dynamic mesh tool can be used to realize complex geometry deformation, because only the position of surface nodes has to be provided while an unstructured volume mesh is both automatically smoothed and remeshed during run time. Because an identical surface mesh topology as well as an identical node numbering order on each of the given surfaces is mandatory for the ability of motion prescription, one has to deal with structured surface meshes and thus quadrilateral surface elements. In consequence of that a transition layer of pyramid cells must be created in order to connect the quadrilaterals on the surface to the tetrahedrals of the unstructured volume mesh. However, pyramid cells are not recommendable for the description of complex geometries on the one hand due to their skews in regions of small angles and on the other hand because they are less flexible to use in very narrow geometric regions. In the following the further steps are illustrated by an application example. For the purpose of flow computation inside blood pump chambers [4] three CAD surfaces of the same chamber at different stages of motion are provided (Figure 1). The approach is to split up the problem into two parts: One part is the description of surface motion represented by the node position prescription of structured surface meshes and

Dipl.-Ing. Markus Perschall University of Karlsruhe Kaiserstraße 12 76131 Karlsruhe Germany

Phone: +49 (0) 721 / 608 - 2591 [email protected]




Figure 1: CAD surfaces of each stage of motion (grey) with structured surface mesh (yellow, meshing was performed in GAMBIT) the other one is the CFD simulation using a fully unstructured grid. The first subproblem is basically treated by interpolation and approximation techniques implemented in the in-house developed PGM-Code (Prescribed Grid Motion). It has to be emphasized that the terms interpolation and approximation used here are related to the calculation of positions of each single node with respect to time (node trajectories). It turned out that approximation curve segments of node trajectories have to show 2nd order parametric steady behaviour at joint points in order to avoid pressure discontinuities in time which are associated with acceleration discontinuities of wall node movement [3]. The two sub problems are now coupled via MpCCI, i.e. structured surface meshes provided by the PGM-Code via interpolation/approximation between the discrete motion stages are associated with an unstructured surface mesh in FLUENT where the CFD computation is performed. To make communication between the PGM-Code and MpCCI possible a code adapter has been implemented [1]. Its task is processing coupling region data of the PGM-Code as well as performing node and element definitions via driver functions on the one hand and passing/ obtaining data to/from MpCCI via the so called Code Coupling Manager on the other hand. The principle is sketched in Figure 2 and the serial coupling scheme applied here can be seen in Figure 3.

Figure 2: Using MpCCI to couple two subproblems


In the beginning of a simulation FLUENT performs an initial exchange of quantities and the PGM-Code does an initial receive. The only quantities exchanged are node position and physical time. When the PGM-Code receives physical time it computes the corresponding new position from it and gives it back to FLUENT which then performs one timestep in the fluid flow computation. The new physical time is transferred to the PGM-Code and so on. In this manner we were able to perform the CFD calculation for the shown geometry deformations. The model was set up for laminar flow. The valves were simplified as pressure inlet/wall and wall/pressure outlet in the inflow phase and

Figure 3: MpCCI coupling scheme between the PGM-Code and FLUENT the outflow phase, respectively. Two cycles with an overall time of 2 seconds were simulated using 2000 time steps. The three meshes in FLUENT that correspond to the stages of motion depicted in figure 1 are shown in Figure 4 and section slices of the volume meshes are depicted in Figure 5. A velocity vector plot during the filling phase of the ventricle can be seen in Figure 6.

Figure 4: Geometries of each stage of motion with unstructured surface mesh



Figure 5: Side view section slices of the unstructured volume mesh smoothed and remeshed by FLUENT

Figure 6: Front view of velocity vector plot during the filling phase of the DLRVAD ventricle CONCLUSION AND OUTLOOK To prescribe extreme types of motion and deformation of surfaces structured meshes have to be used if no other surface mapping or projection technique is available. With the shown method it is possible to perform CFD simulations in FLUENT even for extreme prescribed geometry deformations, because the information of surface motion is transferred via a code adapter and MpCCI from a structured mesh to a completely unstructured tetrahedral mesh which can better handle very small angles and narrow regions. In the example presented here the topology of the unstructured surface mesh remains the same for each time step. In this case it is considerable to only perform the association for the given stages of motion via MpCCI and use a GRID_MOTION udf in FLUENT to prescribe the remaining time approximated stages. The sketch below shows the new software framework established at ISL which shall in future be used modularly to either perform FSI computations [2], or simulate fluid flow in regions bounded by externally prescribed deformations. Depending on further application cases


Figure 7: Software framework consisting of ABAQUS, FLUENT, PGM-Code and MpCCI to realize different modelling approaches future developments might deal with the issue of surface remeshing of the FLUENT side coupling region as well as the involvement of moving prism cell layers in the near wall regions in order to reduce numerical diffusion in near wall gradient computation. ACKNOWLEDGEMENTS Thanks go to Thomas Schmid, DLR Institute of Robotic Systems, Wesseling, Germany for providing the CAD geometries of the presented application example of a sac-type ventricular assist device. REFERENCES [1] Fraunhofer SCAI. MpCCI 3.0.6 Programmers Guide, 2007. [2] S.B.S. Krittian, H. Schmid, T. Schenkel, and H. Oertel. Preconditions for a fluid structure interaction of the human heart using FLUENT, ABAQUS and MpCCI. Proceedings for the 8th MpCCI User Forum, 2007. [3] M. Perschall. Berechnung des Leistungsumsatzes des menschlichen linken Ventrikels durch das kahmo-herzmodell. Studienarbeit, Universität Karlsruhe (TH), 2005. [4] T. Schmid, W. Schiller, K. Spiegel, M. Stock, D. Liepsch, B. Laschka, G. Hirzinger, H. Oertel, and A. Welz. Optimization of ventricle-shaped chambers for the implantable dlr assist device. Journal of Biomechanics, Abstracts of the 5th World Congress of Biomechanics, Munich, Germany, 29 July ­ 4 August 2006, 39(1):250, 2006.





Fraunhofer Institute for Algorithms and Scientific Computing SCAI Klaus Wolf Schloss Birlinghoven 53754 Sankt Augustin, Germany Phone: +49 (0) 2241 / 14 - 2557 [email protected] Visual concept: Markus Möritz Layout: Bianca Backert

ISSN 1860-6296 Copyright © 2007 Fraunhofer Institute SCAI All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means: electronical, electrostatic, magnetic tape, mechanical, photocopy, recording or otherwise, without permission in writing from the publishers. Only the authors but not the Fraunhofer Institute SCAI can be hold liable for any mistakes made in manuscripts of the conference proceedings.




Fraunhofer Institute SCAI Schloss Birlinghoven 53754 Sankt Augustin Germany



136 pages

Report File (DMCA)

Our content is added by our users. We aim to remove reported files within 1 working day. Please use this link to notify us:

Report this file as copyright or inappropriate