********************** Unsolved issues **********************

1. When tree node reaches the MAX_TREE_LEVEL but the particles exceeding MAX_PACKAGE_SIZE, may be not stable.

2. While PM thread runs asyncronizing with Tree threads, sometimes there would occur memory collision, don't know why.

3. After long time simulation, while z is rather small, the particle was clustered, and the computation is not balanced. Work has been down for inner process calculating.

******************** Code fixing history ********************

------------------2015/04/17 14:00--------------------

traversal.h

traversal.c

A full-level expansion scheme for enqueuing during tree traversal.

A mature load balance scheme in local calculation.

ppkernel.h

nbody.256

Parameter adjusting for better performance in current cases.

------------------2015/04/09 17:00--------------------

traversal.h

traversal.c

A two-level expansion for enqueuing during tree traversal.

------------------2015/03/30 10:00--------------------

traversal.h

traversal.c

Modified the task pool with a indirect refering array.

------------------2015/03/28 19:00--------------------

traversal.h

traversal.c

Added checknan/checkpart/checkaccpm functions for debugging.

Eliminated a bug in queue processing.

partmesh.c

Added some codes (commented) for debugging.

main.c

Corrected the barrier issue and the output information.

------------------2015/03/08 17:00--------------------

stepping.c

Using kick1_pm and kick2_pm function instead of the old kick_system function.

initial.c

Corrected the initialization of EPS2.

Changed the random velocity initialization contidion.

main.c

Added the refine step running mode.

------------------2015/02/26 7:00--------------------

ppkernel.c

Rewrite the calculation with macro definitions.

------------------2015/02/08 7:00--------------------

ppkernel.c

Corrected two numeric mistake in ppkernel.

traversal.c

Fixed the addition of acceleration with NMK_PP_TAB defined, do not need to multiply the GRAVITY const.

------------------2015/01/30 0:00--------------------

inital.c

traversal.c

pp9m.c

ppverify.c

nbody.ver

Adding the verifying function in the code. To enable it, set PP_VERIFY macro enabled. To Generate correct PP result file, set PP_GEN_VER_FILE enabled.

The verifying system should be a box with 200000, particles distributed uniformly in [87501-112499].

Usage of ppverify.c : ./ppverify [nPart/1048576 [file to be check/pp.acc [file as standard/ppn.acc [index file/pp.ori [gravity const/1.0]]]]]

ppkernel.c

traversal.c

Fixed the two issues for mass pp calculation.

Fixed the error of setting wrong start and ending block for each thread.

Fixed the search algorithm while TA is leaf buf TB is not.

domain.c

Fixed the mesh masking issue (upper lwer) for the case with large empty space.

Modifid some judge condition to correct the adjoining particle processing error for 'dcount' calculation and unneccesary received partiles.

------------------2015/01/21 22:00--------------------

proto.h

global.h

inital.c

traversal.h

ppkernel.h

ppkernel.c

Add the prefactor in PP calculation. G(erfc(r/2rs) + (r/(sqrt(pi)*rs)*exp(-r^2/2s^2))) * oldfactor.

Modified the EPS2 into constants structure, and PP use the *constants as an parameter.

Add NMK_NAIVE_GRAVITY for naive gravity calculation.

main.c

partmesh.c

Setting FFTW2_LIB as default, since MKL doesn't support FFTW3 MPI interface.

------------------2015/01/13 13:00--------------------

traversal.c

Modified some judge critierian in DTT tree walking on MIC to raise the performance.

traversal.c

initial.h

initial.c

global.c

Add the INPUT VARIBLES and functions for CPU-MIC load balancing.

DYNAMIC_LOAD/DYNLOAD: if non zero, enable the load balancing.

------------------2015/01/13 1:00--------------------

traversal.c

Modified DTT process logic, if A and B are equal in size, seperate A first (B in old versions). If A is leaf, using a DST for B tree entirely, avoid more queue operations. If B's childrens are leaf nodes, also add it to A's pp, instead of enqueue operations.

------------------2015/01/12 14:00--------------------

global.h

initial.h

initial.c

Made the chages for this INPUT VARIBLES:

(varible name/input string: usage notes.)

NTHREAD_CPU/NTHCPU: CPU thread numbers per MPI PROCESS while tree walking.

NTHREAD_MIC/NTHMIC: MIC thread numbers per MIC CARD.

NTHREAD_TREE/NTHTREE: CPU thread numbers per MPI PROCESS for tree building (coz of sometimes too many threads will get an segmentation fault in tree building).

NP_PER_NODE/NPNODE: the number of how many MPI PROCESSes in one node.

NMIC_PER_NODE/MICNUM: the number of MICs in one node.

NTEAM_CPU/NTEAMCPU: master thread numbers for task pools on CPU in single MPI PROCESS.

NTEAM_MIC/NTEAMMIC: master thread numbers for task pools on MIC in single MPI PROCESS.

CPU_RATIO/CPURATIO: the ratio to indicate how much calculation should be done on CPU in tree walking.

RND_PART/RNDPART: if not 0, random uniform distributed particles should be generated instead of file reading.

MAX_TREE_LEVEL/TREEMAX: the maximum level of a local tree.

MIN_TREE_LEVEL/TREEMIN: the minimum level of a local tree.

MAX_PACKAGE_SIZE/NCRITI: the normal Ncritical in tree building.

MIC_PP_THRESHOLD/PPSIZE: the Ncritical used on MIC while tree walking.

Added init_random_particles() function.

Added a section to check the particles not equal to BOX_SIZE due to precision issue.

dbginfo.h

Added extern definition of 'stdio' functions and varibles for correctly offload on MIC.

domain.h

domain.c

Added nnode varibles to remember the number of tree nodes malloced for offloading to MIC.

global.h

global.c

Modified the MPI_Abort() to exit() for offload compilation in system_exit() since MPI_Abort() could not be offloaded.

Added xmemalign() function for aligned memory allocation.

proto.h

Added CalcBode and TransBody structure for force calculation and transfering.

Added rmax to Node stucture to use in acceptance judging.

main.c <-- driver.c

Added timer sections, warmup to MIC function, and tree building and walking functions..

subtree.h

subtree.c

queue.h

queue.c

traversal.h

traversal.c

Tree building and force calculation functions. Cleared from old version, using new data structures.

pp.c

ppm.c

ppkernel.h

ppkernel.c

Makefile

Moved from old version, with minor modification.

------------------2015/01/05 22:00--------------------

initial.c

Correct the char array length of 'name' and 'value'.

stepping.c

Included 'math.h'.

Strictly modified draft() function to adjust border particles correction about its precision.

------------------2015/01/05 12:00--------------------

nmk_utils.h:

New file for utility functions. Now including dbginfo macros and dtime() function.

dbginfo.h:

Macros for log and debug info output.

dtime.h:

dtime() implementation.

offload.h:

Offload macros defination.

global.h:

global.c:

Moving dtime() to dtime.h.

Added xrealloc() function.

partition.c:

MESH_BITS and PEANO bits need to be equal now, one mesh could not be seperated in two MPI processes. Then 'upper' mesh should be exclude in current domain.

domain.h:

domain.c:

Added rank, nproc vars in Domain structure.

Added *tree, *mroot, *pidx to indicate the local tree nodes, root of each mesh, first particle index of each meshes.

Modified the domain creation and deletion process.

Deleted *cuboid.

The mesh 'upper' should be not include in local domain since no mesh should be splited (peano_bits==mesh_bits).

Corrected the bug in adjoint counting. see 'if (x<0 || x>=In) continue;' etc.

Combine the particle data with adjointed particles.

Strictly modified the adjointed particle processing functions in broadcast_frontiers.

------------------2015/01/01 00:00--------------------

proto.h

global.h

global.c

initial.h

initial.c

partition.h

partition.c

partmesh.h

partmesh.c

driver.h

driver.c

iosnap.h

iosnap.c

stepping.h

stepping.c

Based on the new PM code with new structure.

------------------------------------------------------