In determining the RotationGroup of an assembly permutations of all the subunits are generated by combining existing valid permutations. The maximum number of permutations is the factorial of the number of subunits, but this is not taken into account in the recursive method completeGroup() inside PermutationGroup. This means that permutations continue to be combined even if the number of existing permutations is the maximum (so no new ones will be found).

For an 8 subunit example, the maximum number of permutations is 40320 (a feasible number), but if all of them have to be combined to each other (an avoidable situation) the number of iterations rises to nearly 2 billion.

However, this small fix (checking if the maximum number of permutations has been reached) does not solve cases with higher number of subunits (with 13 subunits, 6 billion permutations possible). Other solutions might be a maximum number of permutations or a timeout threshold, like for the local symmetry calculation.

Although these are rare cases, they correspond to the worst case scenario for the method. An example is the domain d1tnsa_. To reproduce the results run the internal symmetry analysis and try to calculate the PG:

String name = "d1tnsa_";

AtomCache cache = new AtomCache();
Structure s = cache.getStructure(name);
Atom[] atoms = StructureTools.getRepresentativeAtomArray(s);

CeSymm ceSymm = new CeSymm();
CESymmParameters params = ceSymm.getParameters();
params.setOptimization(true);
params.setMultipleAxes(false);

MultipleAlignment result = ceSymm.align(atoms, params);

//This line does not terminate (takes very long)
StructureTools.getQuaternarySymmetry(result);