[Bug 18925] libfoundation: Prevent stack overflow on 2+ OOMs

peter-b · peter-b · commit b62fad6a6a7e · 2016-11-28T14:27:31.000Z
The stack trace attached to bug 18925 shows the following calls
occurring recursively until stack overflow:

    ...
    MCMemoryNew()
    __MCValueCreate()
    MCErrorCreateWithMessage()
    MCErrorThrowOutOfMemory()
    MCMemoryNew()
    ...

`MCInitialize()` allocates the error structure for use during OOM into
a global.  Originally, `MCErrorThrowOutOfMemory()` did two things that
seemed sensible:

- if there was no OOM error structure, it would attempt to create one,
  and if that failed, it would exit
- after using the global OOM structure, it would clear it

Unfortunately, this meant that the first OOM that occurred during each
run would be fine -- because there would be a preallocated OOM error
structure -- but on second and subsequent OOM events, the global
structure would be nil.

Attempting to allocate the error would cause a recursive invocation of
`MCMemoryNew()`, which would fail and attempt to throw an OOM error,
which would try to allocate an error, etc.

This patch ensures that `MCErrorThrowOutOfMemory()` never tries to
create a new OOM error structure, and simply immediately aborts if one
is unavailable.  Rather than clearing the global, it keeps it around
for subsequent use.
diff --git a/docs/notes/bugfix-18925.md b/docs/notes/bugfix-18925.md
@@ -0,0 +1 @@
+# Prevent crashes on memory exhaustion
diff --git a/libfoundation/src/foundation-error.cpp b/libfoundation/src/foundation-error.cpp
@@ -397,18 +397,17 @@ MCErrorCreateAndThrow (MCTypeInfoRef p_error_type, ...)
 MC_DLLEXPORT_DEF
 bool MCErrorThrowOutOfMemory(void)
 {
-    if (s_out_of_memory_error == nil &&
-        !MCErrorCreate(kMCOutOfMemoryErrorTypeInfo, nil, s_out_of_memory_error))
+    if (s_out_of_memory_error == nil)
     {
-        exit(-1);
-        return false;
+	    /* This function may be being called from within
+	     * MCMemoryNew().  If there is no error structure already
+	     * allocated, calling MCErrorCreate() to obtain one will call
+	     * MCMemoryNew()... which will re-enter this function,
+	     * probably recursively until the stack overflows. */
+	    abort();
     }
     
-    MCErrorThrow(s_out_of_memory_error);
-    MCValueRelease(s_out_of_memory_error);
-    s_out_of_memory_error = nil;
-    
-    return false;
+    return MCErrorThrow(s_out_of_memory_error);
 }
 
 MC_DLLEXPORT_DEF
