I tried compiling Pandas against this branch and I get a compiler error in Pandas related to the changes to these structs:

     cython -M --fast-fail -3 -Xfreethreading_compatible=true --include-dir /Users/goldbaum/Documents/pandas/.mesonpy-dvqz12__/pandas/_libs/tslibs '-X always_allow_keywords=true' --shared=pandas._libs._cyutility /Users/goldbaum/Documents/pandas/pandas/_libs/tslibs/np_datetime.pyx -o pandas/_libs/tslibs/np_datetime.cpython-314-darwin.so.p/pandas/_libs/tslibs/np_datetime.pyx.c

      Error compiling Cython file:
      ------------------------------------------------------------
      ...

      cdef NPY_DATETIMEUNIT get_datetime64_unit(object obj) noexcept nogil:
          """
          returns the unit part of the dtype for a numpy datetime64 object.
          """
          return <NPY_DATETIMEUNIT>(<PyDatetimeScalarObject*>obj).obmeta.base
                                    ^
      ------------------------------------------------------------
      /Users/goldbaum/Documents/pandas/pandas/_libs/tslibs/np_datetime.pyx:78:30: Cannot convert 'PyDatetimeScalarObject *' to Python object
      [12/159] Compiling Cython source /Users/goldbaum/Documents/pandas/pandas/_libs/tslibs/dtypes.pyx
      [13/159] Generating _cyutility.c with a custom command
      [14/159] Compiling Cython source /Users/goldbaum/Documents/pandas/pandas/_libs/tslibs/conversion.pyx
      FAILED: [code=1] pandas/_libs/tslibs/conversion.cpython-314-darwin.so.p/pandas/_libs/tslibs/conversion.pyx.c
      cython -M --fast-fail -3 -Xfreethreading_compatible=true --include-dir /Users/goldbaum/Documents/pandas/.mesonpy-dvqz12__/pandas/_libs/tslibs '-X always_allow_keywords=true' --shared=pandas._libs._cyutility /Users/goldbaum/Documents/pandas/pandas/_libs/tslibs/conversion.pyx -o pandas/_libs/tslibs/conversion.cpython-314-darwin.so.p/pandas/_libs/tslibs/conversion.pyx.c

      Error compiling Cython file:
      ------------------------------------------------------------
      ...
              return convert_str_to_tsobject(ts, tz, dayfirst, yearfirst)

          if checknull_with_nat_and_na(ts):
              obj.value = NPY_NAT
          elif cnp.is_datetime64_object(ts):
              num = (<PyDatetimeScalarObject*>ts).obmeta.num
                     ^
      ------------------------------------------------------------
      /Users/goldbaum/Documents/pandas/pandas/_libs/tslibs/conversion.pyx:384:15: Cannot convert 'PyDatetimeScalarObject *' to Python object

So I don't think we can unconditionally change these definitions without providing an API to migrate to. We might need to ask David Woods for help if it's not clear how to write the Cython bindings without relying on conditional compilation or other discouraged constructs in Cython (see https://cython.readthedocs.io/en/latest/src/userguide/language_basics.html#conditional-compilation).

Maybe we need new getter functions for obval as well as obmeta?

cdef NPY_DATETIMEUNIT get_datetime64_unit(object obj) noexcept nogil:
"""
returns the unit part of the dtype for a numpy datetime64 object.
"""
return <NPY_DATETIMEUNIT>(<PyDatetimeScalarObject*>obj).obmeta.base

This looks like from older version of this branch, are you trying with the latest commit?

Never mind, this is from pandas internals.

I fixed it by adding it back, so that projects like pandas which directly access the struct can continue working. This also means that anyone trying to access in limited API would see compilation errors. I don't think there is an easy way to add it for limited API, @da-woods?

There's no reason not to give the struct contents in the pxd file - it doesn't generate any code until you try to use it.

It won't actually be possible to use the fields of these structs when using the opaque limited API. I don't currently think there's much way around that.

Possibly:

I added an argument to cython.cast(..., objstruct_cast=True) which gets the objstruct of an opaque type by doing PyObject_GetTypeData. That'd require unmerged and unreleased Cython versions. It would also be a bit dodgy because the original objects aren't actually opaque, but given the object layout you might get away with it. I think Cython might stop you from doing it though because it'll know that it isn't safe.

There's two potential issues with this pattern:

1. I think it's really a violation of the c and c++ strict aliasing rules and so probably undefined behaviour. It's probably not problematic undefined behaviour (since the compiler probably can't see the access through the "other" type in opaque code). But I decided against it as a solution to a similar problem in Cython for now. The problem remains unsolved though so you should probably just do it. But be aware...
2. If you somehow manage to by-pass the macro in non-opaque code this definitely ends up with the wrong answer. You could do that by taking a function pointer for example.

I think it's really a violation of the c and c++ strict aliasing rules and so probably undefined behaviour. It's probably not problematic undefined behaviour (since the compiler probably can't see the access through the "other" type in opaque code).

The pointer is first cast to char * which avoid strict aliasing violation because char * is allowed to alias any other object pointer.

If you somehow manage to by-pass the macro in non-opaque code this definitely ends up with the wrong answer. You could do that by taking a function pointer for example.

I don't think that is an issue as that should never happen.

The pointer is first cast to char * which avoid strict aliasing violation because char * is allowed to alias any other object pointer.

I don't going via a cast to char * avoids it - the issue is accessing the same data through seemingly unrelated structures. The idea of allowing a cast to char * to to provide a way to reading the binary representation of an object rather than to allow any cast through A -> char* -> B.

As I say, I don't think it will be a problem in practice but it makes me a bit nervous