So I was trying to merge many tables taken for the same field (in different filters) together using the astropy table join function, when I ran into an unexpected problem: my tables appear to be too big. Here is the nominal code I am trying to run, a loop over table.join functions:
for (dat,filter) in zip(catdatalist, localfilterlist):
#print filter, counter
if counter==0:
bigcat=dat
else:
bigcat=table.join(bigcat,dat,join_type='inner',keys=['NUMBER', 'ALPHA_J2000', 'DELTA_J2000'],table_names=[filterstring,filter])
filterstring+=filter
counter+=1Which works just fine. However, when I try to write it to the disk like so...
bigcat.write(bigcatfilename,format='fits',overwrite=True) #This is the joint catalog across all filters
I get a bizarre error message. This only failed when I started doing a 6-filter merge, which appears to have more than 1150 columns in it.
Traceback (most recent call last):
File "Pipeline.py", line 2561, in <module>
SetupPhotometry(dir,mastercataloglist,figcounter,filterlist,objectname)
File "Pipeline.py", line 759, in SetupPhotometry
figcounter=ColorMerge(dir+'/coadd/',cataloglist,figcounter)
File "Pipeline.py", line 1679, in ColorMerge
bigcat.write(bigcatfilename,format='fits',overwrite=True) #This is the joint catalog across all filters
File "/usr/local/Cellar/python/2.7.7/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy-0.4.2-py2.7-macosx-10.9-x86_64.egg/astropy/table/table.py", line 1746, in write
io_registry.write(self, *args, **kwargs)
File "/usr/local/Cellar/python/2.7.7/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy-0.4.2-py2.7-macosx-10.9-x86_64.egg/astropy/io/registry.py", line 378, in write
writer(data, *args, **kwargs)
File "/usr/local/Cellar/python/2.7.7/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy-0.4.2-py2.7-macosx-10.9-x86_64.egg/astropy/io/fits/connect.py", line 294, in write_table_fits
table_hdu.writeto(output)
File "/usr/local/Cellar/python/2.7.7/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy-0.4.2-py2.7-macosx-10.9-x86_64.egg/astropy/io/fits/hdu/base.py", line 1713, in writeto
checksum=checksum)
File "/usr/local/Cellar/python/2.7.7/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy-0.4.2-py2.7-macosx-10.9-x86_64.egg/astropy/io/fits/hdu/hdulist.py", line 642, in writeto
self.verify(option=output_verify)
File "/usr/local/Cellar/python/2.7.7/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/astropy-0.4.2-py2.7-macosx-10.9-x86_64.egg/astropy/io/fits/verify.py", line 120, in verify
raise VerifyError('\n' + '\n'.join(messages))
astropy.io.fits.verify.VerifyError:
Verification reported errors:
HDU 1:
'TFIELDS' card has invalid value '1149'.
Note: PyFITS uses zero-based indexing.
Can you please try to explain to me a) what is happening here and b) how I can write my big table to FITS format? It looks like there are issues with the FITS format whenever there are more than 1000 columns (because the header keys become 9 characters long). If I need to use another format I will do that, but I need to figure out what is happening first.
Thanks in advance!
Steven Ehlert
So I was trying to merge many tables taken for the same field (in different filters) together using the astropy table join function, when I ran into an unexpected problem: my tables appear to be too big. Here is the nominal code I am trying to run, a loop over table.join functions:
Which works just fine. However, when I try to write it to the disk like so...
bigcat.write(bigcatfilename,format='fits',overwrite=True) #This is the joint catalog across all filters
I get a bizarre error message. This only failed when I started doing a 6-filter merge, which appears to have more than 1150 columns in it.
Can you please try to explain to me a) what is happening here and b) how I can write my big table to FITS format? It looks like there are issues with the FITS format whenever there are more than 1000 columns (because the header keys become 9 characters long). If I need to use another format I will do that, but I need to figure out what is happening first.
Thanks in advance!
Steven Ehlert