Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
5a19931
init feather implementation
sahithyaravi Nov 6, 2019
87907d9
Merge remote-tracking branch 'origin/develop' into feather_investigation
sahithyaravi Nov 8, 2019
6d2f5c3
sparse matrix
sahithyaravi Nov 8, 2019
33881ea
test notebook
sahithyaravi Nov 8, 2019
55743bd
feather pickle compare
sahithyaravi Nov 8, 2019
1437005
test arrow vs feather
sahithyaravi Nov 11, 2019
ef461d7
Merge remote-tracking branch 'origin/develop' into feather_investigation
sahithyaravi Nov 11, 2019
5c27237
add columns condition
sahithyaravi Nov 11, 2019
f61d9b5
Testing
sahithyaravi Nov 14, 2019
484869e
Merge branch 'develop' into feather_investigation
sahithyaravi Jan 6, 2020
3c513b0
get_dataset add cache format
sahithyaravi Jan 8, 2020
0b3d781
add pyarrow
sahithyaravi Jan 8, 2020
a9becf1
sparse matrix check
sahithyaravi Jan 8, 2020
aff8aff
pep8 and remove files
sahithyaravi Jan 8, 2020
48e2a16
return type
sahithyaravi Jan 8, 2020
19c22fe
fix type annotation
sahithyaravi Jan 8, 2020
98be055
value check
sahithyaravi Jan 8, 2020
112eb1d
change feather condition
sahithyaravi Jan 10, 2020
99fac3d
fixes and test
sahithyaravi Jan 11, 2020
cf3cbad
fix errors
sahithyaravi Jan 13, 2020
7583e88
Merge branch 'develop' into feather_investigation
sahithyaravi Jan 13, 2020
09d6bdb
testing file
sahithyaravi Jan 13, 2020
3aff927
feather new file for attributes
sahithyaravi Jan 14, 2020
b521534
change feather attribute file path
sahithyaravi Jan 14, 2020
8eb77cf
delete testing file
sahithyaravi Jan 14, 2020
4894bbd
testing changes
sahithyaravi Jan 14, 2020
b6839b1
delete pkls
sahithyaravi Jan 14, 2020
131bdad
fixes
sahithyaravi Jan 14, 2020
aeb9b98
fixes
sahithyaravi Jan 14, 2020
865d4dc
add comments
sahithyaravi Jan 15, 2020
701496f
change default caching
sahithyaravi Jan 22, 2020
f689897
pip version
sahithyaravi Jan 27, 2020
74f359e
review comment fixes
sahithyaravi Jan 29, 2020
19272e5
newline
sahithyaravi Jan 29, 2020
09a5469
fix if condition
sahithyaravi Jan 29, 2020
f0da5a1
Update install.sh
sahithyaravi Feb 3, 2020
ed8ca7b
pandas verison due to sparse data
sahithyaravi Feb 3, 2020
d7488f7
review #2
sahithyaravi Feb 11, 2020
d09c431
Update appveyor.yml
sahithyaravi Feb 17, 2020
bf44356
Update appveyor.yml
sahithyaravi Feb 18, 2020
e6bc0b0
rename cache dir
sahithyaravi Feb 18, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
pep8 and remove files
  • Loading branch information
sahithyaravi committed Jan 8, 2020
commit aff8aff29fd75cc38da266ff426a4c56655af8cf
2,963 changes: 0 additions & 2,963 deletions df1.csv

This file was deleted.

2,963 changes: 0 additions & 2,963 deletions df2.csv

This file was deleted.

Binary file removed df_feather.pkl
Binary file not shown.
5 changes: 3 additions & 2 deletions openml/datasets/dataset.py
Original file line number Diff line number Diff line change
Expand Up @@ -438,7 +438,7 @@ def _create_pickle_in_cache(self, data_file: str) -> str:
# We parse the data from arff again and populate the cache with a recent pickle file.
X, categorical, attribute_names = self._parse_data_from_arff(data_file)

if self.cache_format == "feather" and type(X) != scipy.sparse.csr.csr_matrix: # and X.shape[1] <= 1000:
if self.cache_format == "feather" and type(X) != scipy.sparse.csr.csr_matrix:
print("feather write")
feather.write_feather(X, data_feather_file)
with open(data_pickle_file, "wb") as fh:
Expand All @@ -462,7 +462,8 @@ def _load_data(self):
if self.data_pickle_file is None:
if self.data_file is None:
self._download_data()
self.data_pickle_file, self.data_feather_file = self._create_pickle_in_cache(self.data_file)
self.data_pickle_file, self.data_feather_file = self._create_pickle_in_cache(
self.data_file)

try:
if os.path.exists(self.data_feather_file):
Expand Down
Loading