Skip to content

Commit 50699cc

Browse files
shehzan10shehzan10
committed
FEAT Added saveArray and readArray functions for file read/write
The format of the file (version 1) is as follows: Header: Description | Data Type | Size (Bytes) | Detailed Desc ------------|-----------|--------------|-------------- Version | Char | 1 | ArrayFire File Format Version for future use. Currently set to 1 Array Count | Int | 4 | No. of Arrays stored in file Per Array: Description | Data Type | Size (Bytes) | Detailed Desc ------------------------|-----------|--------------|-------------- Length of Key String | Int | 4 | No. of characters (excluding null ending) in the key string Key | Char [] | length | Key of the Array. Used when reading from file Offset | Int64 | 8 | No of bytes between offset and start of next array Array Type | Char | 1 | Type corresponding to af_dtype enum Dims (4 values) | Int64 | 4 * 8 = 32 | Dimensions of the Array Data | Type | sizeof(Type) * dims.elements() | Actual data of the array The offset is equal to 1 byte (type) + 32 bytes (dims) + size of data.
1 parent 07ffda7 commit 50699cc

File tree

4 files changed

+613
-0
lines changed

4 files changed

+613
-0
lines changed

‎docs/details/util.dox‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
/**
2+
\addtogroup arrayfire_func
3+
@{
4+
\defgroup print_func_print print
5+
6+
\brief Print the array to screen
7+
8+
Print Array and dimensions to screen
9+
10+
\ingroup arrayfire_func
11+
12+
=======================================================================
13+
14+
\defgroup stream_func_read readArray
15+
16+
\brief Load an array from a file
17+
18+
The readArray function lets users read arrays saved in files.
19+
Arrays can either be read using the index in the file (0-indexed), or using
20+
the key that was used along with the Array.
21+
22+
Note that if there are multiple arrays with the same key, only the first one
23+
will be read.
24+
25+
The format of the file (version 1) is as follows:
26+
27+
Header:
28+
Description | Data Type | Size (Bytes) | Detailed Desc
29+
------------|-----------|--------------|--------------
30+
Version | Char | 1 | ArrayFire File Format Version for future use. Currently set to 1
31+
Array Count | Int | 4 | No. of Arrays stored in file
32+
33+
34+
Per Array:
35+
Description | Data Type | Size (Bytes) | Detailed Desc
36+
------------------------|-----------|--------------|--------------
37+
Length of Key String | Int | 4 | No. of characters (excluding null ending) in the key string
38+
Key | Char [] | length | Key of the Array. Used when reading from file
39+
Offset | Int64 | 8 | No of bytes between offset and start of next array
40+
Array Type | Char | 1 | Type corresponding to af_dtype enum
41+
Dims (4 values) | Int64 | 4 * 8 = 32 | Dimensions of the Array
42+
Data | Type | sizeof(Type) * dims.elements() | Actual data of the array
43+
44+
The offset is equal to 1 byte (type) + 32 bytes (dims) + size of data.
45+
46+
An file with 2 arrays would look like (representative)
47+
48+
> 1\n
49+
> 2\n
50+
> Array 1 Key Length\n
51+
> Array 1 Key\n
52+
> Array 1 Offset\n
53+
> Array 1 Type\n
54+
> Array 1 Dims\n
55+
> Array 1 Data\n
56+
> Array 2 Key Length\n
57+
> Array 2 Key\n
58+
> Array 2 Offset\n
59+
> Array 2 Type\n
60+
> Array 2 Dims\n
61+
> Array 2 Data\n
62+
63+
\ingroup stream_func
64+
\ingroup arrayfire_func
65+
66+
=======================================================================
67+
68+
\defgroup stream_func_save saveArray
69+
70+
\brief Save an array to a binary file
71+
72+
The saveArray and readArray functions are designed to provide store and
73+
read access to arrays using files written to disk.
74+
75+
The format of the file (version 1) is as follows:
76+
77+
Header:
78+
Description | Data Type | Size (Bytes) | Detailed Desc
79+
------------|-----------|--------------|--------------
80+
Version | Char | 1 | ArrayFire File Format Version for future use. Currently set to 1
81+
Array Count | Int | 4 | No. of Arrays stored in file
82+
83+
84+
Per Array:
85+
Description | Data Type | Size (Bytes) | Detailed Desc
86+
------------------------|-----------|--------------|--------------
87+
Length of Key String | Int | 4 | No. of characters (excluding null ending) in the key string
88+
Key | Char [] | length | Key of the Array. Used when reading from file
89+
Offset | Int64 | 8 | No of bytes between offset and start of next array
90+
Array Type | Char | 1 | Type corresponding to af_dtype enum
91+
Dims (4 values) | Int64 | 4 * 8 = 32 | Dimensions of the Array
92+
Data | Type | sizeof(Type) * dims.elements() | Actual data of the array
93+
94+
The offset is equal to 1 byte (type) + 32 bytes (dims) + size of data.
95+
96+
An file with 2 arrays would look like (representative)
97+
98+
> 1\n
99+
> 2\n
100+
> Array 1 Key Length\n
101+
> Array 1 Key\n
102+
> Array 1 Offset\n
103+
> Array 1 Type\n
104+
> Array 1 Dims\n
105+
> Array 1 Data\n
106+
> Array 2 Key Length\n
107+
> Array 2 Key\n
108+
> Array 2 Offset\n
109+
> Array 2 Type\n
110+
> Array 2 Dims\n
111+
> Array 2 Data\n
112+
113+
Save array allows you to append any number of Arrays to the same file using
114+
the append argument. If the append argument is false, then the contents of the
115+
file are discarded and new array is written anew.
116+
117+
On each append, the array counter in the header is incremented and the new
118+
array is written to the end of the file. This function does not check if the
119+
tag is unique or not.
120+
121+
\ingroup stream_func
122+
\ingroup arrayfire_func
123+
124+
=======================================================================
125+
126+
\defgroup data_func_randn randn
127+
128+
\brief Create a random array sampled from a normal distribution
129+
130+
The distribution is centered around 0
131+
132+
\ingroup data_mat
133+
\ingroup arrayfire_func
134+
135+
@}
136+
*/
137+

‎include/af/util.h‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,54 @@ namespace af
3232
*/
3333
AFAPI void print(const char *exp, const array &arr, const int precision);
3434

35+
/**
36+
\param[in] key is an expression used as tag/key for the array during \ref readArray
37+
\param[in] arr is the array to be written
38+
\param[in] filename is the path to the location on disk
39+
\param[in] append is used to append to an existing file when true and create or
40+
overwrite an existing file when false
41+
42+
\ingroup stream_func_save
43+
*/
44+
AFAPI void saveArray(const char *key, const array &arr, const char *filename, const bool append = false);
45+
46+
/**
47+
\param[in] filename is the path to the location on disk
48+
\param[in] index is the 0-based sequential location of the array to be read
49+
50+
\returns array read from the index location
51+
52+
\note This function will throw an exception if the index is out of bounds
53+
54+
\ingroup stream_func_read
55+
*/
56+
AFAPI array readArray(const char *filename, const unsigned index);
57+
58+
/**
59+
\param[in] filename is the path to the location on disk
60+
\param[in] key is the tag/name of the array to be read. The key needs to have an exact match.
61+
62+
\returns array read by key
63+
64+
\note This function will throw an exception if the key is not found.
65+
66+
\ingroup stream_func_read
67+
*/
68+
AFAPI array readArray(const char *filename, const char *key);
69+
70+
/**
71+
When reading by key, it may be a good idea to run this function first to check for the key
72+
and then call the readArray using the index. This will avoid exceptions in case of key not found.
73+
74+
\param[in] filename is the path to the location on disk
75+
\param[in] key is the tag/name of the array to be read. The key needs to have an exact match.
76+
77+
\returns index of the array in the file if the key is found. -1 if key is not found.
78+
79+
\ingroup stream_func_read
80+
*/
81+
AFAPI int readArrayCheck(const char *filename, const char *key);
82+
3583
// Purpose of Addition: "How to add Function" documentation
3684
AFAPI array exampleFunction(const array& in, const af_someenum_t param);
3785
}
@@ -110,6 +158,51 @@ extern "C" {
110158
*/
111159
AFAPI af_err af_print_array_p(const char *exp, const af_array arr, const int precision);
112160

161+
/**
162+
\param[in] key is an expression used as tag/key for the array during \ref readArray
163+
\param[in] arr is the array to be written
164+
\param[in] filename is the path to the location on disk
165+
\param[in] append is used to append to an existing file when true and create or
166+
overwrite an existing file when false
167+
168+
\ingroup stream_func_save
169+
*/
170+
AFAPI af_err af_save_array(const char* key, const af_array arr, const char *filename, const bool append);
171+
172+
/**
173+
\param[out] out is the array read from index
174+
\param[in] filename is the path to the location on disk
175+
\param[in] index is the 0-based sequential location of the array to be read
176+
177+
\note This function will throw an exception if the key is not found.
178+
179+
\ingroup stream_func_read
180+
*/
181+
AFAPI af_err af_read_array_index(af_array *out, const char *filename, const unsigned index);
182+
183+
/**
184+
\param[out] out is the array read from key
185+
\param[in] filename is the path to the location on disk
186+
\param[in] key is the tag/name of the array to be read. The key needs to have an exact match.
187+
188+
\note This function will throw an exception if the key is not found.
189+
190+
\ingroup stream_func_read
191+
*/
192+
AFAPI af_err af_read_array_key(af_array *out, const char *filename, const char* key);
193+
194+
/**
195+
When reading by key, it may be a good idea to run this function first to check for the key
196+
and then call the readArray using the index. This will avoid exceptions in case of key not found.
197+
198+
\param[out] index of the array in the file if the key is found. -1 if key is not found.
199+
\param[in] filename is the path to the location on disk
200+
\param[in] key is the tag/name of the array to be read. The key needs to have an exact match.
201+
202+
\ingroup stream_func_read
203+
*/
204+
AFAPI af_err af_read_array_key_check(int *index, const char *filename, const char* key);
205+
113206
// Purpose of Addition: "How to add Function" documentation
114207
AFAPI af_err af_example_function(af_array* out, const af_array in, const af_someenum_t param);
115208

0 commit comments

Comments
 (0)