Skip to content
This repository was archived by the owner on Sep 9, 2026. It is now read-only.

Commit da3b32e

Browse files
hanxiaohanxiao
authored
fix(array): push pull use most compact protocol (#6)
* fix(array): push pull use most compact protocol * docs(readme): add first readme
1 parent dbe447b commit da3b32e

8 files changed

Lines changed: 134 additions & 9 deletions

File tree

‎.github/README-img/9nn.png‎

387 KB
Loading

‎.github/README-img/sprite.png‎

631 KB
Loading

‎.github/README-img/tsne.gif‎

7.15 MB
Loading

‎README.md‎

Lines changed: 118 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ The data structure for unstructured data.
66

77
🌌 **All data types**: super-expressive data structure for representing complicated/mixed/nested text, image, video, audio, 3D mesh data.
88

9-
🧑‍🔬 **Data science powerhouse**: easy-to-use functions for facilitating data scientists work on embedding, matching, visualizing unstructured data via Torch/Tensorflow/ONNX/PaddlePaddle.
9+
🧑‍🔬 **Data science powerhouse**: easy-to-use functions for facilitating data scientists work on embedding, matching, visualizing, evaluating unstructured data via Torch/Tensorflow/ONNX/PaddlePaddle.
1010

1111
🚡 **Portable**: ready to wire with efficient conversion from/to Protobuf, binary, JSON, CSV, dataframe.
1212

@@ -21,4 +21,120 @@ pip install docarray
2121
```
2222
To install full dependencies, please use `pip install docarray[full]`.
2323

24-
## [Documentation](https://docarray.jina.ai)
24+
## [Documentation](https://docarray.jina.ai)
25+
26+
## Get Started
27+
28+
Let's use DocArray and ResNet50 to build a meme image search on [Totally Looks Like](https://sites.google.com/view/totally-looks-like-dataset). This dataset contains 6016 image-pairs stored in `/left` and `/right`. Images that shares the same filename are labeled as perceptually similar. For example,
29+
30+
| `/left` | `/right` | `/left` | `/right` |
31+
|---------|----------|---------|----------|
32+
|
33+
34+
Our problem is given an image from `/left` and find its most-similar image in `/right` (without looking at the filename of course).
35+
36+
### Load images
37+
38+
First load images and preprocess them with standard computer vision techniques:
39+
40+
```python
41+
from docarray import DocumentArray, Document
42+
43+
left_da = DocumentArray.from_files('left/*.jpg')
44+
```
45+
46+
To get a feeling of the data you will handle, plot them in one sprite image:
47+
48+
```python
49+
left_da.plot_image_sprites()
50+
```
51+
52+
<p align="center">
53+
<a href="https://docs.jina.ai"><img src="https://github.com/jina-ai/docarray/blob/master/.github/README-img/sprite.png?raw=true" alt="Load totally looks like dataset with docarray API" width="70%"></a>
54+
</p>
55+
56+
### Apply preprocessing
57+
58+
Let's do some standard computer vision preprocessing:
59+
60+
```python
61+
def preproc(d: Document):
62+
return (d.load_uri_to_image_blob() # load
63+
.set_image_blob_normalization() # normalize color
64+
.set_image_blob_channel_axis(-1, 0)) # switch color axis
65+
66+
left_da.apply(preproc)
67+
```
68+
69+
Did I mention `apply` work in parallel?
70+
71+
### Embed images
72+
73+
Now convert images into embeddings using a pretrained ResNet50:
74+
75+
```python
76+
import torchvision
77+
model = torchvision.models.resnet50(pretrained=True) # load ResNet50
78+
left_da.embed(model, device='cuda') # embed via GPU to speedup
79+
```
80+
81+
### Visualize embeddings
82+
83+
You can visualize the embeddings via tSNE in an interactive embedding projector:
84+
85+
```python
86+
left_da.plot_embeddings()
87+
```
88+
89+
<p align="center">
90+
<a href="https://docs.jina.ai"><img src="https://github.com/jina-ai/docarray/blob/master/.github/README-img/tsne.gif?raw=true" alt="Visualizing embedding via tSNE and embedding projector" width="90%"></a>
91+
</p>
92+
93+
Fun is fun, but recall our goal is to match left images against right images and so far we have only handled the left. Let's repeat the same procedure for the right:
94+
95+
```python
96+
right_da = (DocumentArray.from_files('right/*.jpg')
97+
.apply(preproc)
98+
.embed(model, device='cuda'))
99+
```
100+
101+
### Match nearest neighbours
102+
103+
We can now match the left to the right and take the top-9 results.
104+
105+
```python
106+
left_da.match(right_da, limit=9)
107+
```
108+
109+
Let's inspect what's inside `left_da` now:
110+
111+
```python
112+
for d in left_da:
113+
for m in d.matches:
114+
print(d.uri, m.uri, m.scores['cosine'].value)
115+
```
116+
117+
```text
118+
left/02262.jpg right/03459.jpg 0.21102
119+
left/02262.jpg right/02964.jpg 0.13871843
120+
left/02262.jpg right/02103.jpg 0.18265384
121+
left/02262.jpg right/04520.jpg 0.16477376
122+
...
123+
```
124+
125+
Better see it.
126+
127+
```python
128+
(DocumentArray(left_da[12].matches, copy=True)
129+
.apply(lambda d: d.set_image_blob_channel_axis(0, -1)
130+
.set_image_blob_inv_normalization())
131+
.plot_image_sprites('result.png'))
132+
```
133+
134+
<p align="center">
135+
<a href="https://docs.jina.ai"><img src="https://github.com/jina-ai/docarray/blob/master/.github/README-img/9nn.png?raw=true" alt="Visualizing top-9 matches using DocArray API" width="50%"></a>
136+
</p>
137+
138+
### Quantitative evaluation
139+
140+

‎docarray/array/mixins/evaluation.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,7 @@ def evaluate(
5353
metric_name = metric_name or metric_fn.__name__
5454
results = []
5555
for d, gd in zip(self, other):
56-
if not strict or hash_fn(d) != hash_fn(gd):
56+
if strict and hash_fn(d) != hash_fn(gd):
5757
raise ValueError(
5858
f'Document {d} from the left-hand side and '
5959
f'{gd} from the right-hand are not hashed to the same value. '

‎docarray/array/mixins/io/pushpull.py‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,13 @@ def read(self, n=-1):
5050
self._p_bar.update(self._task_id, advance=len(chunk))
5151
return chunk
5252

53-
dict_data = {'file': ('DocumentArray', bytes(self)), 'token': token}
53+
dict_data = {
54+
'file': (
55+
'DocumentArray',
56+
self.to_bytes(protocol='protobuf', compress='lz4'),
57+
),
58+
'token': token,
59+
}
5460

5561
(data, ctype) = requests.packages.urllib3.filepost.encode_multipart_formdata(
5662
dict_data
@@ -97,7 +103,7 @@ def pull(cls: Type['T'], token: str, show_progress: bool = False) -> 'T':
97103
if show_progress:
98104
progress.update(task_id, advance=len(chunk))
99105

100-
return cls.load_binary(f.getvalue())
106+
return cls.from_bytes(f.getvalue(), protocol='protobuf', compress='lz4')
101107

102108

103109
def _get_progressbar(show_progress):

‎docarray/array/mixins/parallel.py‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -44,8 +44,9 @@ def apply(self: 'T', *args, **kwargs) -> 'T':
4444
# noqa: DAR201
4545
:return: a new :class:`DocumentArray`
4646
"""
47-
new_da = type(self)()
48-
new_da.extend(self.map(*args, **kwargs))
47+
from ... import DocumentArray
48+
49+
new_da = DocumentArray(self.map(*args, **kwargs))
4950
self.clear()
5051
self.extend(new_da)
5152
return self
@@ -118,7 +119,9 @@ def apply_batch(self: 'T', *args, **kwargs) -> 'T':
118119
# noqa: DAR201
119120
:return: a new :class:`DocumentArray`
120121
"""
121-
new_da = type(self)()
122+
from ... import DocumentArray
123+
124+
new_da = DocumentArray()
122125
for _b in self.map_batch(*args, **kwargs):
123126
new_da.extend(_b)
124127
self.clear()

‎tests/unit/document/test_converters.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ def test_audio_convert_pipe(pytestconfig, tmpdir):
3434
def test_image_convert_pipe(pytestconfig):
3535
for d in from_files(f'{pytestconfig.rootdir}/.github/**/*.png'):
3636
(
37-
d.convert_uri_to_image_blob()
37+
d.load_uri_to_image_blob()
3838
.convert_uri_to_datauri()
3939
.set_image_blob_shape((64, 64))
4040
.set_image_blob_normalization()

0 commit comments

Comments
 (0)