This repository was archived by the owner on Sep 9, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Sep 9, 2026. It is now read-only.
bug: deserializing from dataframe with numpy==1.26.1 #1821
Description
Initial Checks
- I have searched Google & GitHub for similar requests and couldn't find anything
- I have read and followed the docs and still think this feature is missing
Description
I am looking for an efficient way of creating a DocList or DocVec from a large DataFrame. Currently I am converting the DataFrame to a Dict and then looping over this to create the various objects which is slow.
I did come across the from_dataframe method: https://docs.docarray.org/API_reference/array/da_stack/#docarray.array.doc_vec.doc_vec.DocVec.from_dataframe
However this does not support list like fields, which is quite common when you have field that contains the text embeddings.
Activity
JoanFM commentedon Oct 11, 2023
Can you give an example of a Doc type that would not work?
nikhilmakan02 commentedon Oct 11, 2023
Here is an example taken from the docs which I have modified to now include an embeddings column. Hope that makes it clearer. This example works fine with out the embeddings field.
You can even create a doc list with the embeddings field using a python loop as described above. Then use the to_dataframe() method to export the doc list. But you then can't go back and create the doc list again from that dataframe using the from_dataframe() method.
Note I am using doc list and doc vec interchangeably here, the problem is the same for both.
JoanFM commentedon Oct 11, 2023
What is the error you see? I do not reproduce the issue. My
pandasversion is 2.0.3. What is yours?nikhilmakan02 commentedon Oct 12, 2023
python==3.11docarray==0.39.0pandas==2.1.1Here is the traceback. I downgraded to
pandas==2.0.3same issue.JoanFM commentedon Oct 12, 2023
Can you share all the installed packages? I do not manage to reproduce.
nikhilmakan02 commentedon Oct 12, 2023
I can hopefully do a bit better to recreate the issue for you. I am working in a vs code dev container which is just a docker container.
So you can recreate the issue by using docker, vs code and the vs code dev container extension. Or however other means you would like. The attachment below contains the docker file, requirements.txt file and the code already shown above, there is also a devcontainer.json file which is used by the vs code dev container extension.
github-issue-docarry-df.zip
Hopefully this should create an isolated instance of the problem that you can reproduce on your end.
JoanFM commentedon Oct 12, 2023
Thanks, will try to do it
JoanFM commentedon Oct 15, 2023
Reproduced in the docker container thanks.
This is the list of dependencies:
JoanFM commentedon Oct 15, 2023
Okey, it seems the reason is the
numpyversion is different to the one I have. I will try to fix this, thank you very much.nikhilmakan02 commentedon Oct 15, 2023
Amazing thanks. Out of interest what does the below mean in the docs
"List-like fields (including field of type DocList) are not supported."
I initially raised this as a feature request as I thought that meant you couldn't have lists in a field.
JoanFM commentedon Oct 15, 2023
Oh, this may be part of the issue. However, there should be a better error message.
JohannesMessner commentedon Oct 18, 2023
Hey @nikhilmakan02 , this is indeed a bug, the sentence
"List-like fields (including field of type DocList) are not supported."just means that the Document schema that defines your DocList/DocVec cannot contain another DocList/DocVec, which is not the case in your example.I am working on a fix right now, if things go to plan it should be out today.
[-]Support for List-like fields when using from_dataframe on DocVec or DocList[/-][+]bug: deserializing from dataframe with numpy==1.26.1[/+]JohannesMessner commentedon Oct 18, 2023
@nikhilmakan02 the PR is ready to go, but until it is released you can circumvent the problem by downgrading your numpy version. Thanks for flagging this!
nikhilmakan02 commentedon Oct 19, 2023
@JohannesMessner amazing work! Thank you very much for addressing this.
bionicles commentedon Jan 4, 2024
@JohannesMessner @nikhilmakan02 @JoanFM FYI
found an edge case for you
from NLM UMLS:
"C3842509","Finding","0 - none; 0 = None; 0= None; 0=None","None"
val: <class 'str'> = C3842509
val: <class 'str'> = Finding
val: <class 'str'> = 0 - none; 0 = None; 0= None; 0=None
val: <class 'str'> = None
def _is_none_like(val: Any) -> bool: """ :param val: any value :return: true iff `val` equals to `None`, `'None'` or `''` """ # Convoluted implementation, but fixes https://github.com/docarray/docarray/issues/1821 # tensor-like types can have unexpected (= broadcast) `==`/`in` semantics, # so treat separately is_np_arr = isinstance(val, np.ndarray) if is_np_arr: return False is_torch_tens = is_torch_available() and isinstance(val, torch.Tensor) if is_torch_tens: return False is_tf_tens = is_tf_available() and isinstance(val, tf.Tensor) if is_tf_tens: return False is_jax_arr = is_jax_available() and isinstance(val, jax.numpy.ndarray) if is_jax_arr: return False # "normal" case + print(f"val: {type(val)} = {val}") return val in ['', 'None', None]anybody know how to get docarray to load documents containing "None" as a string