Please follow: https://github.com/linkedin/feathr/blob/main/docs/dev_guide/pull_request_guideline.md

Updated PR description.

If a function depends on a chain of function, or a function depdends on a library that is huge, how does the pickle work? Will it still work(recursively)? Will it the function pickle become too huge?

@hangfei yes it will, cloudpickle serialize object recursively by value, the process stops at the module boundary, and will trigger module importing on deserialization when needed, so the only limitation appears when some local modules are not available on the remote side(there is a workaround but needs few manual steps).
The same process is used by PySpark, it uses cloudpickle to transfer compute tasks between drivers and executors.

The package size will not be significant usually, but some caution needs to be taken, i.e. if you create a 1M elements list in your UDF, the list will also be packed into the output byte array and increase its size dramatically, but this issue is actually inevitable, not matter what solution do you use.

@hangfei yes it will, cloudpickle serialize object recursively by value, the process stops at the module boundary, and will trigger module importing on deserialization when needed, so the only limitation appears when some local modules are not available on the remote side(there is a workaround but needs few manual steps). The same process is used by PySpark, it uses cloudpickle to transfer compute tasks between drivers and executors.

The package size will not be significant usually, but some caution needs to be taken, i.e. if you create a 1M elements list in your UDF, the list will also be packed into the output byte array and increase its size dramatically, but this issue is actually inevitable, not matter what solution do you use.

module is equal to a lib?

Could you provide a user guide for this part? better with examples on their own dependencies, code that they import from their teammates or some third party libs.

I think we can just add a note that refers to the official cloudpickle document, no need to add too many our own words or comments.

I think we can just add a note that refers to the official cloudpickle document, no need to add too many our own words or comments.

can you add a few lines in the developer doc?

d a few lines in the de

Do you mean in addition to user guide, we can mention pickle in dev guide? I feel you are right. We should do both here but with different emphasis. User guide should not mention pickle. Dev guide should mention pickle.

Not sure why the envs are not read correctly (probably because this PR is opened earlier than the PR which enables forked repo test)

Release candidate for v0.4.0

Please fix the tests.

The test fails because of credentials not found. I'm merging it to see the final CI, and will keep guys updated.