this can be expensive on big files, maybe limit to the first 20MB to estimate the total number of lines ?

we could also assume that the files contain the same kind of data, so maybe there is no need to check every single file

That's a great point. Iterating through massive files just for a line count is indeed a bottleneck.

IIUC when num_shards is None then it should shard maximally.

In the case of the CSV loader, the minimum shard size is defined by self.config.chunksize which is 10_000 lines by default. It could make sense to define maximum sharding as aiming for self.config.chunksize lines per shard, WDYT ?

Makes sense. I'll refactor this part to

skiprows may lead some shard to require downloading a lot of unnecessary data to skip the rows, maybe you could skip bytes instead. Something along those lines maybe ?

if skip_bytes is not None:
    with open(file, "rb") as f:
        header = f.readline()
        f.seek(skip_bytes)
        lines = (f.read(n_bytes) + f.readline()).splitlines()
        lines[0] = header
    file = io.BytesIO("\n".join(lines))
csv_file_reader = pd.read_csv(file, iterator=True, dtype=dtype, **read_csv_kwargs)

e.g. this dataset has multiple CSV files of 4GB each: https://huggingface.co/datasets/Koala-36M/Koala-36M-v1

cool, I will try with it

it's probably more practical to have them together like this:

this way later you can do

        for shard_idx, files_iterable in enumerate(files_iterables):
            for file in files_iterable:
                shard_start_line, shard_end_line = None, None
                if isinstance(file, tuple):
                    file, shard_start_line, shard_end_line = file

alternatively you could keep it this way, but in that case you need to zip() on files_iterables:

        shard_start_lines = shard_start_lines or [None] * len(base_files)
        shard_end_lines = shard_start_lines or [None] * len(base_files)
        for base_file, files_iterable, shard_start_line, shard_end_line in zip(base_files, files_iterables, shard_start_lines, shard_end_lines):
            for file in files_iterable:

when sharding maximally we could use the fact that self.config.chunksize is 10MB by default and aim for this maybe

(yes it has the same name as CSV but is in bytes, not lines. This collision is not ideal and I guess comes from the fact that pandas uses chunksize in lines for CSV and pyarrow uses chunksize in bytes for json)

got it