Repository navigation
Feature request: Be able to select multiple files in the "import CSV" dialog #1002
Description
At present our dialog for importing CSV files only allows the selection of a single one.
This turns out to be a fair PITA when needing to import quite a few CSV files into a single database. eg:
https://github.com/andreeaPascu/BikeCountsInEdinburgh/tree/master/outputFiles
The files there all use the same import options, so it would be useful to have the import
options dialog appear once with maybe a checkbox like "[ ] Use these import options
for the other CSV files selected too" to control the usage.
Activity
iKlsR commentedon May 18, 2017
Has this been started by any chance?
justinclift commentedon May 18, 2017
Nope, I wish. 😉
iKlsR commentedon May 18, 2017
If you can outline any more specifics on how this should be implemented I can tackle it this weekend. Currently in need of this and first time to the project page.
justinclift commentedon May 18, 2017
I hadn't really thought it through much, but as a first concept I'm thinking that the initial dialog itself needs to allow multiple-file selection (currently single-file only).
Then the current user facing dialog which presents options should probably also display some kind of list with the selected file names, along with a checkbox next to each (and select/deselect all toggle button).
And some way to apply the set of options to all of the selected files. eg "Import all".
When that runs, all of the checkboxed files would use those options. For any not-checkboxed, the dialog should probably pop up again (with the remaining files listed), allowing different options to be chosen this time.
... and loop around doing that until all of the files are imported.
Er... is that somewhat clear enough? Bearing in mind it's just an initial thought as to an approach. If some better way turns up while you're hacking on it, then by all means go with a different approach. 😀
iKlsR commentedon May 18, 2017
I can work with that, we can always iterate on the PR. Thanks.
karim commentedon May 18, 2017
To add to what @justinclift said, maybe a folder selection too?
If the open file dialog is active and you click on Open button without selecting any file, it will open/select all the files for import from the current directory? One problem for this is that all .csv files must be in the same format to be imported properly.
Also,
Ctrl+ click while the open dialog is open to select files one by one.MKleusberg commentedon May 19, 2017
@iKlsR If you need any technical assistance while working on this, just comment on this issue or write me an email and I'll try to help. I won't be available the entire weekend but you can still give it a try 😄
justinclift commentedon May 20, 2017
@iKlsR Read your initial compile report (tagged onto the end of #482), and yeah, I agree with much of it.
Sounds like you're up and running now? 😄
iKlsR commentedon May 20, 2017
@justinclift Seemed out of place with the actual issue? (didn't read most of the comments) so I removed it. Yup, I have a basic working first draft. I should have it up sometime later, have some other stuff to take care of.
justinclift commentedon May 20, 2017
No worries. Ironically I needed a break from the computer today, so have been keeping mostly offline. Just reading up on git internals on my tablet instead, as we need to adjust the data model we're using for the dbhub.io project.
Tomorrow (eg Sunday) I'll be mostly online (still doing dbhub.io stuff), but should be checking email and Gitter more often if that's useful. 😄
iKlsR commentedon May 23, 2017
Hi guys, real life hit like a sack of bricks so had to afk for a few. Finishing up and will open a PR shortly. Not sure if I will need to write tests as you will see but added some small additional functionality. Will need some help making it conform better to the org style and cleaning up the code, thanks.
@karim I think the features I added maybe negated the need for folder selection and the need for prompting for more files to import after, let me know however, keen to see this in after reviews as I have just used it to import some ~7000 files that hold thousands of rows of training data for a neural net.
If it turns out this went horribly in the wrong direction however ;), hopefully someone can pick some useful bits from my code if I can't find the time to edit.
Some minor thoughts on existing behavior
justinclift commentedon May 24, 2017
Muaaa Haaaa Haaaa. That's awesome. 7k files is definitely in the realm of "wouldn't want to do that by hand". 😁
For your PR, @MKleusberg is the right person to review it. Hopefully he has time to over the next few days.
@MKleusberg ?
MKleusberg commentedon Jun 16, 2017
@justinclift Want to try the new feature? I've just merged Ricardo's PR 😃
justinclift commentedon Jun 16, 2017
Sure can. Compiling now. 😄
justinclift commentedon Jun 16, 2017
Hmmm, when I select only a subset of files in the list, should the window close after the import?
As an example, clone the git repo here for a bunch of sample files:
https://github.com/andreeaPascu/BikeCountsInEdinburgh
Select the 48 individual files named "xx - [somename].csv", then click the Open button:
So far so good.
Click the "Deselect All" button to untick the lot, then choose the entry "04 - Seafield Street", and click "Match Similar". Everything but 4 entries should now be ticked.
Enable the "Separate tables" checkbox and click OK to start the import.
A series of progress dialogs then pops open, showing the individual imports happening.
After the selected one has completed, a confirmation popup shows... then the whole dialog disappears, showing the newly imported tables:
I was expecting the import dialog to still be showing, just listing the remaining few CSV files that I didn't select... eg thinking I'd be able to use different import options or something for these last few.
Does that sound workable? 😀
iKlsR commentedon Jun 16, 2017
@justinclift Ah, yes, you're right, forgot that bit of the workflow. I'll submit a followup pr later. Not a bug, long open PRs make your memory foggy :). Appreciate the detailed explanation.
justinclift commentedon Jun 16, 2017
Cool, you're welcome. 😄
iKlsR commentedon Jun 16, 2017
You can just press Match Similar right off the bat.
Keep testing!
justinclift commentedon Jun 16, 2017
Hmmm, how easy would it be to have each entry be removed from the list as it's import completes?
That way people could see the list shrink visibly onscreen, and the left over list would just still be there in the window when all the selected ones are done, ready for the user to choose new options and go again.
iKlsR commentedon Jun 16, 2017
@justinclift Shouldn't be hard. This whole process could be made even silkier by spinning up a thread or two but don't want to over engineer here, would also introduce concurrency which I don't see any existing use of in the repo.
iKlsR commentedon Jun 20, 2017
Update, started but haven't gotten the time to look back at this yet. It's a tad bit more involved than I expected. Not the list requirement but changing the old and current default behaviour of the dialog which overrode
accept()and actually cemented the operations when the dialog was closed. Will get to it as soon as I can.justinclift commentedon Jun 20, 2017
No worries @iKlsR. We all seem to be going through a busy stage atm. 😉
iKlsR commentedon Jul 31, 2017
@justinclift Could you move the last part into a new issue, thanks. It'll help when I tackle my assigns when the time comes.
justinclift commentedon Jul 31, 2017
Done, in #1072. 😄