Can one of the admins verify this patch?

• tensorflow/python/platform/default -> removed in 2b74220
• tensorflow/contrib/bayesflow/examples/** -> removed in 2ccf3ab
• tensorflow/contrib/ios_examples/** -> moved to tensorflow/examples/ios in fdb8e29
• tensorflow/contrib/learn/python/learn/dataframe -> removed in 27ad708
• tensorflow/contrib/metrics/ops -> moved to tensorflow/core/ops in 1af94c2
• tensorflow/contrib/periodic_resample/python/kernels -> never existed
• tensorflow/contrib/tensor_forest/core/ops -> moved to tensorflow/contrib/tensor_forest/kernels in 877aff8
• tensorflow/contrib/tensor_forest/data -> moved to tensorflow/contrib/tensor_forest/python/ops 70b6a5d

@jart The tests of PR #15166 match the lines inside some text files against the total directory structure to spot invalid paths as well as ring bells on non-existent but expected entries.

To do so, I have included //tensorflow:all_opensource_files as bazel data dependency which works just fine in principle. Although I had to work around Python resolving all imports (even generated) to the runfiles directory.

If you have a way at hand to access (a summary of) the directory structure (file contents not needed), I am of course all open ears.

@tensorflow-jenkins test this please

@jart I don't believe :all_opensource_files can be removed, or at least I don't know how. I thought that there are still rules that require data access to all files. Maybe this has changed?

The way I'd go about removing it, is by changing data = [":all_opensource_files"] to list legitimate top-level build targets instead, which is what :build_pip_package does. Then Bazel figures out all the transitive stuff.

This would bring some small benefits aside from cleanup. For example, :check_futures_test would be able to test generated sources too.

The :all_opensource_files pattern is mostly useful with tools like MOE when Bazel is building your tarball and you're willing to put up with the extra pain to avoid doing things like leaking the assembly code for some space age algorithm that got schlepped in from part of the monolithic codebase you've never seen.

So if we're 100% on board with Copybara, removing should be doable with some toil.

The point of the check_futures_test is that it can check files that you have not thought about. So unless we have a global target which is guaranteed to include all python files, I don't see how we can replace all_opensource_files for it.

And since bazel hides anything inside a directory with a BUILD file, all BUILD files must have their own target.

I now understand what you meant. As far as the :all_opensource_files pattern goes, we implement it in pretty much the best way possible (with the presubmit and all.) I'm just talking about alternatives with different tradeoffs.

For example, on the TensorBoard team, we went along with the Copybara vision of just running these sorts of tests and tools independently of Bazel, and therefore don't include :all_opensource_files rules. For example, we run pylint from our .travis.yml file.

That would be great solution -- we can add these tests to sanity, where we already run a bunch of unrelated test outside of bazel.

Jenkins, test this please.

Test failure check_futures_test due to files that were not found by the test before this cleanup.

Jenkins, test this please.

@martinwicke We should make e.g. check_futures_test a Sanity Check, not depending on things like all_opensource_files. Else we end up with silent errors like the ones I unexpectedly had to fix here.

I agree. We can probably make anything currently relying on all_opensource_files into a check run in sanity.

That already contains a function to get all changed python files, so running a regex over those to check for the future imports should be super-easy. I'd love a PR for that.

I disagree.
We should exclusively keep only quick running things on sanity build. Currently, makefile builds depends on all_opensource_files.

If @yifeif can move makefile to not use all_opensource_files, I am ok with what @martinwicke suggests.

The futures check should be very fast (much faster than, say, pylint: it only check for the existence of from future import lines in all changed python files).

I hadn't considered makefile -- why does that rely on all_opensource_files?

It is due to the new infra. All makefile build is wrapped in a bazel target.

Are we talking about wrapping makefile build output with bazel? The bazel part doesn't use all_opensource_files.

I am removing the dep to third_party/{fft2d,flatbuffers,eigen3} which we don't control fully and doesn't work well internally for now. Also, for some reason the contrib/mpi:all_files and mpi_collectives:all_files are missing. We should figure that out internally.