- Now
AnalyzeDataset,TransformDatasetandAnalyzeAndTransformDatasetcan accept input data that only contains columns needed for that operation as opposed to all columns defined in schema. Utility methods to infer the list of needed columns are added totft.inspect_preprocessing_fn. This makes it easier to take advantage of columnar projection when data is stored in columnar storage formats. - Python 3.5 is supported.
- Version is now accessible as
tensorflow_transform.__version__. - Depends on
apache-beam[gcp]>=2.11,<3. - Depends on
protobuf>=3.7,<4.
- Coders now return index and value features rather than a combined feature for
SparseFeature. - Requires pre-installed TensorFlow >=1.13,<2.
- Python 3.5 readiness complete (all tests pass). Full Python 3.5 compatibility is expected to be available with the next version of Transform (after Apache Beam 2.11 is released).
- Performance improvements for vocabulary generation when using top_k.
- New optimized highly experimental API for analyzing a dataset was added,
AnalyzeDatasetWithCache, which allows reading and writing analyzer cache. - Update
DatasetMetadatato be a wrapper around thetensorflow_metadata.proto.v0.schema_pb2.Schemaproto. TensorFlow Metadata will be the schema used to define data parsing across TFX. The serializedDatasetMetadatais now theSchemaproto in ascii format, but the previous format can still be read. - Change
ApplySavedModelimplementation to usetf.Session.make_callableinstead oftf.Session.runfor improved performance.
tft.vocabularyandtft.compute_and_apply_vocabularynow support filtering based on adjusted mutual information whenuse_adjusetd_mutual_infois set to True.tft.vocabularyandtft.compute_and_apply_vocabularynow takes regularization term 'min_diff_from_avg' that adjusts mutual information to zero whenever the difference between count of the feature with any label and its expected count is lower than the threshold.- Added an option to
tft.vocabularyandtft.compute_and_apply_vocabularyto compute a coverage vocabulary, using the newcoverage_top_k,coverage_frequency_thresholdandkey_fnparameters. - Added
tft.ptransform_analyzerfor advanced use cases. - Modified
QuantilesCombinerto usetf.Session.make_callableinstead oftf.Session.runfor improved performance. - ExampleProtoCoder now also supports non-serialized Example representations.
tft.tfidfnow accepts a scalar Tensor asvocab_size.assertItemsEqualin unit tests are replaced byassertCountEqual.NumPyCombinernow outputs TF dtypes in output_tensor_infos instead of numpy dtypes.- Adds function
tft.apply_pyfuncthat provides limited support fortf.pyfunc. Note that this is incompatible with serving. See documentation for more details. CombinePerKeynow adds a dimension for the key.- Depends on
numpy>=1.14.5,<2. - Depends on
apache-beam[gcp]>=2.10,<3. - Depends on
protobuf==3.7.0rc2. ExampleProtoCoder.encodenow converts a feature whose value isNoneto an empty value, where before it did not acceptNoneas a valid value.AnalyzeDataset,AnalyzeAndTransformDatasetandTransformDatasetcan now accept dictionaries which containNone, and which will be interpreted the same as an empty list. They will never produce an output containingNone.
ColumnSchemaand related classes (Domain,AxisandColumnRepresentationand their subclasses) have been removed. In order to create a schema, usefrom_feature_spec. In order to inspect a schema use theas_feature_specanddomainsmethods ofSchema. The constructors of these classes are replaced by functions that still work when creating aSchemabut this usage is deprecated.- Requires pre-installed TensorFlow >=1.12,<2.
ExampleProtoCoder.decodenow converts a feature with empty value (e.g.features { feature { key: "varlen" value { } } }) or missing key for a feature (e.g.features { }) to aNonein the output dictionary. Before it would represent these with an empty list. This better reflects the original example proto and is consistent with TensorFlow Data Validation.- Coders now returns a
listinstead of anndarrayfor aVarLenFeature.
- 'tft.vocabulary' and 'tft.compute_and_apply_vocabulary' now support filtering
based on mutual information when
labelsis provided. - Export all package level exports of
tensorflow_transform, from thetensorflow_transform.beamsubpackage. This allows users to just import thetensorflow_transform.beamsubpackage for all functionality. - Adding API docs.
- Fix bug where Transform returned a different dtype for a VarLenFeature with 0 elements.
- Depends on
apache-beam[gcp]>=2.8,<3.
- Requires pre-installed TensorFlow >=1.11,<2.
- All functions in
tensorflow_transform.saved.input_fn_makerare deprecated. See the examples for how to construct theinput_fnfor training and serving. Note that the examples demonstrate the use of thetf.estimatorAPI. The functions named *_serving_input_fn were for use with thetf.contrib.estimatorAPI which is now deprecated. We do not provide examples of usage of thetf.contrib.estimatorAPI, instead users should upgrade to thetf.estimatorAPI.
- Performance improvements for vocabulary generation when using top_k.
- Utility to deep-copy Beam
PCollections was added to avoid unnecessary materialization. - Utilize deep_copy to avoid unnecessary materialization of pcollections when
the input data is immutable. This feature is currently off by default and can
be enabled by setting
tft.Context.use_deep_copy_optimization=True. - Add bucketize_per_key which computes separate quantiles for each key and then bucketizes each value according to the quantiles computed for its key.
tft.scale_to_z_scoreis now implemented with a single pass over the data.- Export schema_utils package to convert from the
tensorflow-metadatapackage to the (soon to be deprecated)tf_metadatasubpackage oftensorflow-transform.
- Memory reduction during vocabulary generation.
- Clarify documentation on return values from
tft.compute_and_apply_vocabularyandtft.string_to_int. tft.unitnow explicitly creates Beam PCollections and validates the transformed dataset by writing and then reading it from disk.tft.min,tft.size,tft.sum,tft.scale_to_z_scoreandtft.bucketizenow supporttf.SparseTensor.- Fix to
tft.scale_to_z_scoreso it no longer attempts to divide by 0 when the variance is 0. - Fix bug where internal graph analysis didn't handle the case where an operation has control inputs that are operations (as opposed to tensors).
tft.sparse_tensor_to_dense_with_shapeadded which allows densifying aSparseTensorwhile specifying the resultingTensor's shape.- Add
load_transform_graphmethod toTFTransformOutputto load the transform graph without applying it. This has the effect of adding variables to the checkpoint when calling it from the traininginput_fnwhen usingtf.Estimator. - 'tft.vocabulary' and 'tft.compute_and_apply_vocabulary' now accept an
optional
weightsargument. Whenweightsis provided, weighted frequencies are used instead of frequencies based on counts. - 'tft.quantiles' and 'tft.bucketize' now accept an optoinal
weightsargument. Whenweightsis provided, weighted count is used for quantiles instead of the counts themselves. - Updated examples to construct the schema using
dataset_schema.from_feature_spec. - Updated the census example to allow the 'education-num' feature to be missing and fill in a default value when it is.
- Depends on
tensorflow-metadata>=0.9,<1. - Depends on
apache-beam[gcp]>=2.6,<3.
- We now validate a
Schemain its constructor to make sure that it can be converted to a feature spec. In particular onlytf.int64,tf.stringandtf.float32types are allowed. - We now disallow default values for
FixedColumnRepresentation. - It is no longer possible to set a default value in the Schema, and validation of shape parameters will occur earlier.
- Removed Schema.as_batched_placeholders() method.
- Removed all components of DatasetMetadata except the schema, and removed all related classes and code.
- Removed the merge method for DatasetMetadata and related classes.
- read_metadata can now only read from a single metadata directory and
read_metadata and write_metadata no longer accept the
versionsparameter. They now only read/write the JSON format. - Requires pre-installed TensorFlow >=1.9,<2.
apply_functionis no longer needed and is deprecated.apply_function(fn, *args)is now equivalent tofn(*args). tf.Transform is able to handle while loops and tables without the user wrapping the function call inapply_function.
- Add TFTransformOutput utility class that wraps the output of tf.Transform for use in training. This makes it easier to consume the output written by tf.Transform (see update examples for usage).
- Increase efficiency of
quantiles(and thereforebucketize).
- Change
tft.sum/tft.mean/tft.varto only support basic numeric types. - Widen the output type of
tft.sumfor some input types to avoid overflow and/or to preserve precision. - For int32 and int64 input types, change the output type of
tft.mean/tft.var/tft.scale_to_z_scorefrom float64 to float32 . - Change the output type of
tft.sizeto be always int64. Contextnow accepts passthrough_keys which can be used when additional information should be attached to dataset instances in the pipeline which should not be part of the transformation graph, for example: instance keys.- In addition to using TFTransformOutput, the examples demonstrate new workflows
where a vocabulary is computed, but not applied, in the
preprocessing_fn. - Added dependency on the absl-py package.
TransformTestCasetest cases can now be parameterized.- Add support for partitioned variables when loading a model.
- Export the
coderssubpackage so that users can access it astft.coders, e.g.tft.coders.ExampleProtoCoder. - Setting dtypes for numpy arrays in
tft.coders.ExampleProtoCoderandtft.coders.CsvCoder. tft.mean,tft.maxandtft.varnow supporttf.SparseTensor.- Update examples to use "core" TensorFlow estimator API (
tf.estimator). - Depends on
protobuf>=3.6.0<4.
apply_saved_transformis removed. See note onpartially_apply_saved_transformin theDeprecationssection.- No longer set
vocabulary_fileinIntDomainwhen usingtft.compute_and_apply_vocabularyortft.apply_vocabulary. - Requires pre-installed TensorFlow >=1.8,<2.
- The
expected_asset_file_contentsofTransformTestCase.assertAnalyzeAndTransformResultshas been deprecated, useexpected_vocab_file_contentsinstead. transform_fn_io.TRANSFORMED_METADATA_DIRandtransform_fn_io.TRANSFORM_FN_DIRshould not be used, they are now aliases forTFTransformOutput.TRANSFORMED_METADATA_DIRandTFTransformOutput.TRANSFORM_FN_DIRrespectively.partially_apply_saved_transformis deprecated, users should use thetransform_raw_featuresmethod ofTFTransformOuptutinstead. These differ in thatpartially_apply_saved_transformcan also be used to return both the input placeholders and the outputs. But users do not need this functionality because they will typically create the input placeholders themselves based on the feature spec.- Renamed
tft.uniquestotft.vocabulary,tft.string_to_inttotft.compute_and_apply_vocabularyandtft.apply_vocabtotft.apply_vocabulary. The existing methods will remain for a few more minor releases but are now deprecated and should get migrated away from.
- Depends on
apache-beam[gcp]>=2.4,<3. - Trim min/max value in
tft.bucketizewhere the computed number of bucket boundaries is more than requested. Updated documentation to clearly indicate that the number of buckets is computed using approximate algorithms, and that computed number can be more or less than requested. - Change the namespace used for Beam metrics from
tensorflow_transformtotfx.Transform. - Update Beam metrics to also log vocabulary sizes.
CsvCoderupdated to support unicode.- Update examples to not use the
coderargument for IO, and instead use a separatebeam.Mapto encode/decode data.
- Requires pre-installed TensorFlow >=1.6,<2.
- Batching of input instances is now done automatically and dynamically.
- Added analyzers to compute covariance matrices (
tft.covariance) and principal components for PCA (tft.pca). - CombinerSpec and combine_analyzer now accept multiple inputs/outputs.
- Depends on
apache-beam[gcp]>=2.3,<3. - Fixes a bug where TransformDataset would not return correct output if the output DatasetMetadata contained deferred values (such as vocabularies).
- Added checks that the prepreprocessing function's outputs all have the same size in the batch dimension.
- Added
tft.apply_bucketswhich takes an input tensor and a list of bucket boundaries, and returns bucketized data. tft.bucketizeandtft.apply_bucketsnow set metadata for the output tensor, which means the resulting tf.Metadata for the output of these functions will contain min and max values based on the number of buckets, and also be set to categorical.- Testing helper function assertAnalyzeAndTransformResults can now also test the content of vocabulary files and other assets.
- Reduces the number of beam stages needed for certain analyzers, which can be a performance bottleneck when transforming many features.
- Performance improvements in
tft.uniques. - Fix a bug in
tft.bucketizewhere the bucket boundary could be same as a min/max value, and was getting dropped. - Allows scaling individual components of a tensor independently with
tft.scale_by_min_max,tft.scale_to_0_1, andtft.scale_to_z_score. - Fix a bug where
apply_saved_transformcould only be applied in the global name scope. - Add warning when
frequency_thresholdthat are <= 1. This is a no-op and generally reflects mistakingfrequency_thresholdfor a relative frequency where in fact it is an absolute frequency.
- The interfaces of CombinerSpec and combine_analyzer have changed to allow for multiple inputs/outputs.
- Requires pre-installed TensorFlow >=1.5,<2.
- Added a combine_analyzer() that supports user provided combiner, conforming to beam.CombinFn(). This allows users to implement custom combiners (e.g. median), to complement analyzers (like min, max) that are prepackaged in TFT.
- Quantiles Analyzer (
tft.quantiles), with a correspondingtft.bucketizemapper.
- Depends on
apache-beam[gcp]>=2.2,<3. - Fixes some KeyError issues that appeared in certain circumstances when one would call AnalyzeAndTransformDataset (due to a now-fixed Apache Beam [bug] (https://issues.apache.org/jira/projects/BEAM/issues/BEAM-2966)).
- Allow all functions that accept and return tensors, to accept an optional name scope, in line with TensorFlow coding conventions.
- Update examples to construct input functions by hand instead of using helper functions.
- Change scale_by_min_max/scale_to_0_1 to return the average(min, max) of the range in case all values are identical.
- Added export of serving model to examples.
- Use "core" version of feature columns (tf.feature_column instead of tf.contrib) in examples.
- A few bug fixes and improvements for coders regarding Python 3.
- Requires pre-installed TensorFlow >= 1.4.
- No longer distributing a WHL file in PyPI. Only doing a source distribution
which should however be compatible with all platforms (ie you are still able
to
pip install tensorflow-transformand userequirements.txtorsetup.pyfiles for environment setup). - Some functions now introduce a new name scope when they did not before so the names of tensors may change. This will only affect you if you directly lookup tensors by name in the graph produced by tf.Transform.
- Various Analyzer Specs (_NumericCombineSpec, _UniquesSpec, _QuantilesSpec) are now private. Analyzers are accessible only via the top-level TFT functions (min, max, sum, size, mean, var, uniques, quantiles).
- The
serving_input_fns ontensorflow_transform/saved/input_fn_maker.pywill be removed on a future version and should not be used on new code, see theexamplesdirectory for details on how to migrate your code to define their own serving functions.
- We now provide helper methods for creating
serving_input_receiver_fnfor use with tf.estimator. These mirror the existing functions targeting the legacy tf.contrib.learn.estimators-- i.e. for each*_serving_input_fn()in input_fn_maker there is now also a*_serving_input_receiver_fn().
- Introduced
tft.apply_vocabthis allows users to separately apply a single vocabulary (as generated bytft.uniques) to several different columns. - Provide a source distribution tar
tensorflow-transform-X.Y.Z.tar.gz.
- The default prefix for
tft.string_to_intvocab_filenamechanged fromvocab_string_to_inttovocab_string_to_int_uniques. To make your pipelines resilient to implementation details please setvocab_filenameif you are using the generated vocab_filename on a downstream component.
- Added hash_strings mapper.
- Write vocabularies as asset files instead of constants in the SavedModel.
- 'tft.tfidf' now adds 1 to idf values so that terms in every document in the corpus have a non-zero tfidf value.
- Performance and memory usage improvement when running with Beam runners that use multi-threaded workers.
- Performance optimizations in ExampleProtoCoder.
- Depends on
apache-beam[gcp]>=2.1.1,<3. - Depends on
protobuf>=3.3<4. - Depends on
six>=1.9,<1.11.
- Requires pre-installed TensorFlow >= 1.3.
- Removed
tft.mapusetft.apply_functioninstead (as needed). - Removed
tft.tfidf_weightsusetft.tfidfinstead. beam_metadata_io.WriteMetadatanow requires a secondpipelineargument (see examples).- A Beam bug will now affect users who call AnalyzeAndTransformDataset in
certain circumstances. Roughly speaking, if you call
beam.Pipeline()at some point (as all our examples do) you will not experience this bug. The bug is characterized by an error similar toKeyError: (u'AnalyzeAndTransformDataset/AnalyzeDataset/ComputeTensorValues/Extract[Maximum:0]', None)This bug will be fixed in Beam 2.2.
- Add json-example serving input functions to TF.Transform.
- Add variance analyzer to tf.transform.
- Remove duplication in output of
tft.tfidf. - Ensure ngrams output dense_shape is greater than or equal to 0.
- Alters the behavior and interface of tensorflow_transform.mappers.ngrams.
- Depends on
apache-beam[gcp]=>2,<3. - Making TF Parallelism runner-dependent.
- Fixes issue with csv serving input function.
- Various performance and stability improvements.
tft.mapwill be removed on version 0.2.0, see theexamplesdirectory for instructions on how to usetft.apply_functioninstead (as needed).tft.tfidf_weightswill be removed on version 0.2.0, usetft.tfidfinstead.
- Refactor internals to remove Column and Statistic classes
- Remove collections from graph to avoid warnings
- Return float32 from
tfidf_weights - Update tensorflow_transform to use
tf.saved_modelAPIs. - Add default values on example proto coder.
- Various performance and stability improvements.