Merge pull request tensorflow#11 from xinzha623/master

Project import generated by Copybara.

README.md

@@ -8,6 +8,8 @@ distributed manner, using the same metrics defined in their trainer. These
 metrics can be computed over different slices of data and visualized in Jupyter
 notebooks.
 
+![TFMA Slicing Metrics Browser](g3doc/images/tfma-slicing-metrics-browser.gif)
+
 Caution: TFMA may introduce backwards incompatible changes before version 1.0.
 
 ## Installation
@@ -42,6 +44,11 @@ By default, Apache Beam runs in local mode but can also run in distributed mode
 using [Google Cloud Dataflow](https://cloud.google.com/dataflow/). TFMA is
 designed to be extensible for other Apache Beam runners.
 
+## Getting Started
+
+For instructions on using TFMA, see the [get started
+guide](g3doc/get_started.md) and try out the extensive [end-to-end example](examples/chicago_taxi/README.md).
+
 ## Compatible Versions
 
 The following table is the TFMA package versions that are compatible with each
@@ -50,7 +57,8 @@ combinations may also work.
 
 |tensorflow-model-analysis  |tensorflow    |apache-beam[gcp]|
 |---------------------------|--------------|----------------|
-|GitHub master              |1.7           |2.4.0           |
+|GitHub master              |1.9           |2.5.0           |
+|0.9.0                      |1.9           |2.5.0           |
 |0.6.0                      |1.6           |2.4.0           |
 
 ## Questions

RELEASE.md

@@ -1,23 +1,36 @@
-# Current version (not yet released; still in development)
+# Release 0.9.0
 
 ## Major Features and Improvements
 
+*   Add a TFMA unit test library for unit testing your the exported model and
+    associated metrics computations.
 *   Add `tfma.export.make_export_strategy` which is analogous to
     `tf.contrib.learn.make_export_strategy`.
 *   Add `tfma.exporter.FinalExporter` and `tfma.exporter.LatestExporter` which
     are analogous to `tf.estimator.FinalExporter` and
     `tf.estimator.LastExporter`.
 *   Add `tfma.export.build_parsing_eval_input_receiver_fn` which is analogous to
     `tf.estimator.export.build_parsing_serving_input_receiver_fn`.
-*   Add a new post export metric
-    `tfma.post_export_metrics.post_export_metrics.auc()`.
+*   Add integration testing for DNN-based estimators.
+*   Add new post export metrics:
+    *   AUC (`tfma.post_export_metrics.post_export_metrics.auc`)
+    *   Precision/Recall at K
+        (`tfma.post_export_metrics.post_export_metrics.precision_recall_at_k`)
+    *   Confusion matrix at thresholds
+        (`tfma.post_export_metrics.post_export_metrics.confusion_matrix_at_thresholds`)
 
 ## Bug fixes and other changes
 
- * Peak memory usage for large DataFlow jobs should be lower with a fix in when
-   we compact batches of metrics during the combine phase of metrics
-   computation.
- * Remove batch size override in `chicago_taxi` example.
+*   Peak memory usage for large DataFlow jobs should be lower with a fix in when
+    we compact batches of metrics during the combine phase of metrics
+    computation.
+*   Remove batch size override in `chicago_taxi` example.
+*   Added dependency on `protobuf>=3.6.0<4` for protocol buffers.
+*   Updated SparseTensor code to work with SparseTensors of any dimension.
+    Previously on SparseTensors with dimension 2 (batch_size x values) were
+    supported in the features dictionary.
+*   Updated code to work with SparseTensors and dense Tensors of variable
+    lengths across batches.
 
 ## Breaking changes
 
@@ -27,9 +40,19 @@
         tag, as opposed to the "serving" tag before.
     *   EvalSavedModels now include version metadata about the TFMA version that
         they were exported with.
+*   Metrics and plot outputs now are converted into proto and serialized.
+    Metrics and plots produced by TFMA 0.6.0 will not be compatible with later
+    versions.
+*   Requires pre-installed TensorFlow >=1.9,<2.
+*   TFMA now uses the TensorFlow Estimator functionality for exporting models of
+    different modes behind the scenes. There are no user-facing changes
+    API-wise, but EvalSavedModels produced by earlier versions of TFMA will not
+    be compatible with this version of TFMA.
+*   tf.contrib.learn Estimators are no longer supported by TFMA. Only
+    tf.estimator Estimators are supported.
 *   Metrics and plot outputs now include version metadata about the TFMA version
-    that they were exported with. Metrics and plots produced by TFMA 0.6.0 will
-    not be compatible with later versions.
+    that they were exported with. Metrics and plots produced by earlier versions
+    of TFMA will not be compatible with this version of TFMA.
 
 ## Deprecations
 

WORKSPACE

@@ -2,11 +2,11 @@ workspace(name = "org_tensorflow_model_analysis")
 
 http_archive(
     name = "io_bazel_rules_closure",
-    sha256 = "6691c58a2cd30a86776dd9bb34898b041e37136f2dc7e24cadaeaf599c95c657",
-    strip_prefix = "rules_closure-08039ba8ca59f64248bb3b6ae016460fe9c9914f",
+    sha256 = "424dce95fddfea8dcf8012a4749fd7166a3009d8d7c32942f19dff12d0bbc2e8",
+    strip_prefix = "rules_closure-35ffe0eec59ce21ea6b04a8e3345cfcfcf20f5ed",
     urls = [
-        "https://mirror.bazel.build/github.com/bazelbuild/rules_closure/archive/08039ba8ca59f64248bb3b6ae016460fe9c9914f.tar.gz",
-        "https://github.com/bazelbuild/rules_closure/archive/08039ba8ca59f64248bb3b6ae016460fe9c9914f.tar.gz",  # 2018-01-16
+        "https://mirror.bazel.build/github.com/bazelbuild/rules_closure/archive/35ffe0eec59ce21ea6b04a8e3345cfcfcf20f5ed.tar.gz",
+        "https://github.com/bazelbuild/rules_closure/archive/35ffe0eec59ce21ea6b04a8e3345cfcfcf20f5ed.tar.gz",  # 2018-04-12
     ],
 )
 
@@ -15,9 +15,9 @@ closure_repositories()
 
 http_archive(
     name = "org_tensorflow_tensorboard",
-    sha256 = "a943c0242a07da4d445135ffc9a7c7cb987d9bd948ae733695bc16095dceec20",
-    strip_prefix = "tensorboard-2fdb2199553729a6c5b42b7eb0305a101b454add",
-    urls = ["https://github.com/tensorflow/tensorboard/archive/2fdb2199553729a6c5b42b7eb0305a101b454add.zip"],
+    sha256 = "bd5ce4676158c8e00de43e763e1e6e699b97cade0ce841d5e4b896e3e733dbec",
+    strip_prefix = "tensorboard-270d34d1e2b0fc9401db0549004e4a0f0f1ffd2d",
+    urls = ["https://github.com/tensorflow/tensorboard/archive/270d34d1e2b0fc9401db0549004e4a0f0f1ffd2d.zip"],  # 2018-04-17
 )
 
 load("@org_tensorflow_tensorboard//third_party:workspace.bzl", "tensorboard_workspace")

examples/chicago_taxi/README.md

@@ -4,7 +4,7 @@ The Chicago Taxi example demonstrates the end-to-end workflow and steps of how
 to transform data, train a model, analyze and serve it. It uses:
 
 * [TensorFlow Transform](https://www.tensorflow.org/tfx/transform) for feature preprocessing,
-* TensorFlow [Estimators](https://www.tensorflow.org/programmers_guide/estimators)
+* TensorFlow [Estimators](https://www.tensorflow.org/guide/estimators)
 for training,
 * [TensorFlow Model Analysis](https://www.tensorflow.org/tfx/model_analysis) and Jupyter for evaluation, and
 * [TensorFlow Serving](https://www.tensorflow.org/serving) for serving.
@@ -145,7 +145,7 @@ bash ./train_local.sh
 </pre>
 
 The model leverages TensorFlow’s
-[Estimators](/programmers_guide/estimators) and is created in the
+[Estimators](/guide/estimators) and is created in the
 `build_estimator` function in `model.py`. The trainer's input takes the
 materialized, transformed examples from the previous step. Notice the pattern of
 sharing schema information between preprocessing and training using `taxi.py`
@@ -397,7 +397,7 @@ notebook and set up the output directory to see the results.
 To serve the model from the cloud, run:
 
 <pre class="devsite-terminal devsite-click-to-copy">
-bash ./start_model_server_mlengine.sh
+bash ./start_model_server_cloud.sh
 </pre>
 
 To send a request to the cloud:

examples/chicago_taxi/setup.py

@@ -15,11 +15,12 @@
 import setuptools
 
 if __name__ == '__main__':
-  setuptools.setup(name='chicago_taxi_setup', version='1.0',
-                   packages=setuptools.find_packages(),
-                   install_requires=[
-                       'jupyter==1.0',
-                       'tensorflow==1.6.0',
-                       'tensorflow-model-analysis==0.6.0',
-                       'tensorflow-serving-api==1.6.0',
-                       'tensorflow-transform==0.6.0'])
+  setuptools.setup(
+      name='chicago_taxi_setup',
+      version='1.0',
+      packages=setuptools.find_packages(),
+      install_requires=[
+          'jupyter==1.0', 'tensorflow==1.9.0',
+          'tensorflow-model-analysis==0.9.0', 'tensorflow-serving-api==1.9.0',
+          'tensorflow-transform==0.8.0'
+      ])

g3doc/get_started.md

@@ -124,3 +124,11 @@ with beam.Pipeline(runner=...) as p:
 result = tfma.load_eval_result(output_path='/path/to/out')
 tfma.view.render_slicing_metrics(result)
 ```
+## End-to-end example
+
+Try the extensive [end-to-end example](../examples/chicago_taxi/README.md)
+featuring [TensorFlow Transform](https://github.com/tensorflow/transform) for feature
+preprocessing,
+[TensorFlow Estimators](https://www.tensorflow.org/programmers_guide/estimators) for
+training, [TensorFlow Model Analysis](https://github.com/tensorflow/model-analysis) and Jupyter for
+evaluation, and [TensorFlow Serving](https://github.com/tensorflow/serving) for serving.

setup.py

@@ -243,17 +243,16 @@ def run(self):
         'tensorflow_model_analysis/static/vulcanized_template.html',
     ]),],
     'install_requires': [
-        'apache-beam[gcp]>=2.4,<3',
+        'apache-beam[gcp]>=2.5,<3',
         'grpc-google-iam-v1==0.11.1',
         'numpy>=1.10,<2',
         'jupyter>=1.0,<2',
         'ipywidgets>=7.0,<8',
-        # Protobuf libraries < 3.5.2 do not have 'cpp' implementation of
-        # protobufs for Windows and Mac.
-        'protobuf>=3.5.2,<4',
+        # TF now requires protobuf>=3.6.0.
+        'protobuf>=3.6.0,<4',
         # For apitools.
         'six>=1.9,<2',
-        'tensorflow-transform>=0.6,<1',
+        'tensorflow-transform>=0.8,<1',
     ],
     'python_requires':
         '>=2.7,<3',

tensorflow_model_analysis/BUILD

@@ -27,6 +27,12 @@ py_library(
     ],
 )
 
+py_library(
+    name = "util",
+    srcs = ["util.py"],
+    deps = ["//tensorflow_model_analysis:expect_six_installed"],
+)
+
 py_library(
     name = "version",
     srcs = ["version.py"],

tensorflow_model_analysis/__init__.py

@@ -15,6 +15,7 @@
 
 
 from tensorflow_model_analysis import view
+from tensorflow_model_analysis.api import tfma_unit as test
 from tensorflow_model_analysis.api.model_eval_lib import *  # pylint: disable=wildcard-import
 from tensorflow_model_analysis.constants import DATA_CENTRIC_MODE
 from tensorflow_model_analysis.constants import MODEL_CENTRIC_MODE

tensorflow_model_analysis/api/BUILD

@@ -20,6 +20,7 @@ py_library(
 py_test(
     name = "model_eval_lib_test",
     srcs = ["model_eval_lib_test.py"],
+    shard_count = 3,
     deps = [
         ":model_eval_lib",
         "//tensorflow_model_analysis:expect_tensorflow_installed",
@@ -29,6 +30,38 @@ py_test(
         "//tensorflow_model_analysis/eval_saved_model/example_trainers:fixed_prediction_estimator",
         "//tensorflow_model_analysis/eval_saved_model/example_trainers:linear_classifier",
         "//tensorflow_model_analysis/eval_saved_model/post_export_metrics",
+        "//tensorflow_model_analysis/proto:metrics_for_slice_py_pb2",
         "//tensorflow_model_analysis/slicer",
     ],
 )
+
+py_library(
+    name = "tfma_unit_non_testonly",
+    srcs = ["tfma_unit.py"],
+    deps = [
+        "//tensorflow_model_analysis:expect_apache_beam_installed",
+        "//tensorflow_model_analysis/api/impl:evaluate",
+        "//tensorflow_model_analysis/eval_saved_model:load",
+        "//tensorflow_model_analysis/eval_saved_model:testutil_non_testonly",
+        "//tensorflow_model_analysis/slicer",
+    ],
+)
+
+py_library(
+    name = "tfma_unit",
+    testonly = 1,
+    deps = ["tfma_unit_non_testonly"],
+)
+
+py_test(
+    name = "tfma_unit_test",
+    srcs = ["tfma_unit_test.py"],
+    deps = [
+        ":tfma_unit",
+        "//tensorflow_model_analysis:expect_apache_beam_installed",
+        "//tensorflow_model_analysis:expect_tensorflow_installed",
+        "//tensorflow_model_analysis/eval_saved_model/example_trainers:fixed_prediction_estimator",
+        "//tensorflow_model_analysis/eval_saved_model/example_trainers:fixed_prediction_estimator_extra_fields",
+        "//tensorflow_model_analysis/eval_saved_model/post_export_metrics",
+    ],
+)

tensorflow_model_analysis/api/impl/BUILD

@@ -14,13 +14,15 @@ py_library(
         "//tensorflow_model_analysis/eval_saved_model:util",
         "//tensorflow_model_analysis/eval_saved_model/post_export_metrics:metric_keys",
         "//tensorflow_model_analysis/extractors:feature_extractor",
+        "//tensorflow_model_analysis/proto:metrics_for_slice_py_pb2",
         "//tensorflow_model_analysis/slicer",
     ],
 )
 
 py_test(
     name = "evaluate_test",
     srcs = ["evaluate_test.py"],
+    shard_count = 4,
     deps = [
         ":evaluate",
         "//tensorflow_model_analysis:expect_apache_beam_installed",  # b/73825929
@@ -30,6 +32,7 @@ py_test(
         "//tensorflow_model_analysis/eval_saved_model/example_trainers:linear_classifier",
         "//tensorflow_model_analysis/eval_saved_model/post_export_metrics",
         "//tensorflow_model_analysis/eval_saved_model/post_export_metrics:metric_keys",
+        "//tensorflow_model_analysis/proto:metrics_for_slice_py_pb2",
         "//third_party/tensorflow/core:protos_all_py_pb2",
     ],
 )
@@ -42,6 +45,9 @@ py_library(
         "//tensorflow_model_analysis:expect_apache_beam_installed",  # b/73825929
         "//tensorflow_model_analysis:expect_tensorflow_installed",
         "//tensorflow_model_analysis:version",
+        "//tensorflow_model_analysis/api/impl:evaluate",
+        "//tensorflow_model_analysis/eval_saved_model/post_export_metrics",
+        "//tensorflow_model_analysis/proto:metrics_for_slice_py_pb2",
         "//tensorflow_model_analysis/slicer",
         "//third_party/py/numpy",
     ],
@@ -52,7 +58,9 @@ py_test(
     srcs = ["serialization_test.py"],
     deps = [
         ":serialization",
+        "//tensorflow_model_analysis:types",
         "//tensorflow_model_analysis/eval_saved_model:testutil",
+        "//tensorflow_model_analysis/eval_saved_model/post_export_metrics:metric_keys",
     ],
 )
 

tensorflow_model_analysis/api/impl/evaluate.py

@@ -368,13 +368,15 @@ def _Aggregate(  # pylint: disable=invalid-name
     add_metrics_callbacks,
     desired_batch_size = None,
 ):
-  return (slice_result
-          | 'CombinePerKey' >> beam.CombinePerKey(
-              _AggregateCombineFn(
-                  eval_saved_model_path=eval_saved_model_path,
-                  add_metrics_callbacks=add_metrics_callbacks,
-                  shared_handle=shared.Shared(),
-                  desired_batch_size=desired_batch_size)))
+  return (
+      slice_result
+      | 'CombinePerKey' >> beam.CombinePerKey(
+          _AggregateCombineFn(
+              eval_saved_model_path=eval_saved_model_path,
+              add_metrics_callbacks=add_metrics_callbacks,
+              shared_handle=shared.Shared(),
+              desired_batch_size=desired_batch_size))
+  )
 
 
 @beam.typehints.with_input_types(beam.typehints.Tuple[