Merge pull request tensorflow#7 from chuanyu/master

Project import generated by Copybara.

ISSUE_TEMPLATE.md

@@ -0,0 +1,45 @@
+Please go to Stack Overflow for help and support:
+
+https://stackoverflow.com/questions/tagged/tensorflow-model-analysis
+
+If you open a GitHub issue, here is our policy:
+
+1.  It must be a bug, a feature request, or a significant problem with
+    documentation (for small docs fixes please send a PR instead).
+2.  The form below must be filled out.
+
+**Here's why we have that policy**: TensorFlow Model Analysis developers respond
+to issues. We want to focus on work that benefits the whole community, e.g.,
+fixing bugs and adding features. Support only helps individuals. GitHub also
+notifies thousands of people when issues are filed. We want them to see you
+communicating an interesting problem, rather than being redirected to Stack
+Overflow.
+
+--------------------------------------------------------------------------------
+
+### System information
+
+-   **Have I written custom code (as opposed to using a stock example script
+    provided in TensorFlow Model Analysis)**:
+-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
+-   **TensorFlow Model Analysis installed from (source or binary)**:
+-   **TensorFlow Model Analysis version (use command below)**:
+-   **Python version**:
+-   **Jupyter Notebook version**:
+-   **Exact command to reproduce**:
+
+You can obtain the TensorFlow Model Analysis version with
+
+`python -c "import tensorflow_model_analysis as tfma; print(tfma.VERSION_STRING)"`
+
+### Describe the problem
+
+Describe the problem clearly here. Be sure to convey here why it's a bug in
+TensorFlow Model Analysis or a feature request.
+
+### Source code / logs
+
+Include any logs or source code that would be helpful to diagnose the problem.
+If including tracebacks, please include the full traceback. Large logs and files
+should be attached. Try to provide a reproducible test case that is the bare
+minimum necessary to generate the problem.

README.md

@@ -33,7 +33,12 @@ distributed mode using
 [Google Cloud Dataflow](https://cloud.google.com/dataflow/). TFMA is designed to
 be extensible to other Apache Beam runners.
 
-### Compatible Versions
+## Getting Started
+
+For instructions on using TFMA, see the [getting started
+guide](g3doc/getting_started.md).
+
+## Compatible Versions
 
 This is a table of versions known to be compatible with each other, based on
 our testing framework. Other combinations may also work, but are untested.
@@ -43,11 +48,6 @@ our testing framework. Other combinations may also work, but are untested.
 |GitHub master              |1.7           |2.4.0           |
 |0.6.0                      |1.6           |2.4.0           |
 
-## Getting Started
-
-For instructions on using TFMA, see the [getting started
-guide](g3doc/getting_started.md).
-
 <sup>1</sup> If Jupyter is installed in your home directory, add `--user` for
     all commands; if Jupyter is installed in root or virtualenv is used,
     `--sys-prefix` might be needed.

RELEASE.md

@@ -1,19 +1,24 @@
 # Current version (not yet released; still in development)
 
 ## Major Features and Improvements
- * Add `tfma.export.make_export_strategy` which is analogous to
-   `tf.contrib.learn.make_export_strategy`.
- * Add `tfma.exporter.FinalExporter` and `tfma.exporter.LatestExporter` which
-   are analogous to `tf.estimator.FinalExporter` and
-   `tf.estimator.LastExporter`.
+
+*   Add `tfma.export.make_export_strategy` which is analogous to
+    `tf.contrib.learn.make_export_strategy`.
+*   Add `tfma.exporter.FinalExporter` and `tfma.exporter.LatestExporter` which
+    are analogous to `tf.estimator.FinalExporter` and
+    `tf.estimator.LastExporter`.
+*   Add `tfma.export.build_parsing_eval_input_receiver_fn` which is analogous to
+    `tf.estimator.export.build_parsing_serving_input_receiver_fn`.
 
 ## Bug fixes and other changes
+
  * Peak memory usage for large DataFlow jobs should be lower with a fix in when
    we compact batches of metrics during the combine phase of metrics
    computation.
  * Remove batch size override in `chicago_taxi` example.
 
 ## Breaking changes
+
  * EvalSavedModels produced by TFMA 0.6.0 will not be compatible with TFMA 0.7.0
    due to the following changes:
    * EvalSavedModels are now written out with a custom "eval_saved_model" tag,
@@ -27,4 +32,5 @@
 ## Deprecations
 
 # Release 0.6.0
+
  * Initial release of TensorFlow Model Analysis.

g3doc/getting_started.md

@@ -21,7 +21,7 @@ export the *EvalSavedModel*. You can do this by adding a call to
 
 The following code snippet illustrates this:
 
-```
+```python
 # Define, train and export your estimator as usual
 estimator = tf.estimator.DNNClassifier(...)
 estimator.train(...)
@@ -42,7 +42,7 @@ features. It should additionally parse and return the label.
 The following code snippet illustrates how you might define an
 `eval_input_receiver_fn`:
 
-```
+```python
 country = tf.contrib.layers.sparse_column_with_hash_buckets('country', 100)
 language = tf.contrib.layers.sparse_column_with_hash_buckets(language, 100)
 age = tf.contrib.layers.real_valued_column('age')
@@ -91,7 +91,7 @@ The quickest way to try it out is to use `tfma.run_model_analysis` to perform
 the evaluation. Note that this uses Beam's local runner, so it's mainly for
 quick small-scale experimentation locally. The following code snippet shows how:
 
-```
+```python
 # Note that this code should be run in a Jupyter Notebook.
 
 # This assumes your data is a TFRecords file containing records in the format
@@ -119,7 +119,7 @@ perform the evaluation and write the results out. The results can later be
 loaded for visualization using `tfma.load_eval_result`. The following snippet
 illustrates this:
 
-```
+```python
 # To run the pipeline.
 with beam.Pipeline(runner=...) as p:
   _ = (p

tensorflow_model_analysis/__init__.py

@@ -22,6 +22,7 @@
 from tensorflow_model_analysis.eval_saved_model import exporter
 from tensorflow_model_analysis.eval_saved_model import post_export_metrics
 from tensorflow_model_analysis.slicer.slicer import SingleSliceSpec
+from tensorflow_model_analysis.version import VERSION_STRING
 
 
 def _jupyter_nbextension_paths():

tensorflow_model_analysis/api/impl/serialization.py

@@ -186,7 +186,7 @@ class WriteMetricsPlotsAndConfig(beam.PTransform):
   """Writes metrics, plots and config to the given path.
 
   This is the internal implementation. Users should call
-  model_eval_lib.WriteMetricsAndPlots instead of this.
+  tfma.EvaluateAndWriteResults() instead, which calls this.
   """
 
   def __init__(self, output_path,

tensorflow_model_analysis/eval_saved_model/constants.py

@@ -13,4 +13,5 @@
 # limitations under the License.
 """Constants for the EvalSavedModel."""
 
+EVAL_SAVED_MODEL_EXPORT_NAME = 'TFMA'
 EVAL_SAVED_MODEL_TAG = 'eval_saved_model'

tensorflow_model_analysis/eval_saved_model/export.py

@@ -281,6 +281,37 @@ def export_eval_savedmodel(
       return export_dir
 
 
+def build_parsing_eval_input_receiver_fn(
+    feature_spec, label_key):
+  """Build a eval_input_receiver_fn expecting fed tf.Examples.
+
+  Creates a eval_input_receiver_fn that expects a serialized tf.Example fed
+  into a string placeholder.  The function parses the tf.Example according to
+  the provided feature_spec, and returns all parsed Tensors as features.
+
+  Args:
+    feature_spec: A dict of string to `VarLenFeature`/`FixedLenFeature`.
+    label_key: The key for the label column in the feature_spec. Note that
+      the label must be part of the feature_spec.
+
+  Returns:
+    A eval_input_receiver_fn suitable for use with TensorFlow model analysis.
+  """
+
+  def eval_input_receiver_fn():
+    """An input_fn that expects a serialized tf.Example."""
+    # Note it's *required* that the batch size should be variable for TFMA.
+    serialized_tf_example = tf.placeholder(
+        dtype=tf.string, shape=[None], name='input_example_tensor')
+    features = tf.parse_example(serialized_tf_example, feature_spec)
+    return EvalInputReceiver(
+        features=features,
+        receiver_tensors={'examples': serialized_tf_example},
+        labels=features[label_key])
+
+  return eval_input_receiver_fn
+
+
 def make_export_strategy(
     eval_input_receiver_fn,
     exports_to_keep = 5):
@@ -314,4 +345,5 @@ def export_fn(estimator,
         export_dir_base, exports_to_keep)
     return export_dir
 
-  return tf.contrib.learn.ExportStrategy('TFMA', export_fn)
+  return tf.contrib.learn.ExportStrategy(constants.EVAL_SAVED_MODEL_EXPORT_NAME,
+                                         export_fn)

tensorflow_model_analysis/eval_saved_model/exporter.py

@@ -24,7 +24,7 @@
 import tensorflow as tf
 
 from tensorflow_model_analysis.eval_saved_model import export
-from tensorflow_model_analysis.types_compat import Callable
+from tensorflow_model_analysis.types_compat import Callable, Optional
 from tensorflow.python.estimator import gc
 from tensorflow.python.framework import errors_impl
 from tensorflow.python.platform import gfile
@@ -35,8 +35,8 @@
 class _EvalSavedModelExporter(tf.estimator.Exporter):
   """This class exports the EvalSavedModel.
 
-     This class provides a basic exporting functionality and serves as a
-     foundation for specialized `Exporter`s.
+  This class provides a basic exporting functionality and serves as a
+  foundation for specialized `Exporter`s.
   """
 
   def __init__(self, name,