Update README.md: add usage in detail

README.md

@@ -31,9 +31,6 @@ Both are exposed to users by specifying WHERE CLAUSE, e.g., where column > x and
 ## 
 Creatable DataSource  The libary support both read/write from/to HBase.
 
-##Application API/Usage
-Following the the examples how to write and query a HBase table. Please refer to https://github.com/hortonworks/shc/blob/master/src/test/scala/org/apache/spark/sql/DefaultSourceSuite.scala for details.
-
 ###Compile
 
     mvn package -DskipTests
@@ -49,8 +46,10 @@ Run indiviudal test
 
 The following also illustrate how to run the example in real hbase cluster. You need to provide the hbase-site.xml and related hbase jars. It may subject to change based on your specific cluster configuration.
 
-        ./bin/spark-submit  --class org.apache.spark.sql.execution.datasources.hbase.examples.HBaseSource --master yarn-client     --num-executors 2     --driver-memory 512m     --executor-memory 512m     --executor-cores 1   --jars  /usr/hdp/current/hbase-client/lib/htrace-core-3.1.0-incubating.jar,/usr/hdp/current/hbase-client/lib/hbase-client.jar,/usr/hdp/current/hbase-client/lib/hbase-common.jar,/usr/hdp/current/hbase-client/lib/hbase-server.jar,/usr/hdp/current/hbase-client/lib/guava-12.0.1.jar,/usr/hdp/current/hbase-client/lib/hbase-protocol.jar,/usr/hdp/current/hbase-client/lib/htrace-core-3.1.0-incubating.jar  --files conf/hbase-site.xml /usr/hdp/current/spark-client/lib/hbase-spark-connector-1.0.0.jar
+    ./bin/spark-submit  --class org.apache.spark.sql.execution.datasources.hbase.examples.HBaseSource --master yarn-client     --num-executors 2     --driver-memory 512m     --executor-memory 512m     --executor-cores 1   --jars  /usr/hdp/current/hbase-client/lib/htrace-core-3.1.0-incubating.jar,/usr/hdp/current/hbase-client/lib/hbase-client.jar,/usr/hdp/current/hbase-client/lib/hbase-common.jar,/usr/hdp/current/hbase-client/lib/hbase-server.jar,/usr/hdp/current/hbase-client/lib/guava-12.0.1.jar,/usr/hdp/current/hbase-client/lib/hbase-protocol.jar,/usr/hdp/current/hbase-client/lib/htrace-core-3.1.0-incubating.jar  --files conf/hbase-site.xml /usr/hdp/current/spark-client/lib/hbase-spark-connector-1.0.0.jar
 
+##Application Usage
+The following illustrates the basic procedure on how to use the connector. For more details and advanced use case, such as Avro and composite key support, please refer to the examples in the repository.
 
 ### Defined the HBase catalog
 
@@ -72,7 +71,7 @@ The following also illustrate how to run the example in real hbase cluster. You
          
 The above defines a schema for a HBase table with name as table1, row key as key and a number of columns (col1-col8). Note that the rowkey also has to be defined in details as a column (col0), which has a specific cf (rowkey).
 
-### Write to HBase table to populate data.
+### Write to HBase table to populate data
 
     sc.parallelize(data).toDF.write.options(
       Map(HBaseTableCatalog.tableCatalog -> catalog, HBaseTableCatalog.newTable -> "5"))
@@ -91,7 +90,7 @@ Given a data frame with specified schema, above will create an HBase table with
       .load()
     }
 
-#### Complicated query
+### Complicated query
 
     val df = withCatalog(catalog)
     val s = df.filter((($"col0" <= "row050" && $"col0" > "row040") ||
@@ -104,14 +103,43 @@ Given a data frame with specified schema, above will create an HBase table with
       .select("col0", "col1", "col4")
     s.show
 
-#### SQL support
+### SQL support
 
     // Load the dataframe
     val df = withCatalog(catalog)
     //SQL example
     df.registerTempTable("table")
     sqlContext.sql("select count(col1) from table").show
-
+
+## Configuring Spark-package
+Users can use the Spark-on-HBase connector as a standard Spark package. To include the package in your Spark application use:
+
+spark-shell, pyspark, or spark-submit
+
+    $SPARK_HOME/bin/spark-shell –packages zhzhan:shc:0.0.11-1.6.1-s_2.10
+
+Users can include the package as the dependency in your SBT file as well. The format is the spark-package-name:version
+
+    spDependencies += “zhzhan/shc:0.0.11-1.6.1-s_2.10”
+
+## Running in secure cluster
+
+For running in a Kerberos enabled cluster, the user has to include HBase related jars into the classpath as the HBase token
+retrieval and renewal is done by Spark, and is independent of the connector. In other words, the user needs to initiate the
+environment in the normal way, either through kinit or by providing principal/keytab.  The following examples show how to run
+in a secure cluster with both yarn-client and yarn-cluster mode. Note that SPARK_CLASSPATH has to be set for both modes, and
+the example jar is just a placeholder for Spark.
+
+    export SPARK_CLASSPATH=/usr/hdp/current/hbase-client/lib/hbase-common.jar:/usr/hdp/current/hbase-client/lib/hbase-client.jar:/usr/hdp/current/hbase-client/lib/hbase-server.jar:/usr/hdp/current/hbase-client/lib/hbase-protocol.jar:/usr/hdp/current/hbase-client/lib/guava-12.0.1.jar
+
+Suppose hrt_qa is a headless account, user can use following command for kinit:
+
+    kinit -k -t /tmp/hrt_qa.headless.keytab hrt_qa
+
+    /usr/hdp/current/spark-client/bin/spark-submit –class org.apache.spark.sql.execution.datasources.hbase.examples.HBaseSource –master yarn-client –packages zhzhan:shc:0.0.11-1.6.1-s_2.10 –num-executors 4 –driver-memory 512m –executor-memory 512m –executor-cores 1 /usr/hdp/current/spark-client/lib/spark-examples-1.6.1.2.4.2.0-106-hadoop2.7.1.2.4.2.0-106.jar
+
+    /usr/hdp/current/spark-client/bin/spark-submit –class org.apache.spark.sql.execution.datasources.hbase.examples.HBaseSource –master yarn-cluster –files /etc/hbase/conf/hbase-site.xml –packages zhzhan:shc:0.0.11-1.6.1-s_2.10 –num-executors 4 –driver-memory 512m –executor-memory 512m –executor-cores 1 /usr/hdp/current/spark-client/lib/spark-examples-1.6.1.2.4.2.0-106-hadoop2.7.1.2.4.2.0-106.jar
+
 #### TODO:
 
     val complex = s"""MAP<int, struct<varchar:string>>"""