Skip to content

Commit

Permalink
[MINOR] Update docs for functions.scala to make it clear not all the …
Browse files Browse the repository at this point in the history
…built-in functions are defined there

The title summarizes the change.

Author: Reynold Xin <[email protected]>

Closes apache#21318 from rxin/functions.
  • Loading branch information
rxin authored and gatorsmile committed Jul 28, 2018
1 parent 34ebcc6 commit 6424b14
Showing 1 changed file with 15 additions and 1 deletion.
16 changes: 15 additions & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/functions.scala
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,21 @@ import org.apache.spark.util.Utils


/**
* Functions available for DataFrame operations.
* Commonly used functions available for DataFrame operations. Using functions defined here provides
* a little bit more compile-time safety to make sure the function exists.
*
* Spark also includes more built-in functions that are less common and are not defined here.
* You can still access them (and all the functions defined here) using the `functions.expr()` API
* and calling them through a SQL expression string. You can find the entire list of functions for
* the latest version of Spark at https://spark.apache.org/docs/latest/api/sql/index.html.
*
* As an example, `isnan` is a function that is defined here. You can use `isnan(col("myCol"))`
* to invoke the `isnan` function. This way the programming language's compiler ensures `isnan`
* exists and is of the proper form. You can also use `expr("isnan(myCol)")` function to invoke the
* same function. In this case, Spark itself will ensure `isnan` exists when it analyzes the query.
*
* `regr_count` is an example of a function that is built-in but not defined here, because it is
* less commonly used. To invoke it, use `expr("regr_count(yCol, xCol)")`.
*
* @groupname udf_funcs UDF functions
* @groupname agg_funcs Aggregate functions
Expand Down

0 comments on commit 6424b14

Please sign in to comment.