Clickhouse enhancements (trickstercache#456)

* handles more ClickHouse timeseries query patterns.
* simplifies ClickHouse processing to a single series
* Fix ClickHouse multi group by queries, don't cache bucket result if bucket contains "now"
* Handle CH now()-[x] queries, refactor BackfillTolerance handling slightly.
* Fix CH BETWEEN phrase parsing, fix backfill tolerance for "old" queries, allow timerange setting within subqueries, don't force http compression.

docs/clickhouse.md

@@ -1,22 +1,59 @@
 # ClickHouse Support
 
-Trickster 1.0 provides support for accelerating ClickHouse queries that return time series data normally visualized on a dashboard. Acceleration works by using the Time Series Delta Proxy Cache to minimize the number and time range of queries to the upstream ClickHouse server.
+Trickster 1.1 provides expanded support for accelerating ClickHouse queries that return time series data normally visualized on a dashboard. Acceleration works by using the Time Series Delta Proxy Cache to minimize the number and time range of queries to the upstream ClickHouse server.
 
 ## Scope of Support
 
-Trickster is tested with the [ClickHouse DataSource Plugin for Grafana](https://grafana.com/grafana/plugins/vertamedia-clickhouse-datasource) v1.9.3 by Vertamedia, and supports acceleration of queries constructed by this plugin using the plugin's built-in `$timeSeries` macro.
+Trickster is tested with the [ClickHouse DataSource Plugin for Grafana](https://grafana.com/grafana/plugins/vertamedia-clickhouse-datasource) v1.9.3 by Vertamedia, and supports acceleration of queries constructed by this plugin using the plugin's built-in `$timeSeries` macro.  Trickster also supports several other query formats that return "time series like" data.
 
-Because ClickHouse does not provide a golang-based query parser, Trickster uses pre-compiled Regular Expression pattern matches on the incoming ClickHouse query to deconstruct its components, determine if it is cacheable and, if so, what elements are factored into the cache key derivation. We also determine what parts of the query are template-able (e.g., `time BETWEEN $time1 AND $time2`) based on the provided absolute values, in order to normalize the query before hashing the cache key.
+Because ClickHouse does not provide a golang-based query parser, Trickster uses custom parsing code on the incoming ClickHouse query to deconstruct its components, determine if it is cacheable and, if so, what elements are factored into the cache key derivation. Trickster also determines the requested time range and step based on the provided absolute values, in order to normalize the query before hashing the cache key.
 
-If you find query or response structures that are not yet supported, or providing inconsistent or unexpected results, we'd love for you to report those. We also always welcome any contributions around this functionality. The regular expression patterns we currently use will likely grow in complexity as support for more query patterns is added. Thus, we may need to find a more robust query parsing solution, and welcome any assistance with that as well.
+If you find query or response structures that are not yet supported, or providing inconsistent or unexpected results, we'd love for you to report those. We also always welcome any contributions around this functionality.
 
-Trickster currently supports the following query patterns (case-insensitive) in the JSON response format, which align with the output of the ClickHouse Data Source Plugin for Grafana:
+To constitute a cacheable query, the first column expression in the main or any subquery must be in one of two specific forms in order to determine the timestamp column and step:
 
+#### Grafana Plugin Format 
 ```sql
-SELECT (intDiv(toUInt32(time_col), 60) * 60) * 1000 AS t, countMerge(val_col) AS cnt, field1, field2
-FROM exampledb.example_table WHERE time_col BETWEEN toDateTime(1574686300) AND toDateTime(1574689900)
-    AND field1 > 0 AND field2 = 'some_value' GROUP BY t, field1, field2 ORDER BY t, field1, field2
-FORMAT JSON
+SELECT intDiv(toUInt32(time_col, 60) * 60) [* 1000] [as] [alias]
 ```
+This is the approach used by the Grafana plugin.  The time_col and/or alias is used to determine the requested time range from the WHERE or PREWHERE clause of the query.  The argument to the ClickHouse intDiv function is the step value in seconds, since the toUInt32 function on a datetime column returns the Unix epoch seconds.
 
-In this format, the first column must be the datapoint's timestamp, the second column must be the datapoint's value, and all additional fields define the datapoint's metric name. The time column must be in the format of `(intDiv(toUInt32($time_col), $period) * $period) * 1000`, and the value column must be numeric (integer or floating point). The where clause must include `time_col > toDateTime($epoch)` or `time_col BETWEEN toDateTime($epoch1) AND toDateTime($epoch2)`. Subqueries and other modifications are compatible so long as the key components of the time series, mentioned here, can be extracted.
+#### ClickHouse Time Grouping Function
+```sql
+SELECT toStartOf[Period](time_col) [as] [alias]
+```
+This is the approach that uses the following optimized ClickHouse functions to group timeseries queries:
+```
+toStartOfMinute
+toStartOfFiveMinute
+toStartOfTenMinutes
+toStartOfFifteenMinutes
+toStartOfHour
+toDate
+```
+Again the time_col and/or alias is used to determine the request time range from the WHERE or PREWHERE clause, and the step is derived from the function name.
+
+#### Determining the requested time range
+
+Once the time column (or alias) and step are derived, Trickster parses each WHERE or PREWHERE clause to find comparison operations 
+that mark the requested time range.  To be cacheable, the WHERE clause must contain either a `[timecol|alias] BETWEEN` phrase or 
+a `[time_col|alias] >[=]` phrase.  The BETWEEN or >= arguments must be a parseable ClickHouse string date in the form `2006-01-02 15:04:05`, a
+a ten digit integer representing epoch seconds, or the `now()` ClickHouse function with optional subtraction.
+
+If a `>` phrase is used, a similar `<` phrase can be used to specify the end of the time period.  If none is found, Trickster will still cache results up to
+the current time, but future queries must also have no end time phrase, or Trickster will be unable to find the correct cache key.
+
+Examples of cacheable time range WHERE clauses:
+```sql
+WHERE t >= "2020-10-15 00:00:00" and t <= "2020-10-16 12:00:00"
+WHERE t >= "2020-10-15 12:00:00" and t < now() - 60 * 60
+WHERE datetime BETWEEN 1574686300 AND 1574689900
+```
+
+Note that these values can be wrapped in the ClickHouse toDateTime function, but ClickHouse will make that conversion implicitly and it is not required.   All string times are assumed to be UTC.
+
+### Normalization and "Fast Forwarding"
+
+Trickster will always normalize the calculated time range to fit the step size, so small variations in the time range will still result in actual queries for
+the entire time "bucket".  In addition, Trickster will not cache the results for the portion of the query that is still active -- i.e., within the current bucket
+or within the configured backfill tolerance setting (whichever is greater) 

pkg/proxy/origins/clickhouse/clickhouse.go

@@ -18,18 +18,18 @@
 package clickhouse
 
 import (
+	"github.com/tricksterproxy/trickster/pkg/proxy/request"
 	"net/http"
 	"net/url"
+	"time"
 
 	"github.com/tricksterproxy/trickster/pkg/cache"
 	"github.com/tricksterproxy/trickster/pkg/proxy"
 	"github.com/tricksterproxy/trickster/pkg/proxy/errors"
 	"github.com/tricksterproxy/trickster/pkg/proxy/origins"
 	oo "github.com/tricksterproxy/trickster/pkg/proxy/origins/options"
-	tt "github.com/tricksterproxy/trickster/pkg/proxy/timeconv"
 	"github.com/tricksterproxy/trickster/pkg/proxy/urls"
 	"github.com/tricksterproxy/trickster/pkg/timeseries"
-	"github.com/tricksterproxy/trickster/pkg/util/regexp/matching"
 )
 
 var _ origins.Client = (*Client)(nil)
@@ -92,39 +92,34 @@ func (c *Client) Router() http.Handler {
 
 // ParseTimeRangeQuery parses the key parts of a TimeRangeQuery from the inbound HTTP Request
 func (c *Client) ParseTimeRangeQuery(r *http.Request) (*timeseries.TimeRangeQuery, error) {
-
-	trq := &timeseries.TimeRangeQuery{Extent: timeseries.Extent{}}
-	trq.TemplateURL = urls.Clone(r.URL)
-	qi := trq.TemplateURL.Query()
+	qi := r.URL.Query()
+	var rawQuery string
 	if p, ok := qi[upQuery]; ok {
-		trq.Statement = p[0]
+		rawQuery = p[0]
 	} else {
 		return nil, errors.MissingURLParam(upQuery)
 	}
 
-	mp := []string{"step", "timeField"}
-	found := matching.GetNamedMatches(reTimeFieldAndStep, trq.Statement, mp)
-
-	for _, f := range mp {
-		v, ok := found[f]
-		if !ok || v == "" {
-			return nil, errors.ErrNotTimeRangeQuery
-		}
-		switch f {
-		case "timeField":
-			trq.TimestampFieldName = v
-		case "step":
-			trq.Step, _ = tt.ParseDuration(v + "s")
-		}
+	var bf time.Duration
+	res := request.GetResources(r)
+	if res == nil {
+		bf = 60 * time.Second
+	} else {
+		bf = res.OriginConfig.BackfillTolerance
 	}
 
-	var err error
-	trq.Statement, trq.Extent, _, err = getQueryParts(trq.Statement, trq.TimestampFieldName)
-	if err != nil {
+	// Force gzip compression since Brotli is broken on CH 20.3
+	// See https://github.com/ClickHouse/ClickHouse/issues/9969
+	// Clients that don't understand gzip are going to break, but oh well
+	r.Header.Set("Accept-Encoding", "gzip")
+
+	trq := &timeseries.TimeRangeQuery{Extent: timeseries.Extent{}, BackfillTolerance: bf}
+	if err := parseRawQuery(rawQuery, trq); err != nil {
 		return nil, err
 	}
 
-	// Swap in the Tokenzed Query in the Url Params
+	trq.TemplateURL = urls.Clone(r.URL)
+	// Swap in the Tokenized Query in the Url Params
 	qi.Set(upQuery, trq.Statement)
 	trq.TemplateURL.RawQuery = qi.Encode()
 	return trq, nil

pkg/proxy/origins/clickhouse/clickhouse_test.go

@@ -157,17 +157,17 @@ func TestParseTimeRangeQuery(t *testing.T) {
 		Host:     "blah.com",
 		Path:     "/",
 		RawQuery: testRawQuery(),
-	}}
+	},
+		Header: http.Header{},
+	}
 	client := &Client{}
 	res, err := client.ParseTimeRangeQuery(req)
 	if err != nil {
 		t.Error(err)
 	} else {
-
 		if res.Step.Seconds() != 60 {
 			t.Errorf("expected 60 got %f", res.Step.Seconds())
 		}
-
 		if res.Extent.End.Sub(res.Extent.Start).Hours() != 6 {
 			t.Errorf("expected 6 got %f", res.Extent.End.Sub(res.Extent.Start).Hours())
 		}

pkg/proxy/origins/clickhouse/handler_proxy.go

@@ -24,7 +24,7 @@ import (
 )
 
 // ProxyHandler sends a request through the basic reverse proxy to the origin,
-// and services non-cacheable InfluxDB API calls
+// and services non-cacheable ClickHouse API calls
 func (c *Client) ProxyHandler(w http.ResponseWriter, r *http.Request) {
 	r.URL = urls.BuildUpstreamURL(r, c.baseUpstreamURL)
 	engines.DoProxy(w, r, true)

pkg/proxy/origins/clickhouse/handler_query.go

@@ -28,9 +28,7 @@ import (
 func (c *Client) QueryHandler(w http.ResponseWriter, r *http.Request) {
 
 	rqlc := strings.Replace(strings.ToLower(r.URL.RawQuery), "%20", "+", -1)
-	// if it's not a select statement, just proxy it instead
-	if (!strings.HasPrefix(rqlc, "query=select+")) && (!(strings.Index(rqlc, "&query=select+") > 0)) &&
-		(!strings.HasSuffix(rqlc, "format+json")) {
+	if (!strings.HasPrefix(rqlc, "query=")) && (!(strings.Index(rqlc, "&query=") > 0)) || r.Method != http.MethodGet {
 		c.ProxyHandler(w, r)
 		return
 	}

pkg/proxy/origins/clickhouse/handler_query_test.go

@@ -37,8 +37,7 @@ func testRawQuery() string {
 }
 
 func testNonSelectQuery() string {
-	return url.Values(map[string][]string{"query": {
-		`UPDATE (intDiv(toUInt32(time_column), 60) * 60) * 1000 AS t`}}).Encode()
+	return url.Values(map[string][]string{"enable_http_compression": {"1"}}).Encode()
 	// not a real query, just something to trigger a non-select proxy-only request
 }