rgw/pubsub: notification filtering by object tags

Signed-off-by: Yuval Lifshitz <[email protected]>

doc/radosgw/notifications.rst

@@ -18,7 +18,7 @@ user can only manage its own topics, and can only associate them with buckets it
 In order to send notifications for events for a specific bucket, a notification entity needs to be created. A
 notification can be created on a subset of event types, or for all event types (default).
 The notification may also filter out events based on prefix/suffix and/or regular expression matching of the keys. As well as,
-on the metadata attributes attached to the object.
+on the metadata attributes attached to the object, or the object tags.
 There can be multiple notifications for any specific topic, and the same topic could be used for multiple notifications.
 
 REST API has been defined to provide configuration and control interfaces for the bucket notification
@@ -273,7 +273,8 @@ pushed or pulled using the pubsub sync module.
                    "eTag":"",
                    "versionId":"",
                    "sequencer": "",
-                   "metadata":[]
+                   "metadata":[],
+                   "tags":[]
                }
            },
            "eventId":"",
@@ -298,6 +299,7 @@ pushed or pulled using the pubsub sync module.
 - s3.object.version: object version in case of versioned bucket
 - s3.object.sequencer: monotonically increasing identifier of the change per object (hexadecimal format)
 - s3.object.metadata: any metadata set on the object sent as: ``x-amz-meta-`` (an extension to the S3 notification API) 
+- s3.object.tags: any tags set on the objcet (an extension to the S3 notification API)
 - s3.eventId: unique ID of the event, that could be used for acking (an extension to the S3 notification API)
 
 .. _PubSub Module : ../pubsub-module

doc/radosgw/pubsub-module.rst

@@ -260,6 +260,7 @@ Detailed under: `Bucket Operations`_.
       the associated subscription will not be deleted automatically (any events of the deleted bucket could still be access),
       and will have to be deleted explicitly with the subscription deletion API
     - Filtering based on metadata (which is an extension to S3) is not supported, and such rules will be ignored
+    - Filtering based on tags (which is an extension to S3) is not supported, and such rules will be ignored
 
 
 Non S3-Compliant Notifications
@@ -474,7 +475,8 @@ the events will have an S3-compatible record format (JSON):
                    "eTag":"",
                    "versionId":"",
                    "sequencer":"",
-                   "metadata":[]
+                   "metadata":[],
+                   "tags":[]
                }
            },
            "eventId":"",
@@ -498,6 +500,7 @@ the events will have an S3-compatible record format (JSON):
 - s3.object.version: object version in case of versioned bucket
 - s3.object.sequencer: monotonically increasing identifier of the change per object (hexadecimal format)
 - s3.object.metadata: not supported (an extension to the S3 notification API)
+- s3.object.tags: not supported (an extension to the S3 notification API)
 - s3.eventId: unique ID of the event, that could be used for acking (an extension to the S3 notification API)
 
 In case that the subscription was not created via a non S3-compatible notification, 

doc/radosgw/s3-notification-compatibility.rst

@@ -49,6 +49,8 @@ Ceph's bucket notification API has the following extensions:
 
   - Filtering based on metadata attributes attached to the object
 
+  - Filtering based on object tags
+
 - Filtering overlapping is allowed, so that same event could be sent as different notification
 
 

doc/radosgw/s3/bucketops.rst

@@ -514,7 +514,13 @@ Parameters are XML encoded in the body of the request, in the following format:
                         <Name></Name>
                         <Value></Value>
                     </FilterRule>
-                </s3Metadata>
+                </S3Metadata>
+                <S3Tags>
+                    <FilterRule>
+                        <Name></Name>
+                        <Value></Value>
+                    </FilterRule>
+                </S3Tags>
             </Filter>
        </TopicConfiguration>
    </NotificationConfiguration>
@@ -533,23 +539,30 @@ Parameters are XML encoded in the body of the request, in the following format:
 | ``Event``                     | String    | List of supported events see: `S3 Notification Compatibility`_.  Multiple ``Event``  | No       |
 |                               |           | entities can be used. If omitted, all events are handled                             |          |
 +-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
-| ``Filter``                    | Container | Holding ``S3Key`` and ``S3Metadata`` entities                                        | No       |
+| ``Filter``                    | Container | Holding ``S3Key``, ``S3Metadata`` and ``S3Tags`` entities                            | No       |
 +-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
 | ``S3Key``                     | Container | Holding a list of ``FilterRule`` entities, for filtering based on object key.        | No       |
 |                               |           | At most, 3 entities may be in the list, with ``Name`` be ``prefix``, ``suffix`` or   |          |
 |                               |           | ``regex``. All filter rules in the list must match for the filter to match.          |          |
 +-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
 | ``S3Metadata``                | Container | Holding a list of ``FilterRule`` entities, for filtering based on object metadata.   | No       |
-|                               |           | All filter rules in the list must match the ones defined on the object. The object,  |          |
-|                               |           | have other metadata entitied not listed in the filter.                               |          |
+|                               |           | All filter rules in the list must match the metadata defined on the object. However, |          |
+|                               |           | the object still match if it has other metadata entries not listed in the filter.    |          |
++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
+| ``S3Tags``                    | Container | Holding a list of ``FilterRule`` entities, for filtering based on object tags.       | No       |
+|                               |           | All filter rules in the list must match the tags defined on the object. However,     |          |
+|                               |           | the object still match it it has other tags not listed in the filter.                |          |
 +-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
 | ``S3Key.FilterRule``          | Container | Holding ``Name`` and ``Value`` entities. ``Name`` would  be: ``prefix``, ``suffix``  | Yes      |
 |                               |           | or ``regex``. The ``Value`` would hold the key prefix, key suffix or a regular       |          |
 |                               |           | expression for matching the key, accordingly.                                        |          |
 +-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
 | ``S3Metadata.FilterRule``     | Container | Holding ``Name`` and ``Value`` entities. ``Name`` would be the name of the metadata  | Yes      |
 |                               |           | attribute (e.g. ``x-amz-meta-xxx``). The ``Value`` would be the expected value for   |          | 
-|                               |           | this attribute                                                                       |          |
+|                               |           | this attribute.                                                                      |          |
++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
+| ``S3Tags.FilterRule``         | Container | Holding ``Name`` and ``Value`` entities. ``Name`` would be the tag key,              |  Yes     |
+|                               |           | and ``Value`` would be the tag value.                                                |          | 
 +-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
 
 
@@ -652,7 +665,13 @@ Response is XML encoded in the body of the request, in the following format:
                         <Name></Name>
                         <Value></Value>
                     </FilterRule>
-                </s3Metadata>
+                </S3Metadata>
+                <S3Tags>
+                    <FilterRule>
+                        <Name></Name>
+                        <Value></Value>
+                    </FilterRule>
+                </S3Tags>
             </Filter>
        </TopicConfiguration>
    </NotificationConfiguration>
@@ -684,4 +703,4 @@ HTTP Response
 | ``404``       | NoSuchKey             | The notification does not exist (if provided)            |
 +---------------+-----------------------+----------------------------------------------------------+
 
-.. _S3 Notification Compatibility: ../s3-notification-compatibility
+.. _S3 Notification Compatibility: ../../s3-notification-compatibility

examples/boto3/README.md

@@ -1,5 +1,6 @@
 # Introduction
 This directory contains examples on how to use AWS CLI/boto3 to exercise the RadosGW extensions to the S3 API.
+This is an extension to the [AWS SDK](https://github.com/boto/botocore/blob/develop/botocore/data/s3/2006-03-01/service-2.json).
 
 # Users
 For the standard client to support these extensions, the: ``service-2.sdk-extras.json`` file should be placed under: ``~/.aws/models/s3/2006-03-01/`` directory.

examples/boto3/notification_filters.py

@@ -35,6 +35,10 @@
 			    'FilterRules': [{'Name': 'x-amz-meta-foo', 'Value': 'bar'},
                                             {'Name': 'x-amz-meta-hello', 'Value': 'world'}]
                          },
+                        'Tags': {
+			    'FilterRules': [{'Name': 'foo', 'Value': 'bar'},
+                                            {'Name': 'hello', 'Value': 'world'}]
+                         },
                          'Key': {
                              'FilterRules': [{'Name': 'regex', 'Value': '([a-z]+)'}]
                          }

examples/boto3/service-2.sdk-extras.json

@@ -96,7 +96,13 @@
                     "shape":"S3MetadataFilter",
                     "documentation":"<p/>",
                     "locationName":"S3Metadata"
+                },
+                "Tags":{
+                    "shape":"S3TagsFilter",
+                    "documentation":"<p/>",
+                    "locationName":"S3Tags"
                 }
+
             },
             "documentation":"<p>Specifies object key name filtering rules. For information about key name filtering, see <a href=\"https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html\">Configuring Event Notifications</a> in the <i>Amazon Simple Storage Service Developer Guide</i>.</p>"
         },
@@ -121,6 +127,17 @@
                 }
             },
             "documentation":"<p>A container for metadata filtering rules.</p>"
+        },
+        "S3TagsFilter":{
+            "type":"structure",
+            "members":{
+                "FilterRules":{
+                    "shape":"FilterRuleList",
+                    "documentation":"<p/>",
+                    "locationName":"FilterRule"
+                }
+            },
+            "documentation":"<p>A container for object tags filtering rules.</p>"
         }
     },
     "documentation":"<p/>"

src/common/ceph_json.h

@@ -3,7 +3,7 @@
 
 #include <stdexcept>
 #include <include/types.h>
-
+#include <boost/container/flat_map.hpp>
 
 #include "json_spirit/json_spirit.h"
 
@@ -259,6 +259,22 @@ void decode_json_obj(std::multimap<K, V>& m, JSONObj *obj)
   }
 }
 
+template<class K, class V>
+void decode_json_obj(boost::container::flat_map<K, V>& m, JSONObj *obj)
+{
+  m.clear();
+
+  JSONObjIter iter = obj->find_first();
+
+  for (; !iter.end(); ++iter) {
+    K key;
+    V val;
+    JSONObj *o = *iter;
+    JSONDecoder::decode_json("key", key, o);
+    JSONDecoder::decode_json("val", val, o);
+    m[key] = val;
+  }
+}
 template<class C>
 void decode_json_obj(C& container, void (*cb)(C&, JSONObj *obj), JSONObj *obj)
 {
@@ -397,6 +413,7 @@ static void encode_json(const char *name, const std::list<T>& l, ceph::Formatter
   }
   f->close_section();
 }
+
 template<class T>
 static void encode_json(const char *name, const std::deque<T>& l, ceph::Formatter *f)
 {
@@ -406,6 +423,7 @@ static void encode_json(const char *name, const std::deque<T>& l, ceph::Formatte
   }
   f->close_section();
 }
+
 template<class T, class Compare = std::less<T> >
 static void encode_json(const char *name, const std::set<T, Compare>& l, ceph::Formatter *f)
 {
@@ -426,7 +444,7 @@ static void encode_json(const char *name, const std::vector<T>& l, ceph::Formatt
   f->close_section();
 }
 
-template<class K, class V, class C = std::less<K> >
+template<class K, class V, class C = std::less<K>>
 static void encode_json(const char *name, const std::map<K, V, C>& m, ceph::Formatter *f)
 {
   f->open_array_section(name);
@@ -451,6 +469,20 @@ static void encode_json(const char *name, const std::multimap<K, V>& m, ceph::Fo
   }
   f->close_section();
 }
+
+template<class K, class V>
+static void encode_json(const char *name, const boost::container::flat_map<K, V>& m, ceph::Formatter *f)
+{
+  f->open_array_section(name);
+  for (auto i = m.begin(); i != m.end(); ++i) {
+    f->open_object_section("entry");
+    encode_json("key", i->first, f);
+    encode_json("val", i->second, f);
+    f->close_section();
+  }
+  f->close_section();
+}
+
 template<class K, class V>
 void encode_json_map(const char *name, const std::map<K, V>& m, ceph::Formatter *f)
 {

src/rgw/rgw_auth_s3.cc

@@ -56,7 +56,7 @@ static const auto signed_subresources = {
  */
 
 static std::string
-get_canon_amz_hdr(const std::map<std::string, std::string>& meta_map)
+get_canon_amz_hdr(const meta_map_t& meta_map)
 {
   std::string dest;
 
@@ -117,8 +117,8 @@ void rgw_create_s3_canonical_header(
   const char* const content_md5,
   const char* const content_type,
   const char* const date,
-  const std::map<std::string, std::string>& meta_map,
-  const std::map<std::string, std::string>& qs_map,
+  const meta_map_t& meta_map,
+  const meta_map_t& qs_map,
   const char* const request_uri,
   const std::map<std::string, std::string>& sub_resources,
   std::string& dest_str)
@@ -157,7 +157,7 @@ static inline bool is_base64_for_content_md5(unsigned char c) {
 }
 
 static inline void get_v2_qs_map(const req_info& info,
-				 std::map<std::string, std::string>& qs_map) {
+				 meta_map_t& qs_map) {
   const auto& params = const_cast<RGWHTTPArgs&>(info.args).get_params();
   for (const auto& elt : params) {
     std::string k = boost::algorithm::to_lower_copy(elt.first);
@@ -190,7 +190,7 @@ bool rgw_create_s3_canonical_header(const req_info& info,
   const char *content_type = info.env->get("CONTENT_TYPE");
 
   std::string date;
-  std::map<std::string, std::string> qs_map;
+  meta_map_t qs_map;
 
   if (qsr) {
     get_v2_qs_map(info, qs_map); // handle qs metadata

src/rgw/rgw_auth_s3.h

@@ -421,8 +421,8 @@ void rgw_create_s3_canonical_header(
   const char *content_md5,
   const char *content_type,
   const char *date,
-  const std::map<std::string, std::string>& meta_map,
-  const std::map<std::string, std::string>& qs_map,
+  const meta_map_t& meta_map,
+  const meta_map_t& qs_map,
   const char *request_uri,
   const std::map<std::string, std::string>& sub_resources,
   std::string& dest_str);

src/rgw/rgw_common.cc

@@ -435,7 +435,7 @@ std::ostream& operator<<(std::ostream& oss, const rgw_err &err)
 }
 
 void rgw_add_amz_meta_header(
-  std::map<std::string, std::string>& x_meta_map,
+  meta_map_t& x_meta_map,
   const std::string& k,
   const std::string& v)
 {

src/rgw/rgw_common.h

@@ -1648,11 +1648,12 @@ namespace rgw {
   }
 }
 
+using meta_map_t = boost::container::flat_map <std::string, std::string>;
 
 struct req_info {
   const RGWEnv *env;
   RGWHTTPArgs args;
-  map<string, string> x_meta_map;
+  meta_map_t x_meta_map;
 
   string host;
   const char *method;
@@ -2456,7 +2457,7 @@ static inline uint64_t rgw_rounded_objsize_kb(uint64_t bytes)
 /* implement combining step, S3 header canonicalization;  k is a
  * valid header and in lc form */
 void rgw_add_amz_meta_header(
-  std::map<std::string, std::string>& x_meta_map,
+  meta_map_t& x_meta_map,
   const std::string& k,
   const std::string& v);
 

src/rgw/rgw_loadgen.cc

@@ -18,7 +18,7 @@ void RGWLoadGenRequestEnv::set_date(utime_t& tm)
 
 int RGWLoadGenRequestEnv::sign(RGWAccessKey& access_key)
 {
-  map<string, string> meta_map;
+  meta_map_t meta_map;
   map<string, string> sub_resources;
 
   string canonical_header;
@@ -29,7 +29,7 @@ int RGWLoadGenRequestEnv::sign(RGWAccessKey& access_key)
                                  content_type.c_str(),
                                  date_str.c_str(),
                                  meta_map,
-				 map<string, string>{},
+				                 meta_map_t{},
                                  uri.c_str(),
                                  sub_resources,
                                  canonical_header);

src/rgw/rgw_notify.cc

@@ -40,6 +40,8 @@ void populate_record_from_request(const req_state *s,
   record.bucket_id = s->bucket.bucket_id;
   // pass meta data
   record.x_meta_map = s->info.x_meta_map;
+  // pass tags
+  record.tags = s->tagset.get_tags();
 }
 
 bool match(const rgw_pubsub_topic_filter& filter, const req_state* s, EventType event) {
@@ -52,6 +54,9 @@ bool match(const rgw_pubsub_topic_filter& filter, const req_state* s, EventType
   if (!::match(filter.s3_filter.metadata_filter, s->info.x_meta_map)) {
     return false;
   }
+  if (!::match(filter.s3_filter.tag_filter, s->tagset.get_tags())) {
+    return false;
+  }
   return true;
 }