Merge branch 'wip-sts-authorization'

Merge pull request ceph#14307 from adamemerson/wip-sts-authorization

common: Add make_unique
submodule: Tencent's RapidJSON library
common: Add ISO-8601 Date Support
rgw: Add is_identity to AuthApplier class
rgw: Move globbing flags to header
buffer: Make the use of static areas more convenient
rgw: Add basic support for IAM policies
rgw: Build bucket permission and environment in req_state
rgw: Verify policies as WELL as ACLs
rgw: RESTful bucket policy ops
rgw: Write documentation for bucket policies

Reviewed-By: Casey Bodley <[email protected]>
Reviewed-By: Radoslaw Zarzynski <[email protected]>
Reviewed-By: Kefu Chai <[email protected]>

.gitmodules

@@ -55,3 +55,6 @@
 [submodule "src/blkin"]
 	path = src/blkin
 	url = https://github.com/ceph/blkin
+[submodule "src/rapidjson"]
+	path = src/rapidjson
+	url = https://github.com/ceph/rapidjson

ceph.spec.in

@@ -97,6 +97,7 @@ BuildRequires:	selinux-policy-devel
 BuildRequires:	/usr/share/selinux/devel/policyhelp
 %endif
 BuildRequires:	bc
+BuildRequires:	gperf
 BuildRequires:  cmake
 BuildRequires:	cryptsetup
 BuildRequires:	fuse-devel

debian/control

@@ -9,6 +9,7 @@ Uploaders: Ken Dreyer <[email protected]>,
            Alfredo Deza <[email protected]>
 Build-Depends: bc,
                btrfs-tools,
+	       gperf,
 	       cmake,
                cpio,
 	       cryptsetup-bin | cryptsetup,

doc/radosgw/bucketpolicy.rst

@@ -0,0 +1,133 @@
+===============
+Bucket Policies
+===============
+
+.. versionadded:: Luminous
+
+The Ceph Object Gateway supports a subset of the Amazon S3 policy
+language applied to buckets.
+
+
+Creation and Removal
+====================
+
+Bucket policies are managed through standard S3 operations rather than
+radosgw-admin.
+
+For example, one may use s3cmd to set or delete a policy thus::
+
+  $ cat > examplepol
+  {
+    "Version": "2012-10-17",
+    "Statement": [{
+      "Effect": "Allow",
+      "Principal": {"AWS": ["arn:aws:iam::usfolks:user/fred"]},
+      "Action": "s3PutObjectAcl",
+      "Resource": [
+        "arn:aws:s3:::happybucket/*"
+      ]
+    }]
+  }
+
+  $ s3cmd setpolicy examplepol s3://happybucket
+  $ s3cmd delpolicy s3://happybucket
+
+
+Limitations
+===========
+
+Currently, we support only the following actions:
+
+- s3:AbortMultipartUpload
+- s3:CreateBucket
+- s3:DeleteBucketPolicy
+- s3:DeleteBucket
+- s3:DeleteBucketWebsite
+- s3:DeleteObject
+- s3:DeleteObjectVersion
+- s3:DeleteReplicationConfiguration
+- s3:GetAccelerateConfiguration
+- s3:GetBucketAcl
+- s3:GetBucketCORS
+- s3:GetBucketLocation
+- s3:GetBucketLogging
+- s3:GetBucketNotification
+- s3:GetBucketPolicy
+- s3:GetBucketRequestPayment
+- s3:GetBucketTagging
+- s3:GetBucketVersioning
+- s3:GetBucketWebsite
+- s3:GetLifecycleConfiguration
+- s3:GetObjectAcl
+- s3:GetObject
+- s3:GetObjectTorrent
+- s3:GetObjectVersionAcl
+- s3:GetObjectVersion
+- s3:GetObjectVersionTorrent
+- s3:GetReplicationConfiguration
+- s3:ListAllMyBuckets
+- s3:ListBucketMultiPartUploads
+- s3:ListBucket
+- s3:ListBucketVersions
+- s3:ListMultipartUploadParts
+- s3:PutAccelerateConfiguration
+- s3:PutBucketAcl
+- s3:PutBucketCORS
+- s3:PutBucketLogging
+- s3:PutBucketNotification
+- s3:PutBucketPolicy
+- s3:PutBucketRequestPayment
+- s3:PutBucketTagging
+- s3:PutBucketVersioning
+- s3:PutBucketWebsite
+- s3:PutLifecycleConfiguration
+- s3:PutObjectAcl
+- s3:PutObject
+- s3:PutObjectVersionAcl
+- s3:PutReplicationConfiguration
+- s3:RestoreObject
+
+We do not yet support setting policies on users, groups, or roles.
+
+We use the RGW ‘tenant’ identifier in place of the Amazon twelve-digit
+account ID. In the future we may allow you to assign an account ID to
+a tenant, but for now if you want to use policies between AWS S3 and
+RGW S3 you will have to use the Amazon account ID as the tenant ID when
+creating users.
+
+Under AWS, all tenants share a single namespace. RGW gives every
+tenant its own namespace of buckets. There may be an option to enable
+an AWS-like 'flat' bucket namespace in future versions. At present, to
+access a bucket belonging to another tenant, address it as
+"tenant:bucket" in the S3 request.
+
+In AWS, a bucket policy can grant access to another account, and that
+account owner can then grant access to individual users with user
+permissions. Since we do not yet support user, role, and group
+permissions, account owners will currently need to grant access
+directly to individual users, and granting an entire account access to
+a bucket grants access to all users in that account.
+
+Bucket policies do not yet support string interpolation.
+
+Currently, the only condition keys we support are:
+- aws:CurrentTime
+- aws:EpochTime
+- aws:PrincipalType
+- aws:Referer
+- aws:SecureTransport
+- aws:SourceIp
+- aws:UserAgent
+- aws:username
+
+More may be supported soon as we integrate with the recently rewritten
+Authentication/Authorization subsystem.
+
+Swift
+=====
+
+There is no way to set bucket policies under Swift, but bucket
+policies that have been set govern Swift as well as S3 operations.
+
+Swift credentials are matched against Principals specified in a policy
+in a way specific to whatever backend is being used.

doc/radosgw/index.rst

@@ -51,6 +51,7 @@ you may write data with one API and retrieve it with the other.
    Multi-tenancy <multitenancy>
    Compression <compression>
    Server-Side Encryption <encryption>
+   Bucket Policy <bucketpolicy>
    Data Layout in RADOS <layout>
    Upgrade to Older Versions of Jewel <upgrade_to_jewel>
    troubleshooting

install-deps.sh

@@ -22,6 +22,7 @@ export LC_ALL=C # the following is vulnerable to i18n
 if [ x`uname`x = xFreeBSDx ]; then
     $SUDO pkg install -yq \
         devel/git \
+        devel/gperf \
         devel/gmake \
         devel/cmake \
         devel/yasm \

src/CMakeLists.txt

@@ -335,6 +335,7 @@ add_library(crush_objs OBJECT ${crush_srcs})
 add_subdirectory(json_spirit)
 
 include_directories("${CMAKE_SOURCE_DIR}/src/xxHash")
+include_directories(SYSTEM "${CMAKE_SOURCE_DIR}/src/rapidjson/include")
 
 set(GMOCK_INCLUDE_DIRS
   "${CMAKE_SOURCE_DIR}/src/googletest/googletest/include/gmock")
@@ -439,6 +440,7 @@ set(libcommon_files
   common/TrackedOp.cc
   common/SloppyCRCMap.cc
   common/types.cc
+  common/iso_8601.cc
   log/Log.cc
   log/SubsystemMap.cc
   mon/MonCap.cc

src/common/backport14.h

@@ -0,0 +1,64 @@
+// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- 
+// vim: ts=8 sw=2 smarttab
+/*
+ * Ceph - scalable distributed file system
+ *
+ * Copyright (C) 2004-2006 Sage Weil <[email protected]>
+ *
+ * This is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2.1, as published by the Free Software
+ * Foundation.  See file COPYING.
+ *
+ */
+
+#include <memory>
+#include <type_traits>
+
+#ifndef CEPH_COMMON_BACKPORT14_H
+#define CEPH_COMMON_BACKPORT14_H
+
+// Library code from C++14 that can be implemented in C++11.
+
+namespace ceph {
+template<typename T>
+using remove_extent_t = typename std::remove_extent<T>::type;
+template<typename T>
+using remove_reference_t = typename std::remove_reference<T>::type;
+template<typename T>
+using result_of_t = typename std::result_of<T>::type;
+
+namespace _backport14 {
+template<typename T>
+struct uniquity {
+  using datum = std::unique_ptr<T>;
+};
+
+template<typename T>
+struct uniquity<T[]> {
+  using array = std::unique_ptr<T[]>;
+};
+
+template<typename T, std::size_t N>
+struct uniquity<T[N]> {
+  using verboten = void;
+};
+
+template<typename T, typename... Args>
+inline typename uniquity<T>::datum make_unique(Args&&... args) {
+  return std::unique_ptr<T>(new T(std::forward<Args>(args)...));
+}
+
+template<typename T>
+inline typename uniquity<T>::array make_unique(std::size_t n) {
+  return std::unique_ptr<T>(new remove_extent_t<T>[n]());
+}
+
+template<typename T, class... Args>
+typename uniquity<T>::verboten
+make_unique(Args&&...) = delete;
+} // namespace _backport14
+using _backport14::make_unique;
+} // namespace ceph
+
+#endif // CEPH_COMMON_BACKPORT14_H

src/common/buffer.cc

@@ -2498,6 +2498,25 @@ void buffer::list::hexdump(std::ostream &out, bool trailing_newline) const
   out.flags(original_flags);
 }
 
+
+buffer::list buffer::list::static_from_mem(char* c, size_t l) {
+  list bl;
+  bl.push_back(ptr(create_static(l, c)));
+  return bl;
+}
+
+buffer::list buffer::list::static_from_cstring(char* c) {
+  return static_from_mem(c, std::strlen(c));
+}
+
+buffer::list buffer::list::static_from_string(string& s) {
+  // C++14 just has string::data return a char* from a non-const
+  // string.
+  return static_from_mem(const_cast<char*>(s.data()), s.length());
+  // But the way buffer::list mostly doesn't work in a sane way with
+  // const makes me generally sad.
+}
+
 std::ostream& buffer::operator<<(std::ostream& out, const buffer::raw &r) {
   return out << "buffer::raw(" << (void*)r.data << " len " << r.len << " nref " << r.nref.read() << ")";
 }