diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..e43b0f9 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +.DS_Store diff --git a/Part 01 - What is Cloudant.md b/Part 01 - What is Cloudant.md new file mode 100644 index 0000000..c82f413 --- /dev/null +++ b/Part 01 - What is Cloudant.md @@ -0,0 +1,61 @@ +![](slides/Slide0.png) + +Welcome to the Introduction to Cloudant course, an eighteen part video series that gives you an overview of the IBM Cloudant databases-as-a-service. + +![](slides/Slide1.png) + + +--- + +This is part 1: "What is Cloudant?" + +![](slides/Slide4.png) + +--- + +Cloudant is a database, run as a service in the IBM Cloud. Its job is to store your application's data securely and allow you to retrieve it quickly and efficiently. Cloudant's key features are that it is + +- a database - it stores and retrieves data +- more specifically it is a JSON document store. JSON comes from JavaScript and represents simple objects in a universal file format. The "document" is the unit of storage in Cloudant. Documents are added/updated/deleted in their entirety. +- It has an HTTP API. Any Cloudant operation can be achieved using HTTP. HTTP is the protocol that powers the World Wide Web and Cloudant is a database built for the web. + +Most databases are hidden in a private network, inaccessible but to a handful of machines but the Cloudant service sits (mainly) on the public internet where it be accessed by anyone with an internet connection (and permission to do so!). + +![](slides/Slide5.png) +--- + +Cloudant wasn't written entirely by IBM. It is based on Apache CouchDB, an open source project run by the Apache Foundation. Cloudant employs a number of CouchDB contributors but by the rules of Apache, they cannot monopolise its development. + +Much of what you see in this course is applicable to Apache CouchDB as it is to Cloudant. Their APIs are 99% the same - I'll point out where they diverge. + +Cloudant can be thought of as CouchDB run "as-a-service". A Cloudant service is easily deployed and is managed by IBM engineers 24-7. There's no software to install, no servers to manage, no configuration to understand. The user need not be a CouchDB expert to use and manage it. + +Cloudant being built on truly open-source foundations means that you can be sure that your data layer is not _locked in_ to a particular platform, cloud or vendor and Cloudant can be used in concert with CouchDB to create hybrid applications that share data through replication, as we'll see. + +![](slides/Slide6.png) +--- + +Later on in the course we'll look "under the hood, to see how Cloudant works, but initially we'll treat Cloudant as a "black box". + +![](slides/Slide7.png) +--- + +To summarise + +- Cloudant is based on Apache CouchDB, an open-source project. +- it stores JSON documents. +- it is accessed with an HTTP API and can therefore be accessed by any device on the internet that speaks HTTP: application code, web browser, IoT device or mobile phone. +- Cloudant is a highly-available managed service able to continue to operate with multiple hardware failures + +![](slides/Slide8.png) + +--- + +That's the end of this part. The next part is called ["The Document"](Part\ 02\ -\ The\ Document.md) + +![](slides/Slide0.png) + +--- + + + diff --git a/Part 02 - The Document.md b/Part 02 - The Document.md new file mode 100644 index 0000000..9f93a38 --- /dev/null +++ b/Part 02 - The Document.md @@ -0,0 +1,121 @@ +![](slides/Slide0.png) + +Welcome to the Introduction to Cloudant course, an eighteen part video series that gives you an overview of the IBM Cloudant databases-as-a-service. + +![](slides/Slide1.png) + +--- + +This is part 2: "The Cloudant Document?" + +We've seen that Cloudant is a JSON document store. Let's find out what that means in practice and how that compares to other types of database. + +![](slides/Slide9.png) + +--- + +Most databases store their data in collections called _tables_, where each unit of data is a _row_, each with identical, fixed columns. The schema of each table is predefined: a list of columns with their name, date type, value constraints and relations to other tables carefully defined. Each new record forms a row in a table. + +Cloudant is quite different! + +A Cloudant service has collections called _databases_ (instead of _tables_) each of which containing any number of documents. + +The example on this slide shows the same data expressed in a traditional tabular database and how the same data would be stored in Cloudant as JSON documents. + +So if you come from a relational database background: tables are "databases" in Cloudant, and rows are "documents". + +![](slides/Slide10.png) + +--- + +A Cloudant document must be a JSON object, starting and ending with curly braces and containing a number of key/value attributes. + +JSON objects must be less that 1 megabyte in size and contain any number of strings, numbers, booleans, arrays and objects. The nesting of objects within objects can continue to any depth. + +The keys used can be as brief or verbose as you like. + +Here are some simple example documents showing how each data type is used. + +- the first example shows a person object, storing strings, booleans, and an array of tags. +- the second example shows very brief attribute names, to save on storage and represents a web event such as a click on a website. +- the last example shows how the document may itself contain sub-objects + +A note on dates. JSON has no native Date type so dates are usually stored in 2018-10-30 or similar formats - we will come back to dates later. + +![](slides/Slide11.png) + +--- + +Now for your first practical exercise. Visit www.ibm.com/cloud and register an account with the IBM Cloud, if you don't have one already. + +One registered, you may click on "services", search for the "Cloudant" database and provision a new service. + +The Cloudant "Lite" service provides a free plan to allow users to try Cloudant in a limited capacity while in development. It's bigger brother, the "Standard Plan", is paid-for service where you specify the number of reads/writes/and queries per second your application and that capacity is reserved for you. You pay for the capacity you provision and your data storage usage. + +The Lite plan operates in a similar way, but only has a small provisioned capacity and a fixed storage size, but is fine for "kicking the tyres". + +![](slides/Slide12.png) + +--- + +Cloudant is often referred to as a "schemaless" database - but we have to be careful how we define that term. + +It's true to say that there's no need to define your schema (field names, types, constraints and relationships) ahead of time in a Cloudant database - you may simply write a JSON document of your own design to a database. + +This flexibility is well liked by developers because they can design their data in their code, turn it into JSON and write it to the database. + +It's still important to think about the "shape of your data", especially in terms of how you are going to query and index it, as we'll see later. + +Data design is still required, but strictly speaking that database doesn't need to know about your schema. + +![](slides/Slide13.png) + +--- + +Let's say we want to create a database of US presidents. We can simply devise our "model" of the data in our app, turn it into JSON and write it to the database. In this case we are using a common CouchDB convention: the "type" field indicating the data type of the document. + +![](slides/Slide14.png) + +--- + +If at a future date we decide we want to add additional data to our "schema", we can simply write a new object to the database with no complaints from Cloudant. We could decide to add the "address" object only to: + +- documents that are created from now on +- only documents that we know addresses for + +In other words, documents of the same type can have fields present or missing. + +You database's schema can evolve over time to match your application's needs and you don't (necessarily) need to tell the database about the schema change - just write new documents in the new format. + +![](slides/Slide15.png) + +--- + + +We can even store multiple document "types" in the same database. In this case, people/books/places reside in the same database. We know which is which because of the "type" field (this is a convention and not something that means anything to Cloudant). + +An alternative to this is have three databases people/books/places and keep each data type in its own database. Both approaches are fine: you would choose to have multiple types together in the same database if need to perform queries _across types_ or if you need to replicate all data types together, otherwise the _separate databases_ approach may be better. + +![](slides/Slide16.png) + +--- + +Although Cloudant is "schemaless", this doesn't absolve you of the need to do detailed data design to get the best performance. + +Here are some tips, especially relevant if you have some relational database experience. + +- avoid thinking in joins - a Cloudant document should contain everything you need about that object, so that it can be retrieved in one API call. +- normalisation goes out of the window in JSON store, some repeated values can be tolerated if it makes data retrieval more efficient. +- although we have a 1MB limit on document size, your documents should be much smaller than that - a few KB is typical. +- If your application can embrace a "write only" design pattern, where data is only ever added to a database, then it may make your life easier. You should definitely avoid patterns that rely on updating the same document over and over in small time window. + + +![](slides/Slide17.png) + +--- + +That's the end of this part. The next part is called ["The Document id"](Part\ 03\ -\ The\ Document\ _id) + +![](slides/Slide0.png) + +--- \ No newline at end of file diff --git a/Part 03 - The Document _id.md b/Part 03 - The Document _id.md new file mode 100644 index 0000000..13ea5b4 --- /dev/null +++ b/Part 03 - The Document _id.md @@ -0,0 +1,66 @@ +![](slides/Slide0.png) + +Welcome to the Introduction to Cloudant course, an eighteen part video series that gives you an overview of the IBM Cloudant databases-as-a-service. + +![](slides/Slide1.png) + +--- + +This is part 3: "The Document _id" + +We've seen how data is stored in Cloudant documents with flexibility on how your application stores JSON objects in Cloudant databases. There are, however, a few hard and fast rules. + +![](slides/Slide18.png) + +--- + +One rule is that every document must have a unique identifier call `_id` which is a string. No two documents in the same database can have the same _id field. In other database, you specify which column is the unique identifier, but in Cloudant it's always `_id` and can't be changed. + +![](slides/Slide19.png) + +--- + +Also, unlike relational databases, Cloudant does not have "auto incrementing ids", that is an id field that starts at 1 and increments for each document added. + +Cloudant's `_id` field is either: + +- a 32 character string generated by Cloudant - the id is meaningless sequence of numbers and letters that is guaranteed to be unie other than it uniquely identifies a document in a database +- a string defined by you (if you know something unique about your data) + +![](slides/Slide20.png) + +--- + +Here are some examples of supplying your own document _id : + +- using it to store something that you know is unique i.e. the email address of a user. Your registration mechanism can enforce a one-user-per-email address policy. +- Some users choose to encode the document type in the `_id` e.g. user:56, book:55 +- The last example shows using a 32-digit string (generated in your app) that is designed to sort in approximate date/time order, making it easy to retrieve the latest documents from the database, without a secondary index. + +![](slides/Slide21.png) + +--- + +Cloudant takes your document `_ids` and stores them in an index (like the contents page of book). This primary index is in `_id` order and is used to allow Cloudant to retrieve documents by `_id` - thus behaving like a key/value store. + +By careful design of your `_id` field, you can make use to the primary index to keep data that makes sense to be together in the primary index, which makes it quicker to retrieve that data. We've already seen that using time-sortable `_id`s means that data can be retrieved in date/time order. + +We'll see this later when it comes to retrieving ranges of document ids. + +![](slides/Slide22.png) + +--- + +In conclusion, each document must have a `_id` field that is unique in the database. It can be auto-generated by Cloudant, or can be supplied by your application, which must take responsibility of the uniqueness of the data. + +The `_id` field is the basis of the database's primary index which, as we'll see, can be used for key/value and range lookups. + +![](slides/Slide23.png) + +--- + +That's the end of this part. The next part is called ["The rev token"](Part\ 04\ -\ The\ rev\ token.md) + +![](slides/Slide0.png) + +--- \ No newline at end of file diff --git a/Part 04 - The rev token.md b/Part 04 - The rev token.md new file mode 100644 index 0000000..a878ed2 --- /dev/null +++ b/Part 04 - The rev token.md @@ -0,0 +1,89 @@ +![](slides/Slide0.png) + +Welcome to the Introduction to Cloudant course, an eighteen part video series that gives you an overview of the IBM Cloudant databases-as-a-service. + +![](slides/Slide1.png) + +--- + +This is part 4: "The rev token" + +The second fundamental Cloudant rule is that each document revision is given its own unique revision token. Let's find out what it means + + +![](slides/Slide24.png) + +--- + +You never need to generate a revision token - one is created for you when you add/update/delete a document using the API. + +A revision token consists of two parts: + +- a number 1, 2, 3, etc +- a cryptographic hash of the document's body + +(For the uninitiated, a hash is a digital "fingerprint" of some data. If the data changes, the fingerprint changes. No two fingerprints are the same i.e. no two documents with different content would have the same hash.) + +You can see from the example on the right that our document has a revision token (the key starting `_rev`) that starts with a "1" followed by a dash. That tells us that this is the first revision of the document. The digits starting 04aa8... are the cryptographic hash of the document. + +![](slides/Slide25.png) + +--- + +If we follow the lifecycle of a document, it starts with a "revision 1". When it's modified later, it gets a "revision 2" and so on. With each incrementing revision number, the hash also changes because the content of the document is being modified too. + +One thing to note: + +> It is *possible* for a document to have more than one revisions with the same number. i.e. two "revision 3s". This is called a "conflict" and is "normal" in some circumstances. We'll see why later in the course, but for now we can assume that the revision number will increment with update to a document. + +![](slides/Slide26.png) + +--- + +Let's follow the lifecycle of an example Cloudant document + +When a new document is created (auto-generated `_id` or user-supplied `_id`), it is allocated a "revision 1". You will be sent the token in the response to your API request. Normally you can discard the _id (UNLESS you intend to modify the document in the near future, as we'll see). + +![](slides/Slide27.png) + +--- + +When we modify a document whose `_rev` is at "revision 1" (notice we've change the name from Liz --> Elizabeth), the document is saved and a "revision 2" token is generated and returned to you in the API response. + +All simple enough so far. + +![](slides/Slide28.png) + +--- + +If we delete the document later, A "revision 3" is created ! + +Unlike almost any other database, Cloudant keeps a reference for deleted documents. A deletion is just another document revision - a special one where `_deleted: true` replaces the document body. + +In fact the document's recent revision history (the tree of revisions - remember we could have more than one of each revision number) - is kept. + +Note + +> You can't use Cloudant's revision tree as version control system to retrieve or "rollback" to old revision. Once revision is superceded, the document _body_ of the older revision is deleted and its disk space recovered in a process called "compaction". Compaction occurs automatically in Cloudant, so it's not safe to assume that old revisions will be available to be retrieved. + +![](slides/Slide29.png) + +--- + +To summarise: + +- revision tokens are generated by the database on add/edit/delete. (you never need to create your own revision tokens). +- generally, the revision number increases by one each time, but more complex scenarios are possible (we'll cover this later). +- older document bodies are discarded or _compacted_ (don't rely on being able to get them back). +- all Cloudant operations that change a document need the document's `_id` AND its `_rev` (this is unlike most databases) + +![](slides/Slide30.png) + +--- + + +That's the end of this part. The next part is called ["Authentication"](Part\ 05\ -\ Authentication.md) + +![](slides/Slide0.png) + +--- \ No newline at end of file diff --git a/Part 05 - Authentication.md b/Part 05 - Authentication.md new file mode 100644 index 0000000..b72c28c --- /dev/null +++ b/Part 05 - Authentication.md @@ -0,0 +1,78 @@ +![](slides/Slide0.png) + +Welcome to the Introduction to Cloudant course, an eighteen part video series that gives you an overview of the IBM Cloudant databases-as-a-service. + +![](slides/Slide1.png) + +--- + +This is part 5: "Authentication" + +We said earlier that Cloudant is a web-based service on the public internet. How can we be sure that our data is safe and that only our code can access it? This is where _authentication_ comes in. + +![](slides/Slide31.png) + +--- + +Cloudant supports two types of authentication. + +1. Legacy authentication is where are either supplied with each request using _HTTP Basic Authentication_ or exchanged for cookie, using a one-off `session` API call. The cookie is cycled regularly, so you client code needs to capture a refreshed cookie and stored for subsequent request. +2. IAM authentication is the access management system that powers all of the IBM Cloud services. To authenticate with IAM, you need the IAM API key and the host name of the Cloudant service. The API key is exchanged for a _bearer token_ using the IAM API and bearer token is passed to Cloudant with each request. The _bearer token_ only lasts for an hour, so must be renewed using the IAM service periodically. + +When a Cloudant service is provisioned, you can generate _IAM only_ credentials or both _IAM_ and _Legacy_ credentials or Both - you decide. + +![](slides/Slide32.png) + +--- + +How are credentials generated? + +In the IBM Cloud dashboard under your Cloudant service, in the "Service Credentials" tab, click the "New Credential" button and a JSON document containing the IAM key and the basic auth username/password and the Cloudant hostname is created. + +![](slides/Slide33.png) + +--- + +Here's an example set of credentials: + +- for IAM you need the top two bits: apikey & host +- for Legacy/Basic-Auth you need the URL (which contains the username and password embedded in the URL) + +![](slides/Slide34.png) + +--- + +There are three official Cloudant libraries: Java, Node.js & Python. + +All three handle authentication automatically. You don't need to worry about how it exchanges the API key for a session token or IAM authentication works - it's handled for you. + +When we look at the API from the command-line, we'll be using Basic Auth as a convenience, but it's recommended you use IAM authentication if possible as it allows better integration with the IBM Cloud platform and finer-grained permissions. + +![](slides/Slide35.png) + +--- + +Time for our next practical exercise. + +Log into the IBM Cloud and locate the IBM Cloudant Lite service we created last time. In the "Service Credentials" tab, click "New Credential" button to generate a set of "IAM+Legacy" credentials. Make a note of the JSON it returns - we'll need that for the next exercise. + +Then visit the URL specified in the credentials JSON - what do you see? + +![](slides/Slide36.png) + +-- + +To summarise: + +Credentials are generated from the IBM Cloud dashboard. You can have IAM or both IAM + legacy credentials. Both authentication methods involve exchanging your credentials for a time-limited token (authentication) - the token is then updated periodically as you use the service. The official libraries handle all of this for you. + + +![](slides/Slide37.png) + +--- + +That's the end of this part. The next part is called ["The dashboard"](Part\ 06\ -\ The\ Dashboard.md) + +![](slides/Slide0.png) + +--- \ No newline at end of file diff --git a/Part 06 - The Dashboard.md b/Part 06 - The Dashboard.md new file mode 100644 index 0000000..ca8b7b8 --- /dev/null +++ b/Part 06 - The Dashboard.md @@ -0,0 +1,85 @@ +![](slides/Slide0.png) + +Welcome to the Introduction to Cloudant course, an eighteen part video series that gives you an overview of the IBM Cloudant databases-as-a-service. + +![](slides/Slide1.png) + +--- + +This is part 6: "The Dashboard". + +The easiest way to get started creating databases and adding documents is to use the Cloudant Dashboard. + +![](slides/Slide38.png) + +--- + +The Cloudant Dashboard is a web-app built into the service. It allows basic data manipulation to be performed through a graphical user interface: databases can be created and deleted, documents added, updated and deleted and replication jobs managed. It is also a handy place to perform one-off queries and to set up secondary indexes (as we'll see later). + +It also contains some simple monitoring tools that visualise request rates. + +It's important to note that any task that is achievable in the Cloudant Dashboard is also possible with the Cloudant HTTP API - indeed, the Cloudant Dashboard is simply making standard API calls itself + +![](slides/Slide39.png) + +--- + +To open a Cloudant service's Dashboard, log into the IBM Cloud, find your Cloudant service and click Launch Cloudant Dashboard button. A new window will pop up, logging you into your Cloudant dashboard. + +If you leave the dashboard window unattended for a length of time, you will find yourself logged out (for security purposes) and will have to click Launch again. + +![](slides/Slide40.png) + +--- + +The dashboard has number of tabs on the left hand side. Its default tab "Databases" that allows you to list the databases you have created in pages of 20. Each database is shown with the number of documents it is storing and how much disk space is being used. Click on a listed database name to examine its contents. + +![](slides/Slide41.png) + +--- + +To create a database, click on the Create Database button and supply the name of the database to create. + +![](slides/Slide42.png) + +--- + +We now have a new empty database. The database's documents would be listed here in id order, but as this is a new database, there are none. To add a new document, click Create Document. + +![](slides/Slide43.png) + +--- + +The Cloudant dashboard has created a template document for you with a pre-generated _id. Complete the rest of the attributes yourself and click "Create Document" to save. + +![](slides/Slide44.png) + +--- + +Now it's time for another practical exercise. Create a new database called "books" and in that database create three or more documents with fields: title, author, date, publisher, ISBN. + +Once created, edit one of the documents, modifying the publication date. + +Then delete one of the documents. + +![](slides/Slide45.png) + +--- + +To summarise: + +- The Cloudant Dashboard is a web app app built into the Cloudant service. +- It's used to manage databases, documents, indexes, queries and replication jobs. +- It can also be used to monitor service throughput. +- The Dashboard is simply an API client - anything that can be achieved with the dashboard can scripted by you using the HTTP API. + + +![](slides/Slide46.png) + +--- + +That's the end of this part. The next part is called ["HTTP API Basics"](Part\ 07\ -\ HTTP\ API\ Basics.md) + +![](slides/Slide0.png) + +--- \ No newline at end of file diff --git a/Part 07 - HTTP API Basics.md b/Part 07 - HTTP API Basics.md new file mode 100644 index 0000000..4d2f2ad --- /dev/null +++ b/Part 07 - HTTP API Basics.md @@ -0,0 +1,166 @@ +![](slides/Slide0.png) + +Welcome to the Introduction to Cloudant course, an eighteen part video series that gives you an overview of the IBM Cloudant databases-as-a-service. + +![](slides/Slide1.png) + +--- + +This is part 7: "HTTP API Basics". + +In the previous part we saw the Cloudant dashboard which is a web app that is making HTTP calls to Cloudant's API. In this step we'll be using the command line to make HTTP requests and to try to add/edit/delete some documents from there. + +It's worth understanding the HTTP API from first principles even if you intend to use the higher-level client libraries. + +![](slides/Slide47.png) + +--- + +The advantage of a database having an HTTP API is that any device on the internet can read/write data if you want it to. No special software is required. No drivers speaking a custom protocol. Just an standard HTTP library. Everything speaks HTTP: + +- web browser +- any programming language +- command-line scripting tools like curl +- mobile devices + +We're going to learn the API using curl, a free, open-source command-line tool that can dispatch HTTP requests. curl comes pre-installed on most Macs and Unix-like operating systems. If it's no present on your computer, google "curl" and follow the installation instructions. + +![](slides/Slide48.png) + +--- + +Let's first use curl to fetch a web page - Google's homepage. + +In a command-line terminal type "curl https://www.google.com" + +You should get a pageful of HTML in reply. + +If this works, then you have curl installed and you can proceed with the next tasks. + +![](slides/Slide49.png) + +--- + +Now we don't want to type the URL of our Cloudant service every time so let's save the Cloudant URL in an environment variable called URL. + +`export URL=` creates a variable called URL which we can access later. + +Furthermore, we can create an "alias" (a shortcut) called "acurl" which saves us further typing. This "acurl" command is an alias for curl but with the JSON content-type header and a couple of useful commmand-line switches. + +We can test it by fetching "acurl $URL/" and we should get some JSON back from Cloudant. + +Congratulations! You've just made your first Cloudant API call. + +![](slides/Slide50.png) + +--- + +Now our `acurl` alias is set up, we can start exploring the API. Let's start with the `_all_dbs` endpoint which returns a list of databases. + +Type `acurl $URL/_all_dbs` to see an array of databases. + +A quick note here on formatting JSON on the command-line. We can send the output of our `acurl` command to another tool which will format the data nicely on the terminal. There are are couple of options: + +- jq available from the [URL on screen](https://stedolan.github.io/jq/) which is more than just a JSON formatter - it allows JSON to be parsed, queried and manipulated too. +- or `python -m json.tool` is a simple JSON formatter, if you happen to have Python installed on your computer. + + +So `acurl $URL/_all_dbs | jq` means "pipe the output of acurl into jq" and what you see should be a nicely formatted, coloured output. + +![](slides/Slide51.png) + +--- + +The Cloudant API paths are hierarchical with the top level giving you information about the service and then each database sits at a level below that. + +So `acurl $URL/books` gives us information about the `books` database we created earlier. + +You should see information about how many documents it has, how many deleted documents and how much disk space it's occupying. + +> Note: don't forget to pipe the output to jq or python to get a prettier output. + +![](slides/Slide52.png) + +--- + +If we want to see the documents contained in the database we can use the `_all_docs` endpoint. + +So `acurl $URL/books/_all_docs` means get all the documents from the books database from the Cloudant service at the supplied URL. + +This will return you a list of `_id` and `_rev` values for each document. If you want the document bodies too, then add `?include_docs=true` to your API call. + +![](slides/Slide53.png) + +--- + +If we want to a single document back from the database, then documents sit one level below the database in the hierarchy of the URL. + +So `acurl $URL/books/id` means get get document id from the database books from the Cloudant service at the supplied URL. + +![](slides/Slide54.png) + +--- + +So far we've only used the "GET" HTTP method, which is the default one for curl and the one used when you enter a URL into your web browser. + +Cloudant's API often uses the method as a "verb" to describe the action being asked of the database: GET for fetching data. + +With curl we can specify the method we want to use with the `-X` command-line option. + +![](slides/Slide55.png) + +--- + +So to write a new document to our books database using the API, we're going to use the POST method, massing a document as the body of the HTTP request. + +`acurl -X POST` specifies we're using the POST HTTP method. `-d` specifies the document we want to write, which is sent as the body of the request and finally the URL we are writing to which is `$URL/books` - the books database. + +Alternatively we can use the `PUT` method, if we are supplying the id of the document being written. The URL becomes "$URL/books/" followed by the id we wish to write. + +Both write methods yield identical response. `ok: true` to show that the write was successful. `id` being the document id written and `rev` being the revision token that was generated by the database. + +![](slides/Slide56.png) + +--- + +To modify a document we can use the PUT method, writing the new body to the URL that points to the document id we wish to overwrite. `-d` supplies the new document body and URL not only contains the database and id of the document, but critically the `rev` - the revision of the document we intend to mutate. + +If we forget and omit the the `rev` parameter we will get an error response. + +Note: HTTP response codes tell you whether a request succeed or not. Responses in the 200 range are successful, 400s are user errors (e.g. invalid parameters) and 500s are server-side errors. You can see the full HTTP request and response by supplying the `-v` command-line option to curl/acurl. + +Also note that updates to documents happen in their entirety or not at all, there's no API construct to modify part of a document. A whole document must be supplied to overwrite a previous revision. + +![](slides/Slide57.png) + +--- + +Finally to delete a document we use the `DELETE` method, so `-X DELETE`. We direct the request to the URL that includes the database name and document to be deleted, and critically we also supply the rev - the revision of the document to delete. + +If we omit the revision token, an error is returned and the request fails. + + +![](slides/Slide58.png) + +--- + +To summarise: + +Understanding the HTTP API helps you grasp the relationship between your code and the Cloudant service. + +The URLS are hierarchical: service/database/document + +and the HTTP methods act as "verbs" defining the action to be done. + +All actions can be triggered using simple HTTP API calls, from the command-line or from your code and so can be easily scripted. + +![](slides/Slide59.png) + +--- + + +That's the end of this part. The next part is called ["The Bulk API"](Part\ 08\ -\ The\ Bulk\ API.md) + +![](slides/Slide0.png) + +--- \ No newline at end of file diff --git a/Part 08 - The Bulk API.md b/Part 08 - The Bulk API.md new file mode 100644 index 0000000..e6fcdb3 --- /dev/null +++ b/Part 08 - The Bulk API.md @@ -0,0 +1,55 @@ +![](slides/Slide0.png) + +Welcome to the Introduction to Cloudant course, an eighteen part video series that gives you an overview of the IBM Cloudant databases-as-a-service. + +![](slides/Slide1.png) + +--- + +This is part 8: "The Bulk API". + +In the previous part we saw how documents could be easily added updated, updated and deleted singly using the Cloudant HTTP API. In this part, we'll see how two API calls can be used to achieve all of the basic Cloudant operations, with the added benefit of being able to act on more that one document in an API call. + +![](slides/Slide60.png) + +--- + +We've already met the `_all_docs` endpoint - we used it to fetch a list of all the documents in a database, but it has other features too. + +- the `key` parameter can be used to specify a single document to fetch, making it equivalent of the `GET /db/id` API call. +- similarly the `key` parameter takes an array of document ids and will return them all. +- the `startkey` & `endkey` parameters allow you to fetch a slice of the primary index between the supplied limits. Adding `include_docs=true` instructs Cloudant to supply the document bodies too. +- and `limit` allows you to specify how many documents to return in one API call. + + +![](slides/Slide61.png) + +--- + +The `_bulk_docs` endpoint allows multiple insert, update and delete operations to be performed in one API call. It expects an object containing a `docs` array - each element of that array. The response body is posted to Cloudant, allowing many operations to be packed into a single API call. + +In this example, the first document is an insert: we know this because no revision token is supplied. +The second document is an update to a document because a revision token is supplied with a new document body. +The third document is a deletion. A revision token is supplied, but the body is simply `_deleted: true`, which tells Cloudant to mark the document as deleted. + +It's important to note that isn't like a transaction in a relational database - all or none of these operations could succeed or fail. The response data to this request tells you response for each operation individually. + + +![](slides/Slide62.png) + +--- + +In summary + +With two API calls `_bulk_docs` & `_all_docs` we can perform all create, read, update & delete operations on Cloudant documents and be able to do so in bulk too. `_all_docs` retrieves documents by `_id` or ranges of ids. `_bulk_docs` creates, updates & delete documents in bulk. As a rule of thumb, its recommended that bulk writes are executed in batches of 500; perhaps more than that for tiny documents and fewer than that for large documents. + + +![](slides/Slide63.png) + +--- + +That's the end of this part. The next part is called ["Accessing Cloudant programmatically"](Part\ 09\ -\ Accessing\ Cloudant\ programmatically.md) + +![](slides/Slide0.png) + +--- \ No newline at end of file diff --git a/Part 09 - Accessing Cloudant programmatically.md b/Part 09 - Accessing Cloudant programmatically.md new file mode 100644 index 0000000..cfd993e --- /dev/null +++ b/Part 09 - Accessing Cloudant programmatically.md @@ -0,0 +1,66 @@ +![](slides/Slide0.png) + +Welcome to the Introduction to Cloudant course, an eighteen part video series that gives you an overview of the IBM Cloudant databases-as-a-service. + +![](slides/Slide1.png) + +--- + +This is part 9: "Accessing Cloudant programmatically. + +So far, our API interactions have either been triggered by the dashboard or via `curl` from the command line. In this section we'll see how Cloudant be accessed programmatically. + +![](slides/Slide66.png) + +--- + +The examples will use Node.js, so if you want to try the code examples yourself, you'll need to install `node` & `npm` from [nodejs.org](https://nodejs.org). + +Once installed we can install the official Cloudant node.js library with `npm install @cloudant/cloudant`. (npm is the package manager that comes with Node.js - allowing you to access thousands of open-source projects and build them into your application for free). + +![](slides/Slide67.png) + +--- + +Once the `cloudant` library is installed we can build some source code. Let's go through this code snippet line-by line: + +1. The URL of the Cloudant service is gleaned from the environment variable we created earler. +2. The `@cloudant/cloudant` library is loaded into your Node.js app with the built-in `require` function. +3. We then create an instance of the library configured with our credentials we stored in the first line. +4. We use the `cloudant` object to get a reference to the `books` database and store it in a variable `db`. At this point, we haven't made any API calls - only created data structures that store credentials and which database we are working on. +5. The main function calls `db.list` which maps 1-1 with the `_all_docs` endpoint we saw earlier. The parameters passed to `db.list` should be familiar as the options that `_all_docs` expects to limit the result set and to return document bodies for each id. + +![](slides/Slide68.png) + +--- + +Here's another code snippet that writes a document. + +You can see from the first line that standard JavaScript objects can be used in your code and sent to Cloudant with no conversion, as they turn into JSON natively in JavaScript. + +Writing a document is a matter of simply calling `db.insert` which will map to a PUT/POST API call or to `_bulk_docs`. + +![](slides/Slide69.png) + +--- + +To summarise + +There are official Cloudant libraries for Java, Python and Nodejs. They are thin wrappers around the Cloudant HTTP API - so it's worth understanding the underlying API to get to grips with all the parameters. + +The libraries handle two things for you, which are really useful: + +- authentication - exchanging your keys for tokens, whether it be legacy authentication or IAM. +- retry logic - the libraries can be configured to retry API calls that exceeded your provisioned capacity. If configured this way, they pause and re-attempt the API multiple times with exponential back-off + +Note: retrying such API calls is sensible if your have a temporary and unexpected elevation in traffic, but if you are routinely exceeding your provisioned capacity, no amount of retrying will get the database work done - you need to more capacity! + +![](slides/Slide70.png) + +--- + +That's the end of this part. The next part is called ["Querying"](Part\ 10\ -\ Querying.md) + +![](slides/Slide0.png) + +--- \ No newline at end of file diff --git a/Part 10 - Querying.md b/Part 10 - Querying.md new file mode 100644 index 0000000..d6b878c --- /dev/null +++ b/Part 10 - Querying.md @@ -0,0 +1,119 @@ +![](slides/Slide0.png) + +Welcome to the Introduction to Cloudant course, an eighteen part video series that gives you an overview of the IBM Cloudant databases-as-a-service. + +![](slides/Slide1.png) + +--- + +This is part 10: "Querying". + +So far we have done CRUD operations from the command-line, the dashboard and from code. These are operations centred on the document's `_id`: + +- fetch document by id +- update document whose `_id` = 'x' +- delete document whose `_id` = 'x' +- get documents in the `_id` range 'a' to 'z' + +These operations are the building blocks of a database but they only get you so far. What if you need to return a sub-set of documents matching on fields *within* the document. A person's birth date? A book's title? An order's value? + +This is where Querying comes in.... + +There are several methods for querying data in Cloudant. The first we'll look at is called "Cloudant Query". + +![](slides/Slide71.png) + +--- + +Cloudant Query's language was inspired by the MongoDB query language. Queries are expressed in JSON, where the "selector" attribute describes the subset of data to return. The query JSON is posted to the database's `_find` endpoint to perform a query. + +The simplest form of query is finding documents where an attribute has a fixed value e.g. where author == J Smith. + +The second example shows two clauses in the query, both of which must be satisfied for a document to make it into the search results. e.g. where isbn === 6725252 AND date = 2018-01-01 + +The third example shows how logical operators can be added. The '$gt' operation means "greater than" (there's also gte for greater than or equal to, and lt/lte for the equivalent less than comparators). The `$or` operator is an OR operation, so a matching document must have a date greater than the one in the query an either an author of J Smith OR title of Murder!. + +Note: if you need to access objects within documents, you can use standard "dot notation" e.g address.zipcode to access a zipcode string inside an address object. + +![](slides/Slide72.png) + +--- +We can also add + +- "fields" - to specify the document attributes we want returned (the default is the entire document. +- Sort – to define how the data is to be sorted. Sort is an array, allowing the sort to be calculated on multiple attributes. +- Limit – the number of documents to return + + +If you are from a relational database background, this is the equivalent SQL query to that last Cloudant query example. +![](slides/Slide73.png) + +--- + + +If you are from a relational database background, this is the equivalent SQL query to that last Cloudant query example. The WHERE clause is the equivalent of SELECTOR in Cloudant Query, ORDER and LIMIT are exactly equivalent and the Cloudant Query FIELDS is handled in the list of fields listed after the SELECT keyword. + +The JSON syntax may take a bit of getting used to, but MongoDB users should find it familiar. + +![](slides/Slide74.png) + +--- + +Cloudant queries can be executed in the Cloudant Dashboard. Select the database you are working with e.g. `books` then choose the `Query` tab. + +Enter your Cloudant Query JSON in the box provided and click Run Query when you're ready. The result set will appear on the right-most panel. + +Note: the `Explain` button is used to provide an explanation on how the database interprets the supplied query. This becomes more important when we get to Indexing in the next part. + +![](slides/Slide75.png) + +--- + +Queries can also be triggered from `curl` too. The Query JSON, in this case, is stored in a file and POSTed to the `_find` endpoint using the `-d@` command-line syntax. + +![](slides/Slide76.png) + +--- + +The Node.js codes is very similar. The Query is a standard JavaScript object which is passed to the `db.find` function which is POSTs to the `_find` endpoint on your behalf. + +![](slides/Slide77.png) + +--- + +Now time for a practical exercise. Devise your own Cloudant Query that finds the titles of books _written in the 20th Century_. The Cloudant Query documentation is [here](https://console.bluemix.net/docs/services/Cloudant/api/cloudant_query.html#query +) if you need it. + +Pause the presentation here if you don't want to know the answer... + +![](slides/Slide78.png) + +--- + +Here's one solution: + +I'm using the `$and` operator to combine two clauses on the `date` attribute. One clause to locate documents whose date `>=` 1900, the other to find documents whose date is < the year 2000. Both clauses have to be true to select a document. As we only need the `title` of the matching books, we can supply a `fields` attribute instead of being returned the entire document. + +![](slides/Slide79.png) + +--- + +To summarise + +Cloudant Query is a query language inspired by MongoDB where the syntax is expressed in JSON form. + +Queries select subsets of documents from the database, using clauses operating on data inside the document - not just the document's `_id`. + +Queries are sent to the database's `_find` endpoint, either programmatically, using curl or via the Dashboard. + +The query's selector decides which cut of data is required, + +![](slides/Slide80.png) + +--- + +That's the end of this part. The next part is called ["Indexing"](Part\ 11\ -\ Indexing.md) + +![](slides/Slide0.png) + +--- \ No newline at end of file diff --git a/Part 11 - Indexing.md b/Part 11 - Indexing.md new file mode 100644 index 0000000..c48d63d --- /dev/null +++ b/Part 11 - Indexing.md @@ -0,0 +1,95 @@ +![](slides/Slide0.png) + +Welcome to the Introduction to Cloudant course, an eighteen part video series that gives you an overview of the IBM Cloudant databases-as-a-service. + +![](slides/Slide1.png) + +--- + +This is part 11: "Indexing". + +The queries we executed in the previous part were not optimal: to get the answer, Cloudant had to spool through every document in the database in turn to see if it met with the search criteria. + +To make queries run in a performant and scalable way, we need "Indexing". + +![](slides/Slide81.png) + +--- + +Cloudant allows you to specify any number of "Indexes" (indicies). + +An index is a secondary data structure built from the document list. It contains data sorted by the fields you specify e.g. books sorted by date and title. If you perform a query asking a document's date & title, the index can be used to speed up the query process: instead of scanning through every document in turn, Cloudant can jump to the relevant part of the index (say, the section on 20th century books) and retrieve the data much more quickly. + +There are two types of Cloudant Query indexes: type=json & type=text, which are backed by two underlying indexing technologies we'll meet in subsequent parts of this course. + +An index is defined by POSTing some JSON to a database's `_index` endpoint. The `index` object contains a `fields` array which specifies which document attributes to index. As a rule of the thumb, the fields that need indexing are usually equivalent to the attributes used in the "selector" of a query. i.e. if you need to query by the `date` field, we need to index the `date` field. + +Although the "name" of an index is optional, it's good practice and we'll follow this convention. It's good to ask Cloudant a question and specify the name of the index you intend it to use. This saves Cloudant from having to choose which index it should use from those available and it makes it easy for you to remember which index is which. + +![](slides/Slide82.png) + +--- + +Let's create an index on our books database from the dashboard. Select the database, then choose the "Design Documents" tab and "Query Indexes" from the pop out menu. + +![](slides/Slide83.png) + +--- + +Any existing indexes are listed on the right-hand side: there should be a "special" index representing the primary index, based on the document's `_id`. Complete the index definition with the JSON: + +```js +{ + "index": { + "fields": ["date"] + }, + "name": "booksByDate", + "type": "json" +} +``` + +and click "Create Index" when you're done. + +Clicking the button sends a `POST` request to the `_index` endpoint (other API calls are available to update and delete existing indexes). + +Note: indexes are built asynchronously by Cloudant in the background. For large databases, it may take Cloudant some time construct the index for the first time. The index will not be able to use until that initial build is ready, + +![](slides/Slide84.png) + +--- + +We can repeat our query for books in the 20th century, this time specifying the index name with the `use_index` field and the answer should return, this time powered by our index. You may not notice a speed improvement for a very small database, but the benefit is definitely felt as your data size and query volume grows. Indexing helps your queries remain performant as your application scales. + +![](slides/Slide85.png) + +--- + +When you tell Cloudant to create a secondary index, it kicks off a background task looking at all the documents in turn and creates a new data structure on disk: the index. The index is a balanced tree which pairs the keys (the attribute or attributes you need indexed) with the document `_id` they came from and the fields you specified in your `fields` attribute. + +The index can be used to efficiently look up known keys and ranges of keys without having to rescan the entire database. + +![](slides/Slide86.png) + +--- + +There is another trick that can be employed at index time: the _partial filter_. You may optionally supply a _partial filter_ in your index definition, and this selector will be executed at _index time_ to decide which documents' data make it to the index and which are ignored. In this example, a selector that only allows dates that fall on a weekend to make it to the index. Smaller indexes are faster and more efficient, so if you have a use-case that only needs a subset of the data to be indexed, e.g. only completed orders, or only expired accounts, or only published blog posts, then a _partial filter_ selector at index-time can help to make the index smaller and more efficient. + +![](slides/Slide87.png) + +--- + +To summarise: + +Indexing is used to speed up queries by using a pre-built data structure (the secondary index) to help look up keys and key ranges from values inside the document body more efficiently. Without an index, queries won't scale in a performant manner as data size and query volumes increase. + +The `_index` endpoint is used to define the index and an optional partial filter can be applied at query time to make smaller, sparser indexes. + +![](slides/Slide88.png) + +--- + +That's the end of this part. The next part is called ["MapReduce"](Part\ 12\ -\ MapReduce.md) + +![](slides/Slide0.png) + +--- \ No newline at end of file diff --git a/Part 12 - MapReduce.md b/Part 12 - MapReduce.md new file mode 100644 index 0000000..257bb5b --- /dev/null +++ b/Part 12 - MapReduce.md @@ -0,0 +1,103 @@ +![](slides/Slide0.png) + +Welcome to the Introduction to Cloudant course, an eighteen part video series that gives you an overview of the IBM Cloudant databases-as-a-service. + +![](slides/Slide1.png) + +--- + +This is part 12: "MapReduce". + +We've seen how a combination of the `_find` and `_index` endpoints allows queries to be performed on the contents of JSON documents, backed by secondary indexes to make queries scale as your application grows. + +In this part, we'll introduce another way of configure secondary indexes called MapReduce. + +![](slides/Slide89.png) + +--- + +MapReduce used to be the _only_ way to configure secondary indexes in CouchDB and is still a popular way of querying data from within the document body. + +To create a MapReduce index, you need to supply a JavaScript function wrapped in a special document called a _design document_. Design documents' `_id` fields begin with `_design/` e.g `_design/mydesigndoc`. + +When Cloudant receives the design document it will set up a background indexing task, passing each document from the database to your JavaScript function in turn: the key/values that are _emitted_ by your JavaScript function form the basis of the index that is persisted. + +Let's look at somee example JavaScript functions on the right of the screen. The function accepts one parameter - the document is is passed by the Cloudant indexer. Every time your function calls `emit` the parameters you pass form the key/value of the index. The first example emits a key of `doc.name`, so this is an index for lookups by the name field and there is nothing (null) for the value. + +The second example pre-processes the data prior to emitting. This is a useful way of tidying up strings, trimming whitespace, lower/uppercasing text, applying default values to missing data, or constraining values to certain ranges etc. + +The third example adds logic: only documents that are "published" make it to the index. This is equivalent to the _partial filter_ selector we saw with Cloudant Query. + +Indexes build asynchronously and do not function until they have built completely. Once built, they can be used for selection by key, lists of keys, ranges of keys and can also be used for _aggregation_ of data e.g. "find orders between two dates, and calculate the total value of the orders, grouped by month." + +![](slides/Slide90.png) + +--- + +There are four built-in reducers (or five if you count "none"). + +- `_count` - for counting things. +- `_sum` - for totalising values. +- `_stats` - for providing counts and totals suitable for calculating means, variances and standard deviations. +- `_approx_count_distinct` - for approximate counting of unique values of the key. + +![](slides/Slide91.png) + +--- + +The design document's MAP function is passed a "doc" - the function is called once per document in the database. Any key/value pairs "emit"ed from the MAP function create the index. + +The KEY is the thing (or things) you want to "select" on (e.g. date). + +The VALUE is the thing you need to report on (e.g. total sales). + +The Reducer is _sum, so that the VALUE is totalled for matching keys (e.g orders on the same date). + +Here's what defining a MapReduce looks like in the Cloudant dashboard. + +![](slides/Slide92.png) + +--- + +When the MapReduce view is built, it can be queried to see each KEY/VALUE pair stored in the index. + +Or, if the reducer is switched on result set can be grouped by the value of each key. Here we are totalising each day's sales. + +The view can be queried for individual keys (e.g. sales on a given date), all keys, or a range of keys (e.g. between two dates). + +![](slides/Slide93.png) + +--- + + +MapReduce views are built asynchronously and may take some time to be ready for large data sets. + +Here's some tips: + +- use `if` logic in your JavaScript to only include data that makes sense. e.g. only totalise completed orders. +- index keys don't have to be strings. A common pattern is to use array keys e.g an array of year, month, day. This allows query-time grouping by elements in the array e.g. orders by year, orders by year & month, orders by year & month & day - great for summary reports that allow the user to drill down into the detail. +- the value can by a string, number or sometimes a small object containing a sub-set of the document. The object can be used instead of adding `include_docs=true` which would also return the document's body in the result set. + +![](slides/Slide94.png) + +--- + + +MapReduce is a low-level means of defining indexes that allow the selection and aggregation of data. + +Use JavaScript logic to decide which data makes it to the index. Choose how the index is formed by emitting keys/values. + +Summarise data with the built-in reducers. +Produce concise reports from lots of data very efficiently. + +MapReduce is great for boilerplate queries that your application needs to do again and again. Not for one-off, adhoc-queries such as data exploration. + +![](slides/Slide95.png) + +--- + +That's the end of this part. The next part is called ["Dates"](Part\ 13\ -\ Dates.md) + +![](slides/Slide0.png) + +--- \ No newline at end of file diff --git a/Part 13 - Dates.md b/Part 13 - Dates.md new file mode 100644 index 0000000..640e26b --- /dev/null +++ b/Part 13 - Dates.md @@ -0,0 +1,54 @@ +![](slides/Slide0.png) + +Welcome to the Introduction to Cloudant course, an eighteen part video series that gives you an overview of the IBM Cloudant databases-as-a-service. + +![](slides/Slide1.png) + +--- + +This is part 13: "Dates". + +We saw earlier in this course that JSON only natively models strings, numbers, booleans, objects & arrays. A common data type is to store a date or date/time value in a database. Here's some ideas on how that can be achieved with Cloudant. + +![](slides/Slide96.png) + +--- + +The ISO-8601 string format for representing a time time consists of a `y-m-dTh:m:s.msTIMEZONE` year, month, day, a 'T' characetr, hour, minute, second, millisecond and timezone. + +I always recommend storing dates in the UTC timezone even if collecting data from different geographies. Date stored in this form can easily be transformed into the local timezone at the front end, but it's usually important to store each user's data in the "same units". + +This string format sorts into date/time order (because the most significant date units are at the front of the string) and can be easily parsed in MapReduce functions + +![](slides/Slide97.png) + +--- + +Another option is to store the number of milliseconds since 1970-01-01. This too is a standard, machine-readable way of representing a date & time. + +It too can be parsed in MapReduce functions and is very handy for comparing two dates: simply take one timestamp from another. + +![](slides/Slide98.png) + +--- + +The third option is to store each date/time component in separate fields. This is more verbose than the previous options but has one key advantage if you're using Cloudant Query. Because Cloudant Query can act up on data in the form it exists in the document itself, if you need to query on a single date component e.g the month, then you'd need that item broken out in the document. + +![](slides/Slide99.png) + +--- + +To summarise: + +There's no native date format in JSON so you can store dates and times how you like. ISO-8601 is compact, readable and sorts nicely. As does a timestmp (milliseconds since 1970). + +If you need to use Cloudant Query on one of the component parts, then that would need to be broken out explicitly in the document. + +![](slides/Slide100.png) + +--- +That's the end of this part. The next part is called ["Replication"](Part\ 14\ -\ Replication.md) + +![](slides/Slide0.png) + +--- \ No newline at end of file diff --git a/Part 14 - Replication.md b/Part 14 - Replication.md new file mode 100644 index 0000000..17823a0 --- /dev/null +++ b/Part 14 - Replication.md @@ -0,0 +1,103 @@ +![](slides/Slide0.png) + +Welcome to the Introduction to Cloudant course, an eighteen part video series that gives you an overview of the IBM Cloudant databases-as-a-service. + +![](slides/Slide1.png) + +--- + +This is part 14: "Replication". + + + +![](slides/Slide101.png) + +--- + +Replication is a core feature of Cloudant. It is the transfer of data from one database (the source) to another (the target). + +The source and target databases can reside on the same Cloudant service, or be geographically separated, for example a US Cloudant database replicating to one in Europe. + +The replication protocal is shared with Apache CouchDB, so replication is often used by enterprises copying data from a cloud-based database to one running CouchDB in their own premises. + +PouchDB, a JavaScript-based CouchDB clone that runs in Node.js stacks or in the web browser can also be used to replicate data to or from Cloudant. + +And there is the Cloudant Sync libraries, which allow native iOS or Android apps to sync data to and from a Cloudant service. + +Replication is a one-way operation from source to target which moves all data (deletions, conflicts, attachments as well as documents) and can be triggered in one of two ways: + +1. to run until all the data from the source has reached the target and then stop, or +2. the same as one but to keep the replication running continuously forever, transferring new data from the source to the target as it arrives. + +Replication can also be resumed from where it last left off. Cloudant keeps a note of "checkpoints" between replicating parties to allow the resumption of a pre-existing replication from its last known position. + +![](slides/Slide102.png) + +--- + +There is a replication tab in the Cloudant dashboard. A replication is started by specifying the source & target databases including authentication credentials, and whether this is a one off or continuous operation. + +A replication can also be given a name, which is handy for tracking which replication job is which. + +![](slides/Slide103.png) + +--- + +Now it's time for a practical exercise. + +From your Cloudant dashboard: + +- create a new database called `books2` +- start a continuous replication from `books` --> `books2` +- visit the books2 database to check that documents from books is now in books2 +- add a document to the books database - check the change makes it to the books2 database + +![](slides/Slide104.png) + +--- + +Replication can be used to move data from a Cloudant database to an on-premise CouchDB instance. The replication can be controlled bt the Cloudant or the CouchDB end i.e you can ask Cloudant to send its changes to CouchDB or you can ask CouchDB to pull the changes from Cloudant. Bear in mind that the replication controller must have network visibility of both HTTP APIs. + +PouchDB also speaks the same replication protocol so can be used to transfer data to and from PouchDB & Cloudant. It's most likely that PouchDB would be the replication controller in this case. + +![](slides/Slide105.png) + +--- + +PouchDB is commonly used to create _offline first_ apps which collect data even when not connected to the interent and then write data to Cloudant when they come back online, giving their users an _always on_ service. + +![](slides/Slide106.png) + +--- + +Bear in mind that replication may not always be required. If your application needs to store data and then write it to Cloudant later, then replication isn't strictly speaking required. All that is required is that data is stored on the device and bulk written to Cloudant when the connection is restored. + +![](slides/Slide107.png) + +--- + +As replication is a one way operation, if a master-master setup is required between two Cloudants in different regions, then two replications in opposite directions are required. + +Changes occurring on the London side are sent to Dallas, and vice versa. + +More complex topologies are possible, with data flowing in a ring around a set Cloudant services. + +![](slides/Slide108.png) + +--- + +To summarise: + +Cloudant replication is a mechanism for copying data from a source database to a target database. + +Replications can be one-off or continuous and can optionally be filtered with a JavaScript function or Cloudant Query selector and can be resumed from where a previous replication left of. + +![](slides/Slide109.png) + +--- + +That's the end of this part. The next part is called ["Partitioned Databases"](Part\ 15\ -\ Partitioned Databases.md) + +![](slides/Slide0.png) + +--- \ No newline at end of file diff --git a/Part 15 - Partitioned Databases.md b/Part 15 - Partitioned Databases.md new file mode 100644 index 0000000..ae500a4 --- /dev/null +++ b/Part 15 - Partitioned Databases.md @@ -0,0 +1,82 @@ +![](slides/Slide0.png) + +Welcome to the Introduction to Cloudant course, an eighteen part video series that gives you an overview of the IBM Cloudant databases-as-a-service. + +![](slides/Slide1.png) + +--- + +This is part 15: "Partitioned Database". + +We haven't discussed this yet, but Cloudant is a distributed database. There are many storage nodes that make up a Cloudant service and a database's documents are distributed across the nodes in groups called "shards". A single database is said to be "sharded" or divided into multiple pieces. + +In a normal Cloudant database, a document is allocated a shard algorithmically - effectively documents are distributed around the the shards randomly. + +In a partitioned database, you define which shard the documents are stored in by supplying a _partition key_. + +![](slides/Slide110.png) + +--- + +Partitioned databases are created with the same `PUT /` API call but with an additional query string parameter: `partitioned=true` + +In the first example the `products` database is created as a partitioned database, in the second example as a standard, un-partitioned database. + +![](slides/Slide111.png) + +--- + +When adding documents to a partitioned database you _must_ supply a document `_id` - (there are no auto-generated document `_ids`). A document `_id` has two parts, separated by a colon character: + +1. the partition key - a string which defines on _partition_ to store the document. +2. a document key - a string that uniquely identifies a document within the partition. + +In the top example a book is being added into the `book` partition of the products database. + +Then another document is being added into the `dvd` partition and a third into the `household` partition. + +![](slides/Slide112.png) + +--- + +The effect of this is that documents sharing a partition key reside in the same shard of the databases. Documents in the same partition are stored together in document key order. + +![](slides/Slide113.png) + +--- + +The advantge comes when retrieving data. We can direct Cloudant Queries, MapReduce requests and searches at _a single partition_. In this example, a Cloudant Query selector is being sent to a the `book` partition. This has the effect only exercising a fraction of the Cloudant infrastruture (only the shard that hosts the `book` partition are used, the rest of the cluster remains idle). + +This makes for faster query performance, cheaper query costs and better scalability. + +The key to great partitioned query performance is the choice of partition key: + +- It needs to be a value that repeats within your data set i.e. there a several items in the `book` partition. +- There needs to be many partitions. If there are only a handful of categories, then category is a bad choice of partition key. It needs to be something that has many values e.g. deviceId in an IoT application. +- It needs to match the queries that your application is making. If the most common use-case is searching within a product category, then partitioning by category may be a good fit. +- Avoid hot partitions - traffic should be evenly spread across your partitions. If your choice of partition key is likely to lead to much more traffic hitting a small number of partitions, then this might not be good choice of partition key. + +![](slides/Slide114.png) + +--- + +To summarise + + + +Partitioned atabases are created with the `partitioned=true` flag and documents have a two part it where the partition key and document key are joined by a colon character. + + +Documents in the same partition are stored in document key order in the same database shard. This can be used to make queries directed at a single partition run faster and more cheaply. + +Note it's still possible to query across partitions in a partitioned database. When creating a secondary index, you choose whether its purpose is for per-partition or global scope. + +![](slides/Slide115.png) + +--- + +That's the end of this part. The next part is called ["Cloudant Search"](Part\ 16\ -\ Cloudant Search.md) + +![](slides/Slide0.png) + +--- \ No newline at end of file diff --git a/Part 16 - Cloudant Search.md b/Part 16 - Cloudant Search.md new file mode 100644 index 0000000..942ea11 --- /dev/null +++ b/Part 16 - Cloudant Search.md @@ -0,0 +1,74 @@ +![](slides/Slide0.png) + +Welcome to the Introduction to Cloudant course, an eighteen part video series that gives you an overview of the IBM Cloudant databases-as-a-service. + +![](slides/Slide1.png) + +--- + +This is part 16: "Cloudant Search". + +There is another method of querying and indexing in Cloudant called Cloudant Search which we'll briefly explore in this part. + +![](slides/Slide116.png) + +--- + +Cloudant Search is built on another open-source project, [Apache Lucene](https://lucene.apache.org/), which powers the search capabilities of many products including ElasticSearch. + +It is primarly designed for _free text_ search, that is preprocessing text: removing case, punctuation, common noise words and trimming common language-specfic word endings e.g. farmer becomes farm and farms becomes farm. + +This text-processing is performed by a choice of _analyzers_ which pre-process your data prior into indexing, and at query time, prior to searching. + +It also allows some aggregation functionality using a technique called _faceting_ as we'll see. + +A Cloudant Search is indexed by supplying JavaScript function, not unlike MapReduce except this time the `emit` function is replaced by an `index` function which expects the name of the field, the data itself and some options. + +In this example, the document's `name` and `title` are indexed with default options. The category is nominated for `faceting` (the aggregation functionality) and the `isbn` is stored in the index but not indexed for search itself. Sometimes is more efficient to store some items in the index rather than doing `include_docs=true` at query time. + +![](slides/Slide117.png) + +--- + +Lucene has its own [query language](https://lucene.apache.org/core/2_9_4/queryparsersyntax.html) allowing you to create queries that match combinations of clauses with logic, fuzzy matching, ranges and term boosting. + +Here are some examples: + +1. Find documents whose title matches 'gastby' and whose author starts with 'fitz'. Notice the asterisk wildcard. +2. Find documents whose author is in the range austen --> dickens. This is an example of range querying on a string field. +3. Find documents whose price is between 0 and 100 AND whose year is in the 19th century or whose author matches "charles dickens". This shows how logic can be built into queries. + +Cloudant Search is useful, not only for free-text search, but for when you know which attributes you are going to search on, but the queries are varied, with different combinations of attributes each time. This flexibility is difficult to implement with fixed-order MapReduce indexes. + +![](slides/Slide118.png) + +--- + +Faceting is a form of aggregation. You nominate individual indexed fields for faceting at index-time and activate the aggregation with paramaters at query-time. + +It has two uses: + +1. Counting repeating values in the result set, such as the counts of products that belong to each category in a result set. or, +2. Counting numbers of items in numeric ranges, such as the product counts in each of the price ranges. + +Both of these forms of counts can be presented to a front-end user as a menans of further filtering an existing search, to narrow the scope of the search. e.g a customer searches for Fender, then clicks on Amps then on a price range under $500. All of this search and filtering can be powered by Cloudant Search. + +![](slides/Slide119.png) + +--- + +To summarise + +Cloudant Search indexes are defined with a supplied JavaScript function. They are built on Apache Lucene and are primarily used for free-text search matching, but the query language is useful for building flexible queries on a fixed set of indexed fields. It also has some powerful counting aggregations suitable for drilldown user interfaces. + +Note: Cloudant Search also powers 'type=text' Cloudant Query indexes, so a subset of its capabilities is surfaced using the `_find` API. + +![](slides/Slide120.png) + +--- + +That's the end of this part. The next part is called ["Geospatial"](Part\ 17\ -\ Geospatial.md) + +![](slides/Slide0.png) + +--- \ No newline at end of file diff --git a/Part 17 - Geospatial.md b/Part 17 - Geospatial.md new file mode 100644 index 0000000..c50e4fc --- /dev/null +++ b/Part 17 - Geospatial.md @@ -0,0 +1,40 @@ + +![](slides/Slide0.png) + +Welcome to the Introduction to Cloudant course, an eighteen part video series that gives you an overview of the IBM Cloudant databases-as-a-service. + +![](slides/Slide1.png) + +--- + +This is part 17: "Geospatial". + +The final means of querying data in Cloudant is using _geospatial indexes_. + +![](slides/Slide121.png) + +--- + +Geospatial indexes are unique to Cloudant - you won't find them in the CouchDB codebase. + +Data is stored as [GeoJSON](https://geojson.org/) in the Cloudant database to describe point, line, polygon, multi-point and multi-line and mulit-poloygon objects. Each object, as well as the geographic information, have optional _properties_: meta data about the object which are returned in the search results. + +Again an index is defined as a JavaScript function and then queries can be used ask questions of your collection of geographic features e.g. find me the nearest object to this point, find objects within this polygon, find objects along this path, find objects that intersect with this object. + +![](slides/Slide122.png) + +--- + +To summarise + +Cloudant Geo is something unique to the Cloudant service and is used to perform advanced geospatial queries against your databases of GeoJSON objects. It cannot be combined with other index types so is only of use to Geographic Information Systems or use-cases that have a purely geographic purpose. + +![](slides/Slide123.png) + +--- + +That's the end of this part. The next part is called ["Under the hood"](Part\ 18\ -\ Under\ the\ hood.md) + +![](slides/Slide0.png) + +--- \ No newline at end of file diff --git a/Part 18 - Under the hood.md b/Part 18 - Under the hood.md new file mode 100644 index 0000000..e1e6985 --- /dev/null +++ b/Part 18 - Under the hood.md @@ -0,0 +1,118 @@ +![](slides/Slide0.png) + +Welcome to the Introduction to Cloudant course, an eighteen part video series that gives you an overview of the IBM Cloudant databases-as-a-service. + +![](slides/Slide1.png) + +--- + +This is part 18: "Under the hood". + +Let's take a look at how a Cloudant service is organised: this overview applies to the Cloudant services that map to CouchDB 2 & 3. CouchDB 4 will be built on different technology. + +![](slides/Slide124.png) + +--- + +Cloudant is a distributed database with data stored around a cluster of storage nodes. Picture the Cloudant service as ring of nodes, in this case twelve. Every node can deal with incoming API calls and every node has responsiblity for storing some of the data: shards and associated secondary indexes of databases that exist in the cluster. + +When data is written to Cloudant one of the nodes in the ring will handle the request: it's job is to instruct three copies of the data to be stored in three storage nodes. Data is stored in triplicate in Cloudant, so each shard of a database is stored multiple times, often across a region's availability zones. + +When you make an API call to write data and get a reponse back, we have written the data to at least 2 of the 3 storage nodes. Data is flushed to disk - it isn't cached in memory to be flushed data. We consider that too risky and prone to data loss. + +![](slides/Slide125.png) + +--- + +When you create a database, a number of database _shards_ are created (16 by default) which are spread around the cluster. As there are 3 copies of each shard, that's 48 shard copies. + +You don't see any of this - it's handled for you transparently when you create a database. + +![](slides/Slide126.png) + +--- + +What happens if a node goes down or needs to be rebooted for maintenance? The rest of the cluster continues as normal. Most shards still have three copies of data but some will only have two. API calls will continue to work as normal, only two copies of the data will be written. + +![](slides/Slide127.png) + +--- + +Even if two nodes go down, most shards will still have three copies, some will have two and some will have one. Write will continue to work, although the HTTP response code will reflect that the _quorum_ of 2 node confirmations wasn't reached. + +![](slides/Slide128.png) + +--- + +It's the same store for reads. Service continues will a failed node. We can survive one failed node. + +![](slides/Slide129.png) + +--- + +Or more failed nodes. As long as there is a copy of each node in existance, the API should continue to function. + + +![](slides/Slide130.png) + +--- + +When a node returns, it will catch up any missed data from its peers and then return into service, handling API calls and answering queries for data. + +![](slides/Slide131.png) + +--- + +The nature of this configuration where any node can handle a request and data is distributed around nodes without the sort of locking you would see in a relational database, is that Cloudant exhibits _eventual consistency_: + +- Cloudant favours _availability_ over _consistency_: it would rather be up an answering API calls, than be down because it can't provide consistency guarantees. (a relational database is often configured in the opposite way: it operates in a consistent manner or not at all). +- The upshot of this as developer is that your app should not "read its writes" in a short period of time - there may be a small time window in which it's possible to see an older version of a document than the one you just updated. _Eventually_ the data will flow around the cluster and in most cases, the quorum mechanism will provide the illusion of consistency, but it's best not to rely on it. + +Note in CouchDB 4, and in Cloudant services based on that code version, a different consistency model will be employed. + +![](slides/Slide132.png) + +--- + +If your data model requires you to update a document over and over and a short time window, it's possible that multiple writes for the same revision number are accepted leading to a branch in the revision tree - known as a conflict. In this example revision "2" was modified in two diffent ways, causing 2 revision 3s. It's possible to tidy up conflicts programmatically, but they should be avoided as they can cause performance issues in extreme circumstances. + +Conflicts can also happen when using replication and a document is modified in different ways and then the conflicting revisions are merged in via replication. Cloudant does not throw away data in this scenario - a "winning" revision is chosen, but the non-winning revisions can be accessed and your application can resolve the conflict by electing a new winner, deleting unwanted revisions or any action you need. A conflict is not an error condition, its a side effect of having disconnected copies of data that can be modified without locking. + +![](slides/Slide133.png) + +--- + +To check a document for conflicts, simply add `?conflicts=true` to a fetch of a single document. Any conflicting revisions will be listed in the `_conflicts` array. + +![](slides/Slide134.png) + +--- + +Unwanted revisions can be removed using the normal DELETE operation, specifying the rev token of the revision you wish to delete. The bulk API is also good for removing conflicting revisions, even for removing multiple conflicts from the same document. + +![](slides/Slide135.png) + +--- +To summarise + +Cloudant is a distributed database which stores your database broken into multiple shards, with three copies of each shard spread around a ring of storage nodes. Cloudant is eventual consistent, favouring high availability over strong consistency. + +Avoid writing to the same document over and over so as not to create conflicts, although conflicts are sometime inevitable in replicating situations. + +Embrace eventual consistency - don't try and _make_ Cloudant consistent. + +Note: Cloudant products based of CouchDB 4 may have a different consistency model. + +![](slides/Slide136.png) + +--- + +That's the end of this course. For more information please consult our documentation and our blog + +![](slides/Slide199.png) + +--- + +![](slides/Slide0.png) + +--- \ No newline at end of file diff --git a/slides/Slide0.png b/slides/Slide0.png new file mode 100644 index 0000000..85f1890 Binary files /dev/null and b/slides/Slide0.png differ diff --git a/slides/Slide1.png b/slides/Slide1.png new file mode 100644 index 0000000..eb221ef Binary files /dev/null and b/slides/Slide1.png differ diff --git a/slides/Slide10.png b/slides/Slide10.png new file mode 100644 index 0000000..6840598 Binary files /dev/null and b/slides/Slide10.png differ diff --git a/slides/Slide100.png b/slides/Slide100.png new file mode 100644 index 0000000..8785a6f Binary files /dev/null and b/slides/Slide100.png differ diff --git a/slides/Slide101.png b/slides/Slide101.png new file mode 100644 index 0000000..252936c Binary files /dev/null and b/slides/Slide101.png differ diff --git a/slides/Slide102.png b/slides/Slide102.png new file mode 100644 index 0000000..a4d58eb Binary files /dev/null and b/slides/Slide102.png differ diff --git a/slides/Slide103.png b/slides/Slide103.png new file mode 100644 index 0000000..08553bb Binary files /dev/null and b/slides/Slide103.png differ diff --git a/slides/Slide104.png b/slides/Slide104.png new file mode 100644 index 0000000..ba52404 Binary files /dev/null and b/slides/Slide104.png differ diff --git a/slides/Slide105.png b/slides/Slide105.png new file mode 100644 index 0000000..8a3d262 Binary files /dev/null and b/slides/Slide105.png differ diff --git a/slides/Slide106.png b/slides/Slide106.png new file mode 100644 index 0000000..6eef41a Binary files /dev/null and b/slides/Slide106.png differ diff --git a/slides/Slide107.png b/slides/Slide107.png new file mode 100644 index 0000000..93da81d Binary files /dev/null and b/slides/Slide107.png differ diff --git a/slides/Slide108.png b/slides/Slide108.png new file mode 100644 index 0000000..0b28f67 Binary files /dev/null and b/slides/Slide108.png differ diff --git a/slides/Slide109.png b/slides/Slide109.png new file mode 100644 index 0000000..c857010 Binary files /dev/null and b/slides/Slide109.png differ diff --git a/slides/Slide11.png b/slides/Slide11.png new file mode 100644 index 0000000..bdb93c8 Binary files /dev/null and b/slides/Slide11.png differ diff --git a/slides/Slide110.png b/slides/Slide110.png new file mode 100644 index 0000000..336ffdf Binary files /dev/null and b/slides/Slide110.png differ diff --git a/slides/Slide111.png b/slides/Slide111.png new file mode 100644 index 0000000..058d845 Binary files /dev/null and b/slides/Slide111.png differ diff --git a/slides/Slide112.png b/slides/Slide112.png new file mode 100644 index 0000000..e3ce37a Binary files /dev/null and b/slides/Slide112.png differ diff --git a/slides/Slide113.png b/slides/Slide113.png new file mode 100644 index 0000000..1303021 Binary files /dev/null and b/slides/Slide113.png differ diff --git a/slides/Slide114.png b/slides/Slide114.png new file mode 100644 index 0000000..4237cca Binary files /dev/null and b/slides/Slide114.png differ diff --git a/slides/Slide115.png b/slides/Slide115.png new file mode 100644 index 0000000..a96fc42 Binary files /dev/null and b/slides/Slide115.png differ diff --git a/slides/Slide116.png b/slides/Slide116.png new file mode 100644 index 0000000..5d1c297 Binary files /dev/null and b/slides/Slide116.png differ diff --git a/slides/Slide117.png b/slides/Slide117.png new file mode 100644 index 0000000..80f2a92 Binary files /dev/null and b/slides/Slide117.png differ diff --git a/slides/Slide118.png b/slides/Slide118.png new file mode 100644 index 0000000..c410911 Binary files /dev/null and b/slides/Slide118.png differ diff --git a/slides/Slide119.png b/slides/Slide119.png new file mode 100644 index 0000000..7f6a92b Binary files /dev/null and b/slides/Slide119.png differ diff --git a/slides/Slide12.png b/slides/Slide12.png new file mode 100644 index 0000000..c30c8c2 Binary files /dev/null and b/slides/Slide12.png differ diff --git a/slides/Slide120.png b/slides/Slide120.png new file mode 100644 index 0000000..3cd6487 Binary files /dev/null and b/slides/Slide120.png differ diff --git a/slides/Slide121.png b/slides/Slide121.png new file mode 100644 index 0000000..7084fc6 Binary files /dev/null and b/slides/Slide121.png differ diff --git a/slides/Slide122.png b/slides/Slide122.png new file mode 100644 index 0000000..685720a Binary files /dev/null and b/slides/Slide122.png differ diff --git a/slides/Slide123.png b/slides/Slide123.png new file mode 100644 index 0000000..6c4e8b5 Binary files /dev/null and b/slides/Slide123.png differ diff --git a/slides/Slide124.png b/slides/Slide124.png new file mode 100644 index 0000000..4c639d3 Binary files /dev/null and b/slides/Slide124.png differ diff --git a/slides/Slide125.png b/slides/Slide125.png new file mode 100644 index 0000000..fba5776 Binary files /dev/null and b/slides/Slide125.png differ diff --git a/slides/Slide126.png b/slides/Slide126.png new file mode 100644 index 0000000..f85e2f5 Binary files /dev/null and b/slides/Slide126.png differ diff --git a/slides/Slide127.png b/slides/Slide127.png new file mode 100644 index 0000000..b21340e Binary files /dev/null and b/slides/Slide127.png differ diff --git a/slides/Slide128.png b/slides/Slide128.png new file mode 100644 index 0000000..f1a9716 Binary files /dev/null and b/slides/Slide128.png differ diff --git a/slides/Slide129.png b/slides/Slide129.png new file mode 100644 index 0000000..df6c1ea Binary files /dev/null and b/slides/Slide129.png differ diff --git a/slides/Slide13.png b/slides/Slide13.png new file mode 100644 index 0000000..fc7940a Binary files /dev/null and b/slides/Slide13.png differ diff --git a/slides/Slide130.png b/slides/Slide130.png new file mode 100644 index 0000000..4de95d2 Binary files /dev/null and b/slides/Slide130.png differ diff --git a/slides/Slide131.png b/slides/Slide131.png new file mode 100644 index 0000000..8e3f336 Binary files /dev/null and b/slides/Slide131.png differ diff --git a/slides/Slide132.png b/slides/Slide132.png new file mode 100644 index 0000000..e85eba8 Binary files /dev/null and b/slides/Slide132.png differ diff --git a/slides/Slide133.png b/slides/Slide133.png new file mode 100644 index 0000000..d288c5a Binary files /dev/null and b/slides/Slide133.png differ diff --git a/slides/Slide134.png b/slides/Slide134.png new file mode 100644 index 0000000..bbf904d Binary files /dev/null and b/slides/Slide134.png differ diff --git a/slides/Slide135.png b/slides/Slide135.png new file mode 100644 index 0000000..6400044 Binary files /dev/null and b/slides/Slide135.png differ diff --git a/slides/Slide136.png b/slides/Slide136.png new file mode 100644 index 0000000..132a604 Binary files /dev/null and b/slides/Slide136.png differ diff --git a/slides/Slide14.png b/slides/Slide14.png new file mode 100644 index 0000000..bc9ae3e Binary files /dev/null and b/slides/Slide14.png differ diff --git a/slides/Slide15.png b/slides/Slide15.png new file mode 100644 index 0000000..4d721d5 Binary files /dev/null and b/slides/Slide15.png differ diff --git a/slides/Slide16.png b/slides/Slide16.png new file mode 100644 index 0000000..30fbf22 Binary files /dev/null and b/slides/Slide16.png differ diff --git a/slides/Slide17.png b/slides/Slide17.png new file mode 100644 index 0000000..9720d9f Binary files /dev/null and b/slides/Slide17.png differ diff --git a/slides/Slide18.png b/slides/Slide18.png new file mode 100644 index 0000000..cea185a Binary files /dev/null and b/slides/Slide18.png differ diff --git a/slides/Slide19.png b/slides/Slide19.png new file mode 100644 index 0000000..df0ccae Binary files /dev/null and b/slides/Slide19.png differ diff --git a/slides/Slide199.png b/slides/Slide199.png new file mode 100644 index 0000000..851b328 Binary files /dev/null and b/slides/Slide199.png differ diff --git a/slides/Slide2.png b/slides/Slide2.png new file mode 100644 index 0000000..7b01ab4 Binary files /dev/null and b/slides/Slide2.png differ diff --git a/slides/Slide20.png b/slides/Slide20.png new file mode 100644 index 0000000..94cb543 Binary files /dev/null and b/slides/Slide20.png differ diff --git a/slides/Slide21.png b/slides/Slide21.png new file mode 100644 index 0000000..4bf1652 Binary files /dev/null and b/slides/Slide21.png differ diff --git a/slides/Slide22.png b/slides/Slide22.png new file mode 100644 index 0000000..1d960c1 Binary files /dev/null and b/slides/Slide22.png differ diff --git a/slides/Slide23.png b/slides/Slide23.png new file mode 100644 index 0000000..ad1f48b Binary files /dev/null and b/slides/Slide23.png differ diff --git a/slides/Slide24.png b/slides/Slide24.png new file mode 100644 index 0000000..26d5796 Binary files /dev/null and b/slides/Slide24.png differ diff --git a/slides/Slide25.png b/slides/Slide25.png new file mode 100644 index 0000000..27ac7ef Binary files /dev/null and b/slides/Slide25.png differ diff --git a/slides/Slide26.png b/slides/Slide26.png new file mode 100644 index 0000000..58f2b61 Binary files /dev/null and b/slides/Slide26.png differ diff --git a/slides/Slide27.png b/slides/Slide27.png new file mode 100644 index 0000000..4c8835e Binary files /dev/null and b/slides/Slide27.png differ diff --git a/slides/Slide28.png b/slides/Slide28.png new file mode 100644 index 0000000..564b185 Binary files /dev/null and b/slides/Slide28.png differ diff --git a/slides/Slide29.png b/slides/Slide29.png new file mode 100644 index 0000000..43b6554 Binary files /dev/null and b/slides/Slide29.png differ diff --git a/slides/Slide3.png b/slides/Slide3.png new file mode 100644 index 0000000..cc01294 Binary files /dev/null and b/slides/Slide3.png differ diff --git a/slides/Slide30.png b/slides/Slide30.png new file mode 100644 index 0000000..827c4c6 Binary files /dev/null and b/slides/Slide30.png differ diff --git a/slides/Slide31.png b/slides/Slide31.png new file mode 100644 index 0000000..1f5bce5 Binary files /dev/null and b/slides/Slide31.png differ diff --git a/slides/Slide32.png b/slides/Slide32.png new file mode 100644 index 0000000..b642709 Binary files /dev/null and b/slides/Slide32.png differ diff --git a/slides/Slide33.png b/slides/Slide33.png new file mode 100644 index 0000000..9dc481e Binary files /dev/null and b/slides/Slide33.png differ diff --git a/slides/Slide34.png b/slides/Slide34.png new file mode 100644 index 0000000..eea74c5 Binary files /dev/null and b/slides/Slide34.png differ diff --git a/slides/Slide35.png b/slides/Slide35.png new file mode 100644 index 0000000..37a6dfd Binary files /dev/null and b/slides/Slide35.png differ diff --git a/slides/Slide36.png b/slides/Slide36.png new file mode 100644 index 0000000..dbc6a54 Binary files /dev/null and b/slides/Slide36.png differ diff --git a/slides/Slide37.png b/slides/Slide37.png new file mode 100644 index 0000000..4826409 Binary files /dev/null and b/slides/Slide37.png differ diff --git a/slides/Slide38.png b/slides/Slide38.png new file mode 100644 index 0000000..d677883 Binary files /dev/null and b/slides/Slide38.png differ diff --git a/slides/Slide39.png b/slides/Slide39.png new file mode 100644 index 0000000..30eb2f3 Binary files /dev/null and b/slides/Slide39.png differ diff --git a/slides/Slide4.png b/slides/Slide4.png new file mode 100644 index 0000000..685a73a Binary files /dev/null and b/slides/Slide4.png differ diff --git a/slides/Slide40.png b/slides/Slide40.png new file mode 100644 index 0000000..2b9779a Binary files /dev/null and b/slides/Slide40.png differ diff --git a/slides/Slide41.png b/slides/Slide41.png new file mode 100644 index 0000000..2f7aab3 Binary files /dev/null and b/slides/Slide41.png differ diff --git a/slides/Slide42.png b/slides/Slide42.png new file mode 100644 index 0000000..f213c5a Binary files /dev/null and b/slides/Slide42.png differ diff --git a/slides/Slide43.png b/slides/Slide43.png new file mode 100644 index 0000000..13b6294 Binary files /dev/null and b/slides/Slide43.png differ diff --git a/slides/Slide44.png b/slides/Slide44.png new file mode 100644 index 0000000..0d83e76 Binary files /dev/null and b/slides/Slide44.png differ diff --git a/slides/Slide45.png b/slides/Slide45.png new file mode 100644 index 0000000..c98778a Binary files /dev/null and b/slides/Slide45.png differ diff --git a/slides/Slide46.png b/slides/Slide46.png new file mode 100644 index 0000000..a460357 Binary files /dev/null and b/slides/Slide46.png differ diff --git a/slides/Slide47.png b/slides/Slide47.png new file mode 100644 index 0000000..be4e144 Binary files /dev/null and b/slides/Slide47.png differ diff --git a/slides/Slide48.png b/slides/Slide48.png new file mode 100644 index 0000000..dde760b Binary files /dev/null and b/slides/Slide48.png differ diff --git a/slides/Slide49.png b/slides/Slide49.png new file mode 100644 index 0000000..a303352 Binary files /dev/null and b/slides/Slide49.png differ diff --git a/slides/Slide5.png b/slides/Slide5.png new file mode 100644 index 0000000..25a5eef Binary files /dev/null and b/slides/Slide5.png differ diff --git a/slides/Slide50.png b/slides/Slide50.png new file mode 100644 index 0000000..42885e9 Binary files /dev/null and b/slides/Slide50.png differ diff --git a/slides/Slide51.png b/slides/Slide51.png new file mode 100644 index 0000000..0b22bda Binary files /dev/null and b/slides/Slide51.png differ diff --git a/slides/Slide52.png b/slides/Slide52.png new file mode 100644 index 0000000..a870e78 Binary files /dev/null and b/slides/Slide52.png differ diff --git a/slides/Slide53.png b/slides/Slide53.png new file mode 100644 index 0000000..8cab529 Binary files /dev/null and b/slides/Slide53.png differ diff --git a/slides/Slide54.png b/slides/Slide54.png new file mode 100644 index 0000000..f78aa60 Binary files /dev/null and b/slides/Slide54.png differ diff --git a/slides/Slide55.png b/slides/Slide55.png new file mode 100644 index 0000000..ba82919 Binary files /dev/null and b/slides/Slide55.png differ diff --git a/slides/Slide56.png b/slides/Slide56.png new file mode 100644 index 0000000..8afea9b Binary files /dev/null and b/slides/Slide56.png differ diff --git a/slides/Slide57.png b/slides/Slide57.png new file mode 100644 index 0000000..77c622f Binary files /dev/null and b/slides/Slide57.png differ diff --git a/slides/Slide58.png b/slides/Slide58.png new file mode 100644 index 0000000..d4db4e2 Binary files /dev/null and b/slides/Slide58.png differ diff --git a/slides/Slide59.png b/slides/Slide59.png new file mode 100644 index 0000000..ada2664 Binary files /dev/null and b/slides/Slide59.png differ diff --git a/slides/Slide6.png b/slides/Slide6.png new file mode 100644 index 0000000..00ce11d Binary files /dev/null and b/slides/Slide6.png differ diff --git a/slides/Slide60.png b/slides/Slide60.png new file mode 100644 index 0000000..8772011 Binary files /dev/null and b/slides/Slide60.png differ diff --git a/slides/Slide61.png b/slides/Slide61.png new file mode 100644 index 0000000..df6925a Binary files /dev/null and b/slides/Slide61.png differ diff --git a/slides/Slide62.png b/slides/Slide62.png new file mode 100644 index 0000000..f5fb3ce Binary files /dev/null and b/slides/Slide62.png differ diff --git a/slides/Slide63.png b/slides/Slide63.png new file mode 100644 index 0000000..01084fc Binary files /dev/null and b/slides/Slide63.png differ diff --git a/slides/Slide64.png b/slides/Slide64.png new file mode 100644 index 0000000..7da44b8 Binary files /dev/null and b/slides/Slide64.png differ diff --git a/slides/Slide65.png b/slides/Slide65.png new file mode 100644 index 0000000..54a3362 Binary files /dev/null and b/slides/Slide65.png differ diff --git a/slides/Slide66.png b/slides/Slide66.png new file mode 100644 index 0000000..8da22b2 Binary files /dev/null and b/slides/Slide66.png differ diff --git a/slides/Slide67.png b/slides/Slide67.png new file mode 100644 index 0000000..1a86ad7 Binary files /dev/null and b/slides/Slide67.png differ diff --git a/slides/Slide68.png b/slides/Slide68.png new file mode 100644 index 0000000..9f73fc3 Binary files /dev/null and b/slides/Slide68.png differ diff --git a/slides/Slide69.png b/slides/Slide69.png new file mode 100644 index 0000000..eb7842f Binary files /dev/null and b/slides/Slide69.png differ diff --git a/slides/Slide7.png b/slides/Slide7.png new file mode 100644 index 0000000..4616e1d Binary files /dev/null and b/slides/Slide7.png differ diff --git a/slides/Slide70.png b/slides/Slide70.png new file mode 100644 index 0000000..131735c Binary files /dev/null and b/slides/Slide70.png differ diff --git a/slides/Slide71.png b/slides/Slide71.png new file mode 100644 index 0000000..7a30f76 Binary files /dev/null and b/slides/Slide71.png differ diff --git a/slides/Slide72.png b/slides/Slide72.png new file mode 100644 index 0000000..f1a1085 Binary files /dev/null and b/slides/Slide72.png differ diff --git a/slides/Slide73.png b/slides/Slide73.png new file mode 100644 index 0000000..9fc8f22 Binary files /dev/null and b/slides/Slide73.png differ diff --git a/slides/Slide74.png b/slides/Slide74.png new file mode 100644 index 0000000..b15e8c9 Binary files /dev/null and b/slides/Slide74.png differ diff --git a/slides/Slide75.png b/slides/Slide75.png new file mode 100644 index 0000000..fd69ef5 Binary files /dev/null and b/slides/Slide75.png differ diff --git a/slides/Slide76.png b/slides/Slide76.png new file mode 100644 index 0000000..16ed1a2 Binary files /dev/null and b/slides/Slide76.png differ diff --git a/slides/Slide77.png b/slides/Slide77.png new file mode 100644 index 0000000..d1ecaf9 Binary files /dev/null and b/slides/Slide77.png differ diff --git a/slides/Slide78.png b/slides/Slide78.png new file mode 100644 index 0000000..1c1b712 Binary files /dev/null and b/slides/Slide78.png differ diff --git a/slides/Slide79.png b/slides/Slide79.png new file mode 100644 index 0000000..c4c3df5 Binary files /dev/null and b/slides/Slide79.png differ diff --git a/slides/Slide8.png b/slides/Slide8.png new file mode 100644 index 0000000..f73ccda Binary files /dev/null and b/slides/Slide8.png differ diff --git a/slides/Slide80.png b/slides/Slide80.png new file mode 100644 index 0000000..5b0d19f Binary files /dev/null and b/slides/Slide80.png differ diff --git a/slides/Slide81.png b/slides/Slide81.png new file mode 100644 index 0000000..6e63c73 Binary files /dev/null and b/slides/Slide81.png differ diff --git a/slides/Slide82.png b/slides/Slide82.png new file mode 100644 index 0000000..cf314e5 Binary files /dev/null and b/slides/Slide82.png differ diff --git a/slides/Slide83.png b/slides/Slide83.png new file mode 100644 index 0000000..ccf38e3 Binary files /dev/null and b/slides/Slide83.png differ diff --git a/slides/Slide84.png b/slides/Slide84.png new file mode 100644 index 0000000..53a7e26 Binary files /dev/null and b/slides/Slide84.png differ diff --git a/slides/Slide85.png b/slides/Slide85.png new file mode 100644 index 0000000..2f38d80 Binary files /dev/null and b/slides/Slide85.png differ diff --git a/slides/Slide86.png b/slides/Slide86.png new file mode 100644 index 0000000..2b2bd1c Binary files /dev/null and b/slides/Slide86.png differ diff --git a/slides/Slide87.png b/slides/Slide87.png new file mode 100644 index 0000000..6391150 Binary files /dev/null and b/slides/Slide87.png differ diff --git a/slides/Slide88.png b/slides/Slide88.png new file mode 100644 index 0000000..956040d Binary files /dev/null and b/slides/Slide88.png differ diff --git a/slides/Slide89.png b/slides/Slide89.png new file mode 100644 index 0000000..cfc165f Binary files /dev/null and b/slides/Slide89.png differ diff --git a/slides/Slide9.png b/slides/Slide9.png new file mode 100644 index 0000000..b409271 Binary files /dev/null and b/slides/Slide9.png differ diff --git a/slides/Slide90.png b/slides/Slide90.png new file mode 100644 index 0000000..4542f31 Binary files /dev/null and b/slides/Slide90.png differ diff --git a/slides/Slide91.png b/slides/Slide91.png new file mode 100644 index 0000000..2f76380 Binary files /dev/null and b/slides/Slide91.png differ diff --git a/slides/Slide92.png b/slides/Slide92.png new file mode 100644 index 0000000..5c3d507 Binary files /dev/null and b/slides/Slide92.png differ diff --git a/slides/Slide93.png b/slides/Slide93.png new file mode 100644 index 0000000..892daf1 Binary files /dev/null and b/slides/Slide93.png differ diff --git a/slides/Slide94.png b/slides/Slide94.png new file mode 100644 index 0000000..c42bee2 Binary files /dev/null and b/slides/Slide94.png differ diff --git a/slides/Slide95.png b/slides/Slide95.png new file mode 100644 index 0000000..8f2e80f Binary files /dev/null and b/slides/Slide95.png differ diff --git a/slides/Slide96.png b/slides/Slide96.png new file mode 100644 index 0000000..6420689 Binary files /dev/null and b/slides/Slide96.png differ diff --git a/slides/Slide97.png b/slides/Slide97.png new file mode 100644 index 0000000..1ab0c21 Binary files /dev/null and b/slides/Slide97.png differ diff --git a/slides/Slide98.png b/slides/Slide98.png new file mode 100644 index 0000000..2109d6d Binary files /dev/null and b/slides/Slide98.png differ diff --git a/slides/Slide99.png b/slides/Slide99.png new file mode 100644 index 0000000..4f611db Binary files /dev/null and b/slides/Slide99.png differ