forked from thinkaurelius/titan
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathcassandra.txt
225 lines (149 loc) · 14.2 KB
/
cassandra.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
[[cassandra]]
Cassandra
---------
[.tss-center.tss-width-250]
image:http://cassandra.apache.org/media/img/cassandra_logo.png[link="http://cassandra.apache.org/",width=250]
[quote,'http://cassandra.apache.org/[Apache Cassandra Homepage]']
The Apache Cassandra database is the right choice when you need
scalability and high availability without compromising
performance. Linear scalability and proven fault-tolerance on
commodity hardware or cloud infrastructure make it the perfect
platform for mission-critical data. Cassandra's support for
replicating across multiple datacenters is best-in-class, providing
lower latency for your users and the peace of mind of knowing that you
can survive regional outages. The largest known Cassandra cluster has
over 300 TB of data in over 400 machines.
The following sections outline the various ways in which Titan can be used in concert with Cassandra.
[[cassandra-local-server-mode]]
Local Server Mode
~~~~~~~~~~~~~~~~~
image:titan-modes-local.png[]
Cassandra can be run as a standalone database on the same local host as Titan and the end-user application. In this model, Titan and Cassandra communicate with one another via a `localhost` socket. Running Titan over Cassandra requires the following setup steps:
. http://cassandra.apache.org/download/[Download Cassandra], unpack it, and set filesystem paths in `conf/cassandra.yaml` and `conf/log4j-server.properties`
. Start Cassandra by invoking `bin/cassandra -f` on the command line in the directory where Cassandra was unpacked. Read output to check that Cassandra started successfully.
Now, you can create a Cassandra TitanGraph as follows::
[source,java]
TitanGraph g = TitanFactory.build()
.set("storage.backend","cassandra")
.set("storage.hostname","127.0.0.1")
.open();
In the Gremlin shell, you can not define the type of the variables `conf` and `g`. Therefore, simply leave off the type declaration.
Remote Server Mode
~~~~~~~~~~~~~~~~~~
image:titan-modes-distributed.png[]
When the graph needs to scale beyond the confines of a single machine, then Cassandra and Titan are logically separated into different machines. In this model, the Cassandra cluster maintains the graph representation and any number of Titan instances maintain socket-based read/write access to the Cassandra cluster. The end-user application can directly interact with Titan within the same JVM as Titan.
For example, suppose we have a running Cassandra cluster where one of the machines has the IP address 77.77.77.77, then connecting Titan with the cluster is accomplished as follows (comma separate IP addresses to reference more than one machine):
[source,java]
TitanGraph g = TitanFactory.build()
.set("storage.backend","cassandra")
.set("storage.hostname","77.77.77.77")
.open();
In the Gremlin shell, you can not define the type of the variables `conf` and `g`. Therefore, simply leave off the type declaration.
Remote Server Mode with Rexster
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
image:titan-modes-rexster.png[]
Rexster can be wrapped around each Titan instance defined in the previous subsection. In this way, the end-user application need not be a Java-based application as it can communicate with Rexster over REST. This type of deployment is great for polyglot architectures where various components written in different languages need to reference and compute on the graph.
----
http://rexster.titan.machine1/mygraph/vertices/1
http://rexster.titan.machine2/mygraph/tp/gremlin?script=g.v(1).out('follows').out('created')
----
In this case, each Rexster server would be configured to connect to the Cassandra cluster. The following shows the graph specific fragment of the Rexster configuration. Refer to <<server>> for a complete example.
[source,xml]
<graph>
<graph-name>mygraph</graph-name>
<graph-type>com.thinkaurelius.titan.tinkerpop.rexster.TitanGraphConfiguration</graph-type>
<graph-location></graph-location>
<graph-read-only>false</graph-read-only>
<properties>
<storage.backend>cassandra</storage.backend>
<storage.hostname>77.77.77.77</storage.hostname>
</properties>
<extensions>
<allows>
<allow>tp:gremlin</allow>
</allows>
</extensions>
</graph>
Titan Embedded Mode
~~~~~~~~~~~~~~~~~~~
image:titan-modes-embedded.png[]
Finally, Cassandra can be embedded in Titan, which means, that Titan and Cassandra run in the same JVM and communicate via in process calls rather than over the network. This removes the (de)serialization and network protocol overhead and can therefore lead to considerable performance improvements. In this deployment mode, Titan internally starts a cassandra daemon and Titan no longer connects to an existing cluster but is its own cluster.
To use Titan in embedded mode, simply configure `embeddedcassandra` as the storage backend. The configuration options listed below also apply to embedded Cassandra. In creating a Titan cluster, ensure that the individual nodes can discover each other via the Gossip protocol, so setup a Titan-with-Cassandra-embedded cluster much like you would a stand alone Cassandra cluster. When running Titan in embedded mode, the Cassandra yaml file is configured using the additional configuration option `storage.conf-file`, which specifies the yaml file as a full url, e.g. `storage.conf-file = file:///home/cassandra.yaml`.
When running a cluster with Titan and Cassandra embedded, it is advisable to expose Titan through the Rexster server so that applications can remotely connect to the Titan graph database and execute queries.
Note, that running Titan with Cassandra embedded requires GC tuning. While embedded Cassandra can provide lower latency query answering, its GC behavior under load is less predictable.
Cassandra Specific Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Refer to <<titan-config-ref>> for a complete listing of all Cassandra specific configuration options in addition to the general Titan configuration options.
When configuring Cassandra it is recommended to consider the following Cassandra specific configuration options:
* *read-consistency-level*: Cassandra consistency level for read operations
* *write-consistency-level*: Cassandra consistency level for write operations
* *replication-factor*: The replication factor to use. The higher the replication factor, the more robust the graph database is to machine failure at the expense of data duplication. *The default value should be overwritten for production system to ensure robustness. A value of 3 is recommended.* This replication factor can only be set when the keyspace is initially created. **On an existing keyspace, this value is ignored.**
* *thrift.frame_size_mb*: The maximum frame size to be used by thrift for transport. Increase this value when retrieving very large result sets. **Only applicable when storage.backend=cassandrathrift**
* *keyspace*: The name of the keyspace to store the Titan graph in. Allows multiple Titan graphs to co-exist in the same Cassandra cluster.
For more information on Cassandra consistency levels and acceptable values, please refer to the http://wiki.apache.org/cassandra/API10[Cassandra Thrift API]. In general, higher levels are more consistent and robust but have higher latency.
Global Graph Operations
~~~~~~~~~~~~~~~~~~~~~~~
Titan over Cassandra supports global vertex and edge iteration. However, note that all these vertices and/or edges will be loaded into memory which can cause `OutOfMemoryException`. Use <<hadoop,Titan-Hadoop>> to iterate over all vertices or edges in large graphs effectively.
Deploying on Amazon EC2
~~~~~~~~~~~~~~~~~~~~~~~
[.tss-center.tss-width-250]
image:http://cdn001.practicalclouds.com/user-content/1_Dave%20McCormick//logos/Amazon%20AWS%20plus%20EC2%20logo_scaled.png[link="http://aws.amazon.com/ec2/"]
[quote,'http://aws.amazon.com/ec2/[Amazon EC2]']
Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers.
Follow these steps to setup a Cassandra cluster on EC2 and deploy Titan over Cassandra. To follow these instructions, you need an Amazon AWS account with established authentication credentials and some basic knowledge of AWS and EC2.
Setup Cassandra Cluster
^^^^^^^^^^^^^^^^^^^^^^^
These instructions for configuring and launching the DataStax Cassandra Community Edition AMI are based on the http://www.datastax.com/docs/datastax_enterprise2.0/ami/install_dse_ami[DataStax AMI Docs] and focus on aspects relevant for a Titan deployment.
Setting up Security Group
^^^^^^^^^^^^^^^^^^^^^^^^^
* Navigate to the EC2 Console Dashboard, then click on "Security Groups" under "Network & Security".
* Create a new security group. Click Inbound. Set the "Create a new rule" dropdown menu to "Custom TCP rule". Add a rule for port 22 from source 0.0.0.0/0. Add a rule for ports 1024-65535 from the security group members. If you don't want to open all unprivileged ports among security group members, then at least open 7000, 7199, and 9160 among security group members. Tip: the "Source" dropdown will autocomplete security group identifiers once "sg" is typed in the box, so you needn't have the exact value ready beforehand.
Launch DataStax Cassandra AMI
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* "Launch the https://aws.amazon.com/amis/datastax-auto-clustering-ami-2-2[DataStax AMI] in your desired zone
* On the Instance Details page of the Request Instances Wizard, set "Number of Instances" to your desired number of Cassandra nodes. Set "Instance Type" to at least m1.large. We recommend m1.large.
* On the Advanced Instance Options page of the Request Instances Wizard, set the "as text" radio button under "User Data", then fill this into the text box:
----
--clustername [cassandra-cluster-name]
--totalnodes [number-of-instances]
--version community
--opscenter no
----
[number-of-instances] in this configuration must match the number of EC2 instances configured on the previous wizard page. [cassandra-cluster-name] can be any string used for identification. For example:
----
--clustername titan
--totalnodes 4
--version community
--opscenter no
----
* On the Tags page of the Request Instances Wizard you can apply any desired configurations. These tags exist only at the EC2 administrative level and have no effect on the Cassandra daemons' configuration or operation.
* On the Create Key Pair page of the Request Instances Wizard, either select an existing key pair or create a new one. The PEM file containing the private half of the selected key pair will be required to connect to these instances.
* On the Configure Firewall page of the Request Instances Wizard, select the security group created earlier.
* Review and launch instances on the final wizard page.
Verify Successful Instance Launch
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* SSH into any Cassandra instance node: `ssh -i [your-private-key].pem ubuntu@[public-dns-name-of-any-cassandra-instance]`
* Run the Cassandra nodetool `nodetool -h 127.0.0.1 ring` to inspect the state of the Cassandra token ring. You should see as many nodes in this command's output as instances launched in the previous steps.
Note, that the AMI takes a few minutes to configure each instance. A shell prompt will appear upon successful configuration when you SSH into the instance.
Launch Titan Instances
^^^^^^^^^^^^^^^^^^^^^^
Launch additional EC2 instances to run Titan which are either configured in Remote Server Mode or Remote Server Mode with Rexster as described above. You only need to note the IP address of one of the Cassandra cluster instances and configure it as the host name. The particular EC2 instance to run and the particular configuration depends on your use case.
Example Titan Instance on Amazon Linux AMI
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* Launch the http://aws.amazon.com/amazon-linux-ami[Amazon Linux AMI] in the same zone of the Cassandra cluster. Choose your desired EC2 instance type depending on the amount of resources you need. Use the default configuration options and select the same Key Pair and Security Group as for the Cassandra cluster configured in the previous step.
* SSH into the newly created instance via `ssh -i [your-private-key].pem ec2-user@[public-dns-name-of-the-instance]`. You may have to wait a little for the instance to launch.
* https://github.com/thinkaurelius/titan/wiki/Downloads[Download] the current Titan distribution with `wget` and unpack the archive locally to the home directory. Start the gremlin shell to verify that Titan runs successfully. For more information on how to unpack Titan and start the gremlin shell, please refer to the <<getting-started, Getting Started guide>>
* Create a configuration file with `vi titan.properties` and add the following lines::
storage.backend = cassandra
storage.hostname = [IP-address-of-one-Cassandra-EC2-instance]
You may add additional configuration options found on this page or in <<titan-config-ref>>.
* Start the gremlin shell again and type the following::
gremlin> g = TitanFactory.open('titan.properties')
==>titangraph[cassandra:[IP-address-of-one-Cassandra-EC2-instance]]
You have successfully connected this Titan instance to the Cassandra cluster and can start to operate on the graph.
Connect to Cassandra cluster in EC2 from outside EC2
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Opening the usual Cassandra ports (9160, 7000, 7199) in the security group is not enough, because the Cassandra nodes by default broadcast their ec2-internal IPs, and not their public-facing IPs.
The resulting behavior is that you can open a Titan graph on the cluster by connecting to port 9160 on any Cassandra node, but all requests to that graph time out. This is because Cassandra is telling the client to connect to an unreachable IP.
To fix this, set the "broadcast-address" property for each instance in /etc/cassandra/cassandra.yaml to its public-facing IP, and restart the instance. Do this for all nodes in the cluster. Once the cluster comes back, nodetool reports the correct public-facing IPs to which connections from the local machine are allowed.
Changing the "broadcast-address" property allows you to connect to the cluster from outside ec2, but it might also mean that traffic originating within ec2 will have to round-trip to the internet and back before it gets to the cluster. So, this approach is only useful for development and testing.