Service Installation

Create the WORKER AMI

Workers are the machines that run the indexes JVMs (the IndexTank-Engine part of the system). The Nebu component will manage the workers (create new ones, keep track of the existing ones). This is why it is a necessity to provide a way for Nebu to start new machines. The current codebase assumes it is running in AWS, so here's a guide to create an AWS image for new workers. But playing around with the WorkerManager subcomponent (worker_manager.py file) will allow you to migrate it to a different cloud provider.

For both the worker and the main server, you can create the image from scratch (more customizable) or use the AWS AMIs we provide and do some minor configuration.

From the AMI

1. Launch instance
   Launch a new EC2 instance using the public AMI "ami-ae73a8c7" ("IndexTank Worker Image").

2. Install access keys
   Install your public keys for login access in /home/indextank/.ssh/authorized_keys. Change the owner of that file to indextank and the mode to 600.

3. Install worker_manager keys
   Create a private / public key combination called “id_rsa_worker” and place the private key in the /home/indextank/.ssh directory.

4. Save new AMI
   Create an AMI from this instance and note down the AMI ID (you will use it for the worker_manager).

5. You can now terminate this instance.

From scratch

1. Create model instance
   Create a 64-bit transient storage ubuntu AWS instance (for example Canonical ami-4fa37426).

2. Create access
   Connect to it as root and create a new user “indextank”. Install public keys in /home/indextank/.ssh/authorized_keys, change the owner of that file to indextank and the mode to 600.

3. Install dependencies
   Manually install sun java 6 jre, and run apt-get install daemontools ec2-ami-tools ec2-api-tools git python-simplejson (you will need to add multiverse to the /etc/apt/sources.list file and run “apt-get update” to get the ec2 tools).

4. Install worker_manager keys
   Create a private / public key combination called “id_rsa_worker” and place the private key in the /home/indextank/.ssh directory.

5. Get IndexTank-Service
   Clone the github.com/linkedin/indextank-services repository and move the nebu directory to the indextank home. The rest of the repository is not needed and can be removed.

6. Configure the controller service
   Copy the files found in /home/indextank/nebu/sample-config to the corresponding destinations to configure svc, rc.local and init.d.

7. Save new AMI
   Create an AMI from this instance and note down the AMI ID.

8. You can now terminate this instance.

Create the main server

The main server is the instance that will run all services: the API, the frontend webapp, the backoffice webapp, the log storage and the nebu components.

For both the worker and the main server, you can create the image from scratch (more customizable) or use the AWS AMIs we provide and do some minor configuration.

From the AMI

1. Launch instance
   Launch a new EC2 instance using the public AMI "ami-1a73a873" ("IndexTank Service Image").

2. Install access keys
   Install your public keys for login access in /home/indextank/.ssh/authorized_keys. Change the owner of that file to indextank and the mode to 600.

3. Create mysql user
   Login to mysql with root/root and an add a user for the "indextank" database (grant all on indextank.* to ‘’@’localhost’ identified by ‘’).

4. Edit settings.py (django)
   Edit the settings.py file in storefront, backoffice, api and nebu (all four file must contain the same following values):

• Change the database user and password to match the one you created.
• Change the email host, port, user and password.
• Change the APIKEY_KEY, ONETIMEPASS_KEY and FORGOTPASS_KEY values (you can get new values from http://www.grc.com/passwords.htm).

5. Create and populate database
   Create the database schema (cd /home/indextank/storefront; python manage.py syncdb). When asked if you want to create a superuser, answer “no”.

6. Manually populate the service table
   Add rows to the storefront_service table in the database. You need a row for “storage” and a row for “recovery” services. It tells nebu where to find the LogStorage processes. Your table should have 2 records:

• id: 1; name: 'storage'; type: 'master'; host: [YOUR PRIVATE DNS]; port: 15000; timestamp: now()
• id: 2; name: 'recovery'; type: 'dns'; host: [YOUR PRIVATE DNS]; port: 15100; timestamp: now()

7. Start the services
   Run /etc/rc.local then touch /data/master and then /etc/rc.local again.

8. Provisionally point the hosts file
   Add the following line to the hosts file in your workstation, replacing the IP address of the EC2 instance (remember to remove this later, it’s only for testing):
   <IP_ADDRESS> indextank.com www.indextank.com backoffice.indextank.com api.indextank.com dbajo.api.indextank.com
   Add the same line in the servers /etc/hosts file (the storefront communicates with the API).
   NOTE: The hosts files configuration is intended for testing purposes. With this configuration you can only create ONE fully functional account.
   When you have a domain of your own that you can point to your server you should do the following:

• Make sure the wildcard *.yourdomain.com (it will actually have up to 2 levels) is pointing to your server.
• Replace, in /etc/nginx/sites-available/api-uwsgi.conf (line 9), backoffice_fcgi.conf (lines 3) and storefront_fcgi.conf (lines 3 and 33) files, “indextank.com” (just that part) with your own domain name.
• Replace in api/models.py, storefront/api_linked_models.py and backoffice/api_linked_models.py (lines 239 and 242) “indextank.com” (just that part) with your domain name.

From scratch

1. Clone and build the engine
   Clone the github.com/linkedin/indextank-engine repository and compile the jar files (mvn package assembly:assembly). Rename the target jar files:

• indextank-engine-1.0.0.jar → indextank-trunk.jar
• indextank-engine-1.0.0-jar-with-dependencies.jar → indextank-trunk-deps.jar

2. Create instance
   Create a 64-bit transient storage ubuntu AWS instance (for example Canonical ami-4fa37426).

3. Install the worker key
   Connect to it as root and create a new user “indextank”. Install the public key “id_rsa_worker.pub" (and any other you may want to use) in /home/indextank/.ssh/authorized_keys, change the owner of that file to indextank and the mode to 600.

4. Install dependencies
   Manually install django 1.2.7 and sun java 6 jre, and run apt-get install daemontools apache2-utils mysql-server-5.1 git nginx uwsgi uwsgi-plugin-python python-mysqldb python-anyjson python-flup python-pip python-boto.

5. Manually populate the service table
   Clone the github.com/linkedin/indextank-service repository to the indextank dir, so that the api, storefront, nebu and storage directories are directly in /home/indextank (for example, you should have the /home/indextank/api/start_webapp.sh file).

6. Clone service
   Clone the github.com/flaptor/indextank-py repository to the /home/indextank/api/lib directory, so that you have the file /home/indextank/api/lib/indextank/client.py file.

7. Create service symlinks
   You also need this resource in storefront and nebu. Create symbolic links in both folders: cd /home/indextank/nebu/lib
ln -s ../../api/lib/indextank/ indextank
cd /home/indextank/storefront/lib
ln -s ../../api/lib/indextank/ indextank

8. Copy server configurations
   Copy the files found in /home/indextank/sample-config and /home/indextank/server_configurations to the corresponding destinations to configure svc, nginx, rc.local and hosts.

9. Configure SSL
   Add your SSL keys to nginx.

10. Configure worker_manager to use your AWS account and worker AMI
    Edit the file /home/indextank/nebu/amazon_credential.py and set your AWS Access Key ID and Secret Access Key.
    Edit the file /home/indextank/nebu/worker_manager.py and set the AMI ID for your worker image.

11. Create folder structure and install PIP
    Run the following commands:
    ln -s /mnt /data
rm -fr /mnt/lost+found
mkdir -p /data/pids /data/bkp /data/logs/api-rot
echo PROD >/data/env.name
mkdir /var/nginx; chown indextank:indextank /var/nginx
pip install indextank
chown -R indextank:indextank ~indextank /mnt
touch /data/master
touch /data/safe_to_read
su - indextank
cp -r gen-py/flaptor storefront
cp -r storefront/lib/indextank backoffice/lib
cp -r storefront/lib/indextank api/lib
cp -r storefront/flaptor backoffice
cp -r storefront/flaptor api
mkdir nebu/indextank_lib
mkdir storage/lib

12. Set the engine jar files
    Put the two index-engine jar files in /home/indextank/nebu/indextank_lib and storage/lib.

13. Create database
    Create a mysql database “indextank” and an add a user (create database indextank; grant all on indextank.* to ‘’@’localhost’ identified by ‘’).

14. Edit settings.py
    Edit the settings.py file in storefront, backoffice, api and nebu (all four file must contain the same following values):

• Change the database user and password to match the one you created.
• Change the email host, port, user and password.
• Change the APIKEY_KEY, ONETIMEPASS_KEY and FORGOTPASS_KEY values (you can get new values from http://www.grc.com/passwords.htm).

15. Create schema and populate database
    Create the database schema (cd /home/indextank/storefront; python manage.py syncdb). When asked if you want to create a superuser, answer “no”.

16. Manually populate the service table
    Add rows to the storefront_service table in the database. You need a row for “storage” and a row for “recovery” services. It tells nebu where to find the LogStorage processes. Your table should have 2 records:

• id: 1; name: 'storage'; type: 'master'; host: [YOUR PRIVATE DNS]; port: 15000; timestamp: now()
• id: 2; name: 'recovery'; type: 'dns'; host: [YOUR PRIVATE DNS]; port: 15100; timestamp: now()

17. Start nebu services
    Login as root and start nebu: nohup /usr/bin/svscanboot &

18. Start the webapps
    Logout of the instance and login as indextank (if you use su - indextank the ulimits are ignored) to start the main services:
    cd storefront; ./start_webapp
cd ../backoffice; ./start_webapp
cd ../api; ./start_webapp

19. Provisionally point the hosts file
    Add the following line to the hosts file in your workstation, replacing the IP address of the EC2 instance (remember to remove this later, it’s only for testing):
    <IP_ADDRESS> indextank.com www.indextank.com backoffice.indextank.com api.indextank.com dbajo.api.indextank.com
    Add the same line in the servers /etc/hosts file (the storefront communicates with the API).
    NOTE: The hosts files configuration is intended for testing purposes. With this configuration you can only create ONE fully functional account.
    When you have a domain of your own that you can point to your server you should do the following:

• Make sure the wildcard *.yourdomain.com (it will actually have up to 2 levels) is pointing to your server.
• Replace, in /etc/nginx/sites-available/api-uwsgi.conf (line 9), backoffice_fcgi.conf (lines 3) and storefront_fcgi.conf (lines 3 and 33) files, “indextank.com” (just that part) with your own domain name.
• Replace in api/models.py, storefront/api_linked_models.py and backoffice/api_linked_models.py (lines 239 and 242) “indextank.com” (just that part) with your domain name.

20. Test the frontend
    Browse to www.indextank.com/get-started, enter the email address of the admin user and click on “SIGN UP”.

21. Set the staff user
    Log into the database (you can now do it via django: cd /home/indextank/storefront; python manage.py dbshell) and run the following command:
    update auth_user set is_staff = 1 where id = 1;
    This will allow the admin user to log into backend.indextank.com.