This document describes how to bootstrap into the webapplication. The documentation is updated for Ubuntu 18.04 LTS and Debian 10 (Buster).
The following other libraries and programs are needed
-
Ruby > 2.3
-
Apache 2.4
-
MySQL > 8.0.4 or MariaDB 10.x
-
Git
-
Passenger for Apache 2
-
rails
-
ruby-bundler
-
libmysqlclient-dev
-
imagemagick
-
libmagickwand-dev
-
Java RE 11+
NOTE From Muscat 6.1 MySQL *8.0.4 is required* for autocomplete and comments to properly work, as the REGEX library was changed. MariaDB 10.x, as provided by Debian 10 (Buster) also seems to work properly.
Depending on your system, a package for Mysql 8 may not exist. In this case use the official package provided by Oracle dev.mysql.com/downloads/repo/apt/
NOTE: Download the latest version or the GPG keys may have been expired! Copy the file from the link provided in that page, there is no need for an account, follow the “No thanks, just start my download” link. In this case it points to dev.mysql.com/get/mysql-apt-config_0.8.15-1_all.deb
wget https://dev.mysql.com/get/mysql-apt-config_0.8.15-1_all.deb sudo dpkg -i mysql-apt-config_0.8.15-1_all.deb
(obviously subsitute with the exact file name in the link). dpk will prompt a screen to configure tha package, select option to configure the server version and select 8, then exit.
Then update and install mysql:
sudo apt-get update sudo apt-get install mysql-server
When prompted, select *use legacy passwords* as not all connectors for the moment work with the new system. This will be updated in the future. Also install the newest versions of the connector libraries:
sudo apt-get install libmysqlclient21 libmysqlclient-dev
If you are using Debian 10 (Buster), the default MariaDB server (mariadb-server, that pulls 10.3) and clients (ruby-mysql2) are just fine.
Muscat 6 requires Ruby 2.3 or higher and Bundler 2.1.4 or higher. Rails (5.2) is installed with the correct version with bundle.
Recent Ubuntu releases (from 18.4 LTS) and Debian 10 (Buster) provide come Ruby 2.5 by default so no actions are necessary. On Debian Buster, Bundler 2.1.4 is provided as a backport. Versions for Passenger, Java (as required for Solr) are also fine.
sudo apt-get update sudo apt-get install mysql-server libmysqlclient-dev # If mysql was not installed separately sudo apt-get install apache2 git ruby ruby-dev gcc zlib1g-dev libxml2-dev imagemagick libmagickcore-6.q16-dev libmagickwand-6.q16-dev openjdk-11-jdk-headless passenger make libapache2-mod-passenger libsqlite3-dev g++ default-libmysqld-dev sudo a2enmod passenger
Passenger also wants to compile a binary module, this is optional but will enable it
sudo mkdir /var/www/.passenger sudo chown -R www-data:www-data /var/www/.passenger
Get the sources if necessary (github.com/rism-ch/muscat and github.com/rism-ch/muscat-guidelines):
git clone https://github.com/rism-ch/muscat.git --recursive sudo gem install bundler # We are compatibile with 2.1 now sudo bundle install #--deployment # deployment is only for the production system
Alternatively, you can just download and use the stable release tarballs found in github.com/rism-ch/muscat/releases.
Install base configuration:
cd muscat/config cp sample_database.yml database.yml cp sample_application.rb application.rb
Install the base css:
cd muscat/config cp muscat-custom-sample.scss ../vendor/assets/stylesheets/muscat-custom.scss
Make sure the directory is owned by the right user (www-data on Ubuntu or Debian)
sudo chown -R www-data:www-data muscat/
Set up databases access:
sudo mysql # Or log in with a user that has user creation privileges
Create the user, substitute with an appropriate user and password!
CREATE DATABASE muscat_development CHARACTER SET utf8 COLLATE utf8_general_ci; CREATE USER 'rism'@'localhost'; ALTER USER 'rism'@'localhost' IDENTIFIED WITH mysql_native_password BY 'password'; GRANT ALL ON muscat_development.* TO 'rism'@'localhost';
Remember username and pass should be the same as in database.yml. Then, migrate the database (that us, update the structure):
bundle exec rake db:migrate # development
NOTE: in a production environment all the tasks must declare RAILS_ENV=production to run. For local development this is not necessary
sudo RAILS_ENV=production bundle exec rake db:migrate ## on a production system
Add basic dataset:
bundle exec rake db:seed
All the secret keys are now stored in the rails 5.2 credentials file, which is not included in the repo. A blank on must be created with:
rails credentials:edit
This will automatically create credentials.yml.enc and the master key, along with a cookie secret key. If you are not familiarised with this Rails 5.2 feature, this guide is very didactic: youtu.be/BHgvPPr2nLE.
Default configuration for Muscat requires the user to check the recaptcha challenge in order to download bibliographic records. So you will have to create a (free) Google Recaptcha account for your site (www.google.com/recaptcha/about/) in order to get the site_key and the secret_key.
The user and password are needed for background search and export, and should be a muscat user with minimum guest access. So, to use the other services on Muscat, such as history, add to this file:
recaptcha: site_key: 'xx' secret_key: 'xx' export: user: "xx" password: "xx"
Remember that they key must be readable from the user running apache, so check the permissions and change the user is needed:
chown www-data:www-data config/master.key
Default (development) startup:
bundle exec rake sunspot:solr:start rails s -e development
Try to open $IP_ADDRESS:3000/
For startup in production mode see “Starting Daemons in Production Mode”
For refreshing an installation in production mode (with sudo, as root or sudo -u www-data depending on the permissions on the installation dir):
bundle exec rake RAILS_ENV=production assets:clean bundle exec rake RAILS_ENV=production assets:precompile
A default administrative user has been created as part of the installation process. To log in, go to ‘ip:3000/admin` and log in with the following credentials:
username: [email protected] password: password
It is advised that you delete this account after creating a new administrative user in the admin interface.
rake sunspot:reindex
In production mode run:
RAILS_ENV=production rake sunspot:reindex
Specify only a model:
rake sunspot:reindex[,Person]
Do reindex in 1 record batches, useful if reindex crashes to see in which one (very slow to start)
rake sunspot:reindex[1]
With MySQL 64 bit binaries, add:
export DYLD_LIBRARY_PATH="/usr/local/mysql/lib:$DYLD_LIBRARY_PATH"
Example Apache configuration:
# create /usr/local/etc/apache/Includes/default.conf
# and add default site:
<VirtualHost *:80>
# ServerName www..org
DocumentRoot /var/rails/rism-ch/public
<Directory "/var/rails/rism-ch/public">
Options All -Indexes +ExecCGI -Includes +MultiViewsny
Require all granted
</Directory>
RailsEnv production
</VirtualHost>
Double check permissions in the muscat installation. Also make sure DocumentRoot and Directory point to public/ in the muscat installation location.
Start Apache, Solr and the related services in production mode (see below).
Alternatively, you can take advantatge of the Apache itk module, that simplifies not only having more than one Muscat in a single server using virtual hosts, but also the whole file permissions and ownership altogether. This Apache module (mpm-itk.sesse.net/) makes the Apache web server to run as a plain user, so it is the owner of the files, and it is no longer necessary file ownership via chown or users via sudo. Apache-itk is an official Debian and Ubuntu package, so:
apt install libapache2-mpm-itk
And add the following stanza in the apache configuration file, for example just before the DocumentRoot declaration:
<IfModule mpm_itk_module> AssignUserId muscat_linux_user_name muscat_linux_group </IfModule>
where muscat_linux_user_name and muscat_linux_group are the first two fields of the output of the id shell command.
If you run more than Muscat in a single server, there is another change to be done: the Solr ports, so each Muscat access its own Solr server. This port number has to be the same in those two files:
config/sunspot.yml config/blacklight.yml
Muscat has two running daemons that should be started. Note that CRONO is deprecated and no longer used!:
# Make sure solr is not running as root! sudo -u www-data bundle exec rake RAILS_ENV=production sunspot:solr:start
‘DelayedJob` now is a bit more tricky, as different queues get different worker pools:
sudo -u www-data RAILS_ENV=production bin/delayed_job start --pool=reindex,triggers,folders:10 --pool=sub_reindex:10 --pool=resave --pool=export
Other ways to start delayed_job:
sudo RAILS_ENV=production bin/delayed_job start rake jobs:work # Run in foreground rake jobs:workoff # Run all jobs in foreground and exit rake jobs:clear # Clear failed jobs ## in production it is then... RAILS_ENV=production bundle exec rake jobs:<xxx>
Depending on the env, remove sudo if necessary. RAILS_ENV is needed only on production, and has to come after sudo.
It is also good to block access to Solr:
sudo iptables -A INPUT -p tcp -s localhost --dport 8983 -j ACCEPT sudo iptables -A INPUT -p tcp --dport 8983 -j DROP
Also make sure that if there is a proxy, it is configured in ‘config/initializers/recaptcha.rb`:
config.proxy = 'http://myproxy.com.au:8080'
In version 7.0 the crono scheduler was removed and a normal crontab plus a script is provided. The default crontab file should be linked in /etc/conn.d. Edit the muscat_crontab to set PATH_TO to the correct muscat installation, you will need to do it by hand.
On debian/ubuntu this should work:
ln -s config/muscat_crontab /etc/cron.d chmod +x /etc/cron/muscat_crontab sudo service cron restart
If your crontab still has this line:
0 2 * * * cd $PATH_TO && $PATH_TO/bin/rake blacklight:delete_old_searches[7] RAILS_ENV=production
It is save to delete it as it is now taken care but the global muscat crontab. NOTE If you ever change the cron file name, it CANNOT contains dots! Or it will silently fail.
Add to /etc/mysql.cnf
[mysqld] innodb_buffer_pool_size = 8G innodb_log_buffer_size = 512M innodb_log_file_size = 1G innodb_write_io_threads = 16 innodb_flush_log_at_trx_commit = 0
To speed up imports. (See here for more)
On a full muscat DB there can be many many _old versions_ in the “‘versions“` table, since version snapshots are kept for each saved item. When doing development this can be quite annoying since it can take up to 30 minutes to restore a db.
sed '/INSERT INTO `versions`/d' muscat_dump.sql > muscat_no_versions.sql
Removes all the old versions. From 30 minutes to 9.
On older Ubuntu systems add the following repositories
deb http://ppa.launchpad.net/brightbox/ruby-ng/ubuntu trusty main deb https://oss-binaries.phusionpassenger.com/apt/passenger trusty main
(On some systems it is necessary to uncomment therubyracer from the gemfile) On Ubuntu 18.04 only passenger is needed:
deb https://oss-binaries.phusionpassenger.com/apt/passenger bionic main
(Add /etc/apt/sources.list or create a relevant file in /etc/apt/sources.list.d) Also make sure that universe repositories are enabled!
On 18.40, also enable GPG verification of Passenger:
sudo apt-get install -y dirmngr gnupg sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 561F9B9CAC40B2F7 sudo apt-get install -y apt-transport-https ca-certificates
Then install all the dependencies and setup passenger: