Skip to content

Latest commit

 

History

History
171 lines (141 loc) · 8.47 KB

BUILD.md

File metadata and controls

171 lines (141 loc) · 8.47 KB

Build

Dependencies

The build process of OpenIO SDS depends on several third-party projects.

When building only the SDK, OpenIO only depends on:

  • cmake, make : involved in the build process.
  • bison, flex : generates expression parsers.
  • glib2, glib2-devel
  • curl, libcurl, libcurl-devel
  • json-c, json-c-devel
  • asn1c : Now only necessary at the compile time, this is our ASN.1 codec forked from Lev Walkin's excellent ASN.1 codec. The purpose of our fork is simply to provide codec for explicitely sized integers (int{8,16,32,64} instead of long int) and GLib-2.0 memory allocations

Building the entire project will require the SDK dependencies, but also:

  • python: Pure python code generator (no dependency), and python modules.
  • python-distutils-extra: required for the installation process
  • httpd, httpd-devel : server base for RAWX and RAINX services
  • apr, apr-util-devel, apr-devel : internally used by RAINX and RAWX modules
  • attr, libattr-devel : we use xattr a lot to stamp RAWX chunks and repositories base directory.
  • gridinit
  • lzo, lzo-devel : RAWX compression
  • sqlite, sqlite-devel : base storage for META{0,1,2} and SQLX services.
  • zeromq3, zeromq3-devel : communication of events between services and forward agents.
  • zookeeper-devel, libzookeeper_mt.so : building with distribution's zookeeper client is OK, but the package ships with a lot of dependencies, including the openjdk. We recommand to use the official Oracle/Sun JDK, and to build your own zookeeper client from the source to avoid a huge waste of space and bandwith.
  • python-setuptools
  • python-pbr
  • beanstalkd: you need it to have the event-agent working

In addition, there some additional dependencies at runtime:

  • python-eventlet
  • python-werkzeug
  • python-gunicorn
  • python-plyvel
  • python-redis
  • python-requests
  • python-simplejson
  • pyxattr (python-xattr on Debian/Ubuntu)
  • libapache2-mod-wsgi (as named on Ubuntu), the WSGI module pour apache2
  • python-cliff
  • python-pyeclib
  • python-futures

The account service will require an up and running backend:

  • redis

Generating the documentation will require:

  • epydoc: available in your python virtualenv

Configuration

The Makefile's generation is performed by cmake. The master CMake directives files accepts several options. Each option has to be specified on the cmake's command line with the following format:

cmake -D${K}=${V} ${SRCDIR}

In addition to common cmake options, these specific options are also available:

Directive Help
LD_LIBDIR Path suffix to the installation prefix, to define the default directory for libraries. E.g. "lib" or "lib64", depending on the architecture.
STACK_PROTECTOR Trigger stack protection code. Only active when CMAKE_BUILD_TYPE is set to "Debug" or "RelWithDebInfo"
GRIDD_PLUGINS Installation directory for gridd plugins.
APACHE2_MODDIR Installation directory for apache2 modules.
ALLOW_BACKTRACE generate backtraces in errors.
FORBID_DEPRECATED define it to turn into errors the warnings for deprecated symbols from the GLib2.
EXE_PREFIX Defines a prefix to all CLI tool. By default, set to "sds".
SOCKET_OPTIMIZED define if to use socket3 and accept4 syscalls
SOCKET_DEFAULT_LINGER_ONOFF (integer value) triggers the onoff value of the SO_LINGER configuration.
SOCKET_DEFAULT_LINGER_DELAY (integer value) set it to the delay in milliseconds, this will the delay part of the SO_LINGER configuration.
SOCKET_DEFAULT_QUICKACK boolean
SOCKET_DEFAULT_NODELAY boolean

In addition, some options axist to specify uncommon installation paths. Their format is ${DEP}_INCDIR or ${DEP}_LIBDIR, and DEP might take the given values APACHE2, ASN1C, ATTR, CURL, JSONC, LIBRAIN, LZO, ZK, ZLIB, ZMQ

Building

Now that cmake succeeded, it is time to build and install the binaries with make.

make
make test
make DESTDIR=${install_dir} install

Install python module (preferably inside a virtualenv):

python setup.py develop

Compile-time configuration

Used by cmake

Macro Default Description
OIOSDS_RELEASE "master" Global release name
OIOSDS_PROJECT_VERSION_SHORT "1.0" Minor version number

Used by gcc

Macro Default Description
DAEMON_DEFAULT_TIMEOUT_READ 1000 How long a gridd will block on a recv() (in milliseconds)
DAEMON_DEFAULT_TIMEOUT_ACCEPT 1000 How long a gridd will block on a accept() (in milliseconds)
Macro Default Description
OIO_EVT_BEANSTALKD_DEFAULT_TUBE "oio"
Macro Default Description
GCLUSTER_RUN_DIR "/var/run" Prefix to spool.
GCLUSTER_CONFIG_FILE_PATH "/etc/oio/sds.conf" System-wide configuration file
GCLUSTER_CONFIG_DIR_PATH "/etc/oio/sds.conf.d" System-wide configuration directory for additional files.
GCLUSTER_CONFIG_LOCAL_PATH ".oio/sds.conf" Local configuration directory.
GCLUSTER_AGENT_SOCK_PATH "/var/run/oio-sds-agent.sock" Default path for agent's socket.
Macro Default Description
PROXYD_PREFIX "v3.0" Prefix applied to proxyd's URL, second version (with accounts)
PROXYD_HEADER_PREFIX "X-oio-" Prefix for all the headers sent to the proxy
PROXYD_HEADER_REQID PROXYD_HEADER_PREFIX "req-id" Header whose value is printed in access log, destined to agregate several requests belonging to the same session.
PROXYD_HEADER_NOEMPTY PROXYD_HEADER_PREFIX "no-empty-list" Flag sent to the proxy to turn empty list (results) into 404 not found.
Macro Default Description
MALLOC_TRIM_SIZE 0
PERIODIC_MALLOC_TRIM_SIZE 0
OIO_STAT_PREFIX_REQ "counter req.hits"
OIO_STAT_PREFIX_TIME "counter req.time"
Macro Default Description
SQLX_DIR_SCHEMAS NULL Default directory used to gather applicative schema of SQLX bases. NULL by default, meaning that no directory is set, so that there is no attempt to load a schema.
SQLX_ADMIN_PREFIX_SYS "sys." Prefix used for keys used in admin table of sqlite bases
SQLX_ADMIN_PREFIX_USER "user." Prefix used for keys used in admin table of sqlite bases
SQLX_ADMIN_INITFLAG SQLX_ADMIN_PREFIX_SYS "sqlx.init" Key used in admin table of sqlite bases
SQLX_ADMIN_STATUS SQLX_ADMIN_PREFIX_SYS "sqlx.flags" Key used in admin table of sqlite bases
SQLX_ADMIN_REFERENCE SQLX_ADMIN_PREFIX_SYS "sqlx.reference" Key used in admin table of sqlite bases
SQLX_ADMIN_BASENAME SQLX_ADMIN_PREFIX_SYS "sqlx.name" Key used in admin table of sqlite bases
SQLX_ADMIN_BASETYPE SQLX_ADMIN_PREFIX_SYS "sqlx.type" Key used in admin table of sqlite bases
SQLX_ADMIN_NAMESPACE SQLX_ADMIN_PREFIX_SYS "sqlx.ns" Key used in admin table of sqlite bases
Macro Default Description
M2V2_ADMIN_PREFIX_SYS SQLX_ADMIN_PREFIX_SYS "m2."
M2V2_ADMIN_PREFIX_USER SQLX_ADMIN_PREFIX_USER "m2."
M2V2_ADMIN_VERSION M2V2_ADMIN_PREFIX_SYS "version"
M2V2_ADMIN_QUOTA M2V2_ADMIN_PREFIX_SYS "quota"
M2V2_ADMIN_SIZE M2V2_ADMIN_PREFIX_SYS "usage"
M2V2_ADMIN_CTIME M2V2_ADMIN_PREFIX_SYS "ctime"
M2V2_ADMIN_VERSIONING_POLICY M2V2_ADMIN_PREFIX_SYS "policy.version"
M2V2_ADMIN_STORAGE_POLICY M2V2_ADMIN_PREFIX_SYS "policy.storage"
M2V2_ADMIN_KEEP_DELETED_DELAY M2V2_ADMIN_PREFIX_SYS "keep_deleted_delay"
META2_INIT_FLAG M2V2_ADMIN_PREFIX_SYS "init"
Macro Default Description
OIO_META1_DIGITS_KEY "meta1_digits" Variable name in the /etc/oio/sds.conf to configure the number of digits that name a meta1 base.
OIO_META1_DIGITS_DEFAULT 4 Default number of digits to name a meta1 database
Macro Default Description
RAWX_HEADER_PREFIX "X-oio-chunk-meta-" Prefix applied to proxyd's URL, second version (with accounts)
Macro Default Description
OIO_USE_OLD_FMEMOPEN undefined Use the old implementation of glibc's fmemopen. Starting from glibc 2.22 the new implementation lacks the binary mode which made things work.