MINT Framework Core Platform version 2012¶ ↑

• www.multi-access.de

DESCRIPTION:¶ ↑

Multimodal systems realizing a combination of speech, gesture and graphical-driven interaction are getting part of our everyday life.

Examples are in-car assistance systems or recent game consoles. Future interaction will be embedded into smart environments offering the user to choose and to combine a heterogeneous set of interaction devices and modalities based on his preferences realizing an ubiquitous and multimodal access.

This framework enables the modeling and execution of multimodal interaction interfaces for the web based on ruby and implements a server-sided synchronisation of all connected modes and media. Currenlty the framework considers gestures, head movements, multi touch and the mouse as principle input modes. The priciple output media is a web application based on a rails frontend as well as sound support based on the SDL libraries.

Building this framework is an ongoing effort and it has to be pointed out that it serves to demonstrate scientific research results and is not targeted to we applied to serve productive systems as they are several limitations that need to be solved (maybe with your help?) like for instance multi-user support and authentification.

The MINT core gem contains all basic AUI and CUI models as well as the basic infrastructure to create interactors and mappings. For presenting the user interface on a specific platform a “frontend framework” is required. For the first MINT version (2010) we used Rails 2.3 (See github.com/sfeu/MINT-rails). The current version uses nodeJS as the frontend framework (See github.com/sfeu/MINT-nodejs). But for initial experiements ist enough to follow the installation instructions of this document.

There is still no documentation for the framework, but a lot of articles about the concepts and theories of our approach have already been published and can be accessed from our project site www.multi-access.de .

FEATURES:¶ ↑

• Interactor-based modeling (Widgets) of interface elements by using statemachines (see github.com/sfeu/MINT-statemachine )

• Different to approaches like e.g. XHTML+Voice the mode and media synchronisation is implemented on the web server side, which enables interactions including several devices at the sam time and dynamically adding further media and modes during the interaction.

• The srver side is implemented as a distributed system based on software agents and a tuple space that enables to distribute arbitrary parts of the system to different machines.

• Interactors are modelled using s basic subset of SCXML (see github.com/sfeu/scxml )

• A basic set of abstract (modality independent) interactors

• A very limited set of concrete interactors for the web based on a socketstream/jquery frontend (see github.com/sfeu/MINT-nodejs )

• Sound support (see github.com/sfeu/MINT-sdl )

• A basic model inspector to observe the state of all interactors during runtime (see github.com/sfeu/MINT-debugger )

• Automatic synchronisation of an abritary set of connected webbrowsers

• Basic constraint-driven automated ayouting on the server side that calculates pixel-exact coordinates for all interface elements based on the Cassowary constraint solver (see github.com/sfeu/cassowary )

LIMITATIONS:¶ ↑

• No multi-user support or authentification, if another user connects to an application the presentation of the first user is mirrored to the second one.

• The framework only runs on Ubuntu and installation has only be tested for version 10.04 LTS

• We require ruby 1.8, the application threading model inside one agent is currently limited, but critical parts can be distributed to several agents instead.

• we currenlty only support rails 2.3, because we rely on the execute action plugin that has no been ported to rails 3 so far.

CHANGES to the initial MINT 2010 version¶ ↑

• We switched from Rails 2.3 / Juggernaut to NodeJS / Socketstream 0.2 for the frontend framework since NodeJS allows a faster processing of the server-sided mode/media synchronization.

• We can offer limited multi-user support. The devices of each user get synchronized separately.

• We support Ubuntu 12.04 LTS instead of Ubuntu 10.04 LTS.

SYNOPSIS:¶ ↑

For testing the framework please try first to run our example application! - See the install section for the instructions to get it running. This section describes how to setup the basic structure of a new application.

To start a new project:

• install the MINT core gem

sudo gem install MINT-core

• create a new socketstream application

socketstream new <name_of_your_project>

• use the startup script to start the application

socketstream start

• Use Chrome our Chromium browser to access the application:

http://127.0.0.1:3000

• You should see the MINT-Logo as well as a working mouse pointer synchronisation (the green/red dot that follows the mouse pointer)

• If your have troube, please contact us.

REQUIREMENTS:¶ ↑

• Ubuntu LTS 12.04

• Ruby 1.9

• WxWidgets (MINT-debugger)

• libSDL (MINT-sdl for sound support)

• libCassowary (for constraint solver)

• libMagick (for layouting)

• several gems managed by bundler

INSTALL:¶ ↑

The following instructions work only with Ubuntu 10.04 - we have chosen this release because it is an officially long-term supported release (LTS).

• Install ruby 1.9.3 on your Ubuntu 12.04 machine and set a symlink

sudo apt-get install ruby1.9.3

• Install redis-server 2.2.11

sudo apt-get install redis-server

• prevent installation of docs for all dependend gems (optional - but this will save you a lot of time)

include into file .gemrc in home folder

gem: --no-ri --no-rdoc

• install latest stable git

sudo gem install bundler
sudo apt-get install git-core

• Install node.js and the node package manager

sudo apt-get install nodejs npm

• Install socketstream version 0.2 and dependencies using npm

sudo npm install [email protected] -g hiredis redis

• Install libcassowary by adding a new package repository to /etc/apt/sources.lst

deb http://packages.multi-access.de lucid/

• update system to consider new repository

sudo aptitude update

• install native library dependencies that some gems like eventmachine, SDL or Magick or our framework requires and a working C compiler to compile them

sudo apt-get install build-essential curl libsdl-dev libsdl-mixer1.2-dev libmagickcore-dev libmagickwand-dev libwxbase2.8-0 libwxgtk2.8-0 libssl-dev libfreetype6-dev gsfonts libjpeg62-dev

• install cassowary library

sudo apt-get install libcassowary0

• create a directory inside your home folder and change into it

mkdir rails-projects
cd rails-projects

• checkout statemachine, scxml, em-hiredis, dm-redis-adapter, MINT-core, MINT-nodejs, MINT-debugger

git clone ssh://[email protected]/statemachine
git clone ssh://[email protected]/scxml
git clone -b experimental git://github.com/mloughran/em-hiredis.git
git clone -b test_natural_keys ssh://[email protected]/dm-redis-adapter
git clone ssh://[email protected]/MINT-core
git clone ssh://[email protected]/MINT-nodejs
git clone ssh://[email protected]/MINT-debugger

• install all the requried dependencies of MINT-core

sudo gem install bundler
cd MINT-core
sudo bundle install
cd ../MINT-nodeJS
sudo bundle install

• start redis-server in separate terminal (if not already started)

redis-server

• change to MINT-nodejs folder and start the MINT platform

cd MINT-nodejs
socketstream start

• check if platform has been started by accessing the following url with google chrome

localhost:3000

• log in with user “testuser” and you shold see the server-sided mouse pointer synchronization by a green dot following the mouse pointer

• start the example app in separate terminals

bundle exec ruby agents/mappings.rb

bundle exec ruby agents/model.rb

• If it is stil not working - write us an email.

• Now you can play around with the debugger ( see github.com/sfeu/MINT-debugger for source code ) to observe all interactors state changes during runtime within the tuple space

cd MINT-debugger
sudo bundle install
bundle exec ./schript/mint/debugger.sh

• Check if the sound is working ( see github.com/sfeu/MINT-sdl for source code ) - a clicking sound should be played while pointing to the different cells.

• Buy colored gloves and try our gesture recognition app ( will be releasen soon ) to navigate through the cells using hand postures.

• Browse the source code, read (and cite) our papers to get a basic understanding of the framework components.

• You have further ideas? Email us and help us improving the framework.

LICENSE:¶ ↑

The MINT framework is developed by Sebastian Feuerstack Copyright © 2010, 2011, 2012 Sebastian Feuerstack

This program is free software; you can redistribute it and/or modify it under the terms of the GNU Affero General Public License version 3 as published by the Free Software Foundation.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program; if not, see www.gnu.org/licenses or write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

You can contact Sebastian Feuerstack using the contact formular on his homepage www.feuerstack.org.

The modified source and object code versions of this program must display Appropriate Legal Notices, as required under Section 5 of the GNU Affero General Public License version 3.

In accordance with Section 7(b) of the GNU Affero General Public License version 3, these Appropriate Legal Notices must retain the display of the “Powered by MINT Framework” logo. If the display of the logo is not reasonably feasible for technical reasons, the Appropriate Legal Notices must display the words “Powered by the MINT Framework”.