The SwiftRiver engine has a few moving parts. This guide covers what you need to do to get a basic SwiftRiver installation going. Mix and match these instructions to match what you know you need.

Recommended Platforms

This guide does not cover installation under Windows systems. If you want to do that, please use the SwiftRiver windows instructions and update them with anything you learn.

Requirements

Handy guides

Notes

Installing the API

Set up API war file in Tomcat

  1. Checkout API

    # Create directories to hold code and war file
mkdir -p /opt/swiftriver/src
mkdir -p /opt/swiftriver/api
cd /opt/swiftriver/src


# Clone API code
git clone git://github.com/ushahidi/SwiftRiver-API.git

  2. Compile API and copy resources to the correct directories

    # Go into code directory 
cd /opt/swiftriver/src/SwiftRiver-API
 
# Package file
mvn clean package

# Move war file to api folder
cp target/swiftriver-api.war /opt/swiftriver/api
 
# Copy configuration files to correct api places
cp target/classes/config/swiftriver-api.xml $TOMCAT_HOME/Catalina/localhost
cp target/classes/indexer.properties /opt/swiftriver/api
chmod 666 /opt/swiftriver/api/indexer.properties

  3. Update the entries below in file swiftiver-api.xml. You shouldn't need to change any other values (More details are in the file below)

    <!-- SET DOCBASE TO /opt/swiftriver/api/swiftriver-api.war -->
<Context docBase="/opt/swiftriver/api/swiftriver-api.war" path="/swiftriver-api">

		<!-- UPDATE THE DATABASE INFO BELOW TO REFLECT THE CREDENTIALS CREATED ON THE SECTION BELOW (db: swiftriver, user: swiftriver, password:swiftriver -->
        <Resource auth="Container" driverClassName="com.mysql.jdbc.Driver"
                maxActive="8" maxIdle="4" name="jdbc/SwiftRiverDB" type="javax.sql.DataSource"
                url="jdbc:mysql://localhost/swiftriver?zeroDateTimeBehavior=convertToNull"
                username="swiftriver" password="swiftriver" />
...
		<!-- UPDATE solr/serverURL TO http://localhost:8080/solr/swiftriver -->
        <Environment name="solr/serverURL" type="java.lang.String" value="http://localhost:8080/solr/swiftriver"/>	
 
		<!-- UPDATE solr/indexerProperties TO /opt/swiftriver/api/indexer.properties -->
        <Environment name="solr/indexerProperties" type="java.lang.String" value="/opt/swiftriver/api/indexer.properties" />
...

Setting up the database

Log into your MySQL database and execute the following commands:

-- Creates `swiftriver` database
CREATE DATABASE swiftriver CHARACTER SET utf8 COLLATE utf8_unicode_ci;
 
-- Adds user with password 'swiftriver'
GRANT ALL PRIVILEGES ON swiftriver.* TO swiftriver@'localhost' IDENTIFIED BY 'swiftriver';

Then exit the MySQL prompt and import the mysql schema into the database:

mysql -uswiftriver -p swiftriver < /opt/swiftriver/src/SwiftRiver-API/src/main/resources/config/sql/schema.sql

Tomcat, Java and JDBC

If your application fails to start later on, it might be because it's not connecting to the MySQL database. To solve that, you'll need to make sure that the JDBC driver is installed on Tomcat.

# Install JDBC
sudo apt-get install libmysql-java

# Copy driver to Tomcat's lib (make sure to double check these paths make sense on your system!)
cp /usr/share/java/mysql.jar /usr/share/tomcat7/lib

Setting up Solr

To get Solr going, we need to move some files around:

# Copy configuration files to solr directory
cp -r /opt/swiftriver/src/SwiftRiver-API/solr/swiftriver/conf /var/lib/tomcat7/solr/swiftriver
 
# Make sure Tomcat can write on that dir
chown -R tomcat7:tomcat7 /var/lib/tomcat7/solr/swiftriver

 

Once that's done, navigate to http://localhost:8080/solr, Go to Core Admin > Add core and fill in the fields as shown below.

Get it going!

With all that in place, all you have to do is:

# Restart tomcat7
service tomcat7 restart
 
# Test it!
curl http://localhost:8080/swiftriver-api/
 
# Expected answer:
{"error":"unauthorized","error_description":"An Authentication object was not found in the SecurityContext"}

 

If that's what you got, do a little dance! You're half way there!

(If not, check your logs, and battle it kid!)

 

SwiftRiver UI Client

As you can see from the architecture, the UI client is separate from the API. If you're feeling Rambo, just jump straight into interacting with the API.

If not, here's what you have to do to get the UI going.

(Assuming that your DocumentRoot is on /var/www)

# Make sure you have all the libraries required by the front end
sudo apt-get install php5-curl php5-mcrypt php5-mysql php5-gd
 
# Go into your webroot
cd /var/www
 
# Set up git for directoy
git init
git remote add origin git://github.com/ushahidi/SwiftRiver.git
git fetch
 
# Checkout version 1.0 of code and grab all submodules
git checkout master
git submodule init && git submodule update
 
# Create necessary dirs
mkdir application/cache
chmod 777 application/cache
mkdir application/logs
chmod 777 application/logs
 
# Init all config files
cp application/config/site.php.template application/config/site.php
cp application/config/database.php.template application/config/database.php
cp application/config/cache.php.template application/config/cache.php
cp application/config/auth.php.template application/config/auth.php
cp application/config/cookie.php.template application/config/cookie.php

Once that's done, update the following files:

return array(
	'driver'        => 'SwiftRiver',
	
	// OAuth Parameters
	'token_endpoint' => 'http://localhost:8080/swiftriver-api/oauth/token', // UPDATE THIS TO POINT TO YOUR API URL
	'client_id'     => 'trusted-client',
	'client_secret' => 'somesecret',
	'grant_type'    => 'password',
	
	// No authentication for these controllers
	'ignore_controllers' => array('login', 'error_handler', 'welcome')
);
...
 
	/* Default maximum number of rivers a user can create */
	'default_river_quota' => 1,


	/* Default maximum number of drops a river can hold */
	'default_river_drop_quota' => 10000,


	/* Site url */
	'site_url' => 'http://localhost', // UPDATE THIS TO THE ADDRESS WHERE YOUR UI IS BEING DEPLOYED
 
	// IF YOU ARE SETTING UP PROPER EMAIL SERVERS, UPDATE THE VALUES BELOW TO MAKE EVERYTHING NEAT
	/* Default domain for outgoing emails */
	'email_domain' => 'example.com',


	/* Default domain for outgoing comment notification emails */
	'comments_email_domain' => 'example.com'
 
...
return array(
	'base_url' => 'http://localhost:8080/swiftriver-api/v1', // UPDATE THIS TO POINT TO YOUR API URL
);

You shouldn't need to change any other files.

Once that's done, do an apache2ctl restart for good measure and you should be good to go. Visit http://localhost and do another dance.

mod_rewrite

If you get a "Not Found" response when the system redirects you to /login, maybe mod_rewrite is getting in the way. If that's the case:

Indexing / Crawling / Messaging services

If you've done everything above, you should be able to log into the UI, but nothing much else. Let's change that! There's quite a bit to be done, so get a cup of coffee and hold tight.

Please make sure the RabbitMQ server is installed before proceeding - you shouldn't need to change any configurations for it.

  1. First, install all the requirements for the services below

    apt-get -y install python2.7 python-pip python-mysqldb python-imaging python-lxml python-httplib2
pip install tweepy
pip install pika
pip install feedparser
pip install python-cloudfiles

  2. Check out the Swiftriver Core

    # Create services directory
mkdir /opt/swiftriver/services/python
 
# Create logs directory
mkdir /opt/swiftriver/services/python/logs
 
# Get back to src directory 
cd /opt/swiftriver/src
 
# Clone core
git clone git://github.com/ushahidi/SwiftRiver-Core.git

# Copy the rss, semanticsqueue amd mediaextractor apps to the service directory
cp -rf SwiftRiver-Core/rss /opt/swiftriver/services/python
cp -rf SwiftRiver-Core/twitter /opt/swiftriver/services/python
cp -rf SwiftRiver-Core/semanticsqueue /opt/swiftriver/services/python
cp -rf SwiftRiver-Core/mediaextractor /opt/swiftriver/services/python

  3. Configure services

    # Configure RSS
cd /opt/swiftriver/services/python/rss/config
 
# Create log dir
mkdir /opt/swiftriver/services/python/rss/logs



# Copy templates
cp rss_fetcher.cfg.template rss_fetcher.cfg
cp rss_scheduler.cfg.template rss_scheduler.cfg

# For both files
# Update "/path/to/log" to "/opt/swiftriver/services/python/rss/logs"
# On pid_file, update "/path/to" to "/opt/swiftriver/services/python/rss"
# Update DB details too
host=localhost
port=3306
user=swiftriver
pass=swiftriver
database=swiftriver
 
# Run SQL install
cd /opt/swiftriver/services/python/install
mysql -uswiftriver -p swiftriver < rss.sql
    # Create log dir
mkdir /opt/swiftriver/services/python/mediaextractor/logs
 
# Go to config directory
cd /opt/swiftriver/services/python/mediaextractor/config
 
# Copy template
cp mediaextractor.cfg.template mediaextractor.cfg
 
# Edit file and Update "/path/to/log" to "/opt/swiftriver/services/python/mediaextractor/logs"
# On pid_file, update "/path/to" to "/opt/swiftriver/services/python/mediaextractor" 
    # Create log dir
mkdir /opt/swiftriver/services/python/semanticsqueue/logs

# Go to config directory
cd /opt/swiftriver/services/python/semanticsqueue/config
 
# Copy template
cp semanticsqueue.cfg.template semanticsqueue.cfg

# Update "/path/to/log" to "/opt/swiftriver/services/python/semanticsqueue/logs"
# On pid_file, update "/path/to" to "/opt/swiftriver/services/python/semanticsqueue"
    # Create log dir
mkdir /opt/swiftriver/services/python/twitter/logs
 
# Create Twitter cache file
touch /var/cache/twitter.cache

# Go to config directory
cd /opt/swiftriver/services/python/twitter/config

# Copy templates
cp firehose.cfg.template firehose.cfg
cp manager.cfg.template manager.cfg
 
# For both files
# Update "/path/to/log" to "/opt/swiftriver/services/python/twitter/logs"
# On pid_file, update "/path/to" to "/opt/swiftriver/services/python/twitter"

# Update all Twitter credentials on firehose.cfg (you'll need to have an app registered on twitter for that) 
consumer_key=TWITTER_API_KEY
consumer_secret=TWITTER_API_SECRET
token_key=TWITTER_ACCESS_TOKEN
token_secret=TWITTER_ACCESS_TOKEN_SECRET

# FOR manager.cfg
# Update Twitter cache file
cache_file=/var/cache/twitter.cache

# Update database credentials
host=localhost
port=3306
user=swiftriver
pass=swiftriver
database=swiftriver

  4. Copy lib folder to services folder

    # Back to source directory
cd /opt/swiftriver/src/
 
# Copy lib folder
cp -rf SwiftRiver-Core/lib /opt/swiftriver/services/python

  5. Build and install the Swiftriver API Client

    # Go to source directory
cd /opt/swiftriver/src/

# Clone repo
git clone git://github.com/ushahidi/swiftriver-api-java.git
 
# Build it
cd /opt/swiftriver/src/swiftriver-api-java
mvn clean install

  6. Install the dropqueue processor

    # Source dir
cd /opt/swiftriver/src

# Clone repo
git clone git://github.com/ushahidi/swiftriver-core-dropqueue-processor.git dropqueue-processor


# Before building, it's worth noting that the dropqueue-processor/pom.xml file asks for a huge amount of memory for the war file.
# If you have less than 4GB of RAM on your machine, make sure to edit the pom.xml file accordingly or else everything will explode.
# You need to change the "jvmSettings > initialMemorySize" and "jvmSettings > maxMemorySize".
# I would recommend setting half the system memory to maxMemorySize, and half of that to initialMemorySize.
# Also note that this might play on performance, so be smart about your system hardware requirements!
 
# Build 
cd dropqueue-processor
mvn clean package

# Move build and config files to services folder
cp -rf target/generated-resources/appassembler/jsw/dropqueue-processor /opt/swiftriver/services
cp config/* /opt/swiftriver/services/dropqueue-processor/conf
 
# Create wrapper conf file and make wrappers executable
cd /opt/swiftriver/services
perl -p -i -e 's/-Dext\.prop\.dir/-Dext\.prop\.dir=\/opt\/swiftriver\/services\/dropqueue-processor\/conf/g' dropqueue-processor/conf/wrapper.conf
chmod +x dropqueue-processor/bin/dropqueue-processor
chmod +x dropqueue-processor/bin/wrapper-linux*

  7.  Install rules processor

    # Source dir
cd /opt/swiftriver/src

# Clone repo
git clone git://github.com/ushahidi/swiftriver-core-rules-processor.git rules-processor

# Before building, it's worth noting that the rules-processor/pom.xml file asks for a huge amount of memory for the war file.
# If you have less than 4GB of RAM on your machine, make sure to edit the pom.xml file accordingly or else everything will explode.
# You need to change the "jvmSettings > initialMemorySize" and "jvmSettings > maxMemorySize".
# I would recommend setting half the system memory to maxMemorySize, and half of that to initialMemorySize.
# Also note that this might play on performance, so be smart about your system hardware requirements!
 
# Build 
cd rules-processor
mvn clean package

# Move build and config files to services folder
cp -rf target/generated-resources/appassembler/jsw/rules-processor /opt/swiftriver/services
cp config/* /opt/swiftriver/services/rules-processor/conf
 
# Update the rules /opt/swiftriver/services/rules-processor/conf/rules-processor.properties file with
db.driverClassName=com.mysql.jdbc.Driver
db.url=jdbc:mysql://localhost/swiftriver?zeroDateTimeBehavior=convertToNull
db.username=swiftriver
db.password=swiftriver

# Create wrapper conf file and make wrappers executable
cd /opt/swiftriver/services
perl -p -i -e 's/-Dext\.prop\.dir/-Dext\.prop\.dir=\/opt\/swiftriver\/services\/rules-processor\/conf/g' rules-processor/conf/wrapper.conf
chmod +x rules-processor/bin/rules-processor
chmod +x rules-processor/bin/wrapper-linux*
  8. This is everything to configure for now. Well done for making it to here! You are a champion!

Installing Swiftriver as a service

Now that we configured everything, all you have to do is copy and paste the lines below on your command line so that Swiftriver runs as a service

# Run the remaining ops from the home directory
cd ~/
 
# Create the swiftriver script
cat <<EOF > swiftriver
#!/bin/bash
### BEGIN INIT INFO
# Provides:	SwiftRiver service stack bootstrap
# Required-Start:
# Required-Stop:
# Default-Start:
# Default-Stop:
# Short-Description: Automatically start the background crawlers for SwiftRiver
# Description
### END INIT INFO
 
# Set the PYTHONPATH
export PYTHONPATH=$PYTHONPATH:/opt/swiftriver/services/python/lib
 
# cd into the directory with the services dir
cd /opt/swiftriver/services
 
start_services() {
	# RSS Scheduler and Fetcher 
	python python/rss/rss_scheduler.py start
	python python/rss/rss_fetcher.py start
 
	# Semantics queue and media extraction
	python python/semanticsqueue/semanticsqueue.py start
	python python/mediaextractor/mediaextractor.py start
	
	# Twitter
	python python/twitter/manager.py start
	python python/twitter/firehose.py start
 
	# DropQueue processor and rules processor
	dropqueue-processor/bin/dropqueue-processor start
	rules-processor/bin/rules-processor start
}
 
# Stops the content services
stop_services() {
	# Twitter
	python python/twitter/firehose.py stop
	python python/twitter/manager.py stop
 
	# RSS
	python python/rss/rss_fetcher.py stop
	python python/rss/rss_scheduler.py stop
 
	# Queues
	python python/semanticsqueue/semanticsqueue.py stop
	python python/mediaextractor/mediaextractor.py stop
 
	# DropQueue and rules processor
	dropqueue-processor/bin/dropqueue-processor stop
	rules-processor/bin/rules-processor stop
}
 
case \$1 in
	start)
		echo "Starting SwiftRiver content services"
		start_services
	;;
	stop)
		echo "Stopping SwiftRiver content services"
		stop_services
	;;
	restart)
		echo "Restarting SwiftRiver content services"
		stop_services
		start_services
	;;
	*)
		echo "Usage: /etc/init.d/swiftriver (start|stop|restart)"
		exit 1
	;;
esac
EOF
 
cp swiftriver /etc/init.d
chmod a+x /etc/init.d/swiftriver
 
# Add the swiftriver service
update-rc.d swiftriver defaults 95 05

Now, all you have to do is run service swiftriver start, and enjoy your success!


More info

Logs to watch

If during installation, anything doesn't behave as expected, here are the logs you should check (double points if you paste log dumps into bug reports!):

We can't stress enough how helpful these are. (smile)

Issues with RabbitMQ

If you run rabbitmqctl status and you see something like Error: unable to connect to node 'rabbit@swiftriver-dev': nodedown as a response, RabbitMQ is having problems starting.

Run sudo service rabbitmq-server restart and RabbitMQ should be good again.


Example swiftriver-api.xml

    <Context docBase="/opt/swiftriver/api/swiftriver-api.war" path="/swiftriver-api" >
        <!-- SwiftRiver Database configuration -->
        <Resource auth="Container" driverClassName="com.mysql.jdbc.Driver" 
            maxActive="8" maxIdle="4" 
            name="jdbc/SwiftRiverDB"
            type="javax.sql.DataSource"
            url="jdbc:mysql://localhost/swiftriver?zeroDateTimeBehavior=convertToNull"
            username="swiftriver"
            password="swiftriver"/>
    
        <!-- Encryption Key -->
        <Environment name="encryptionKey" type="java.lang.String" value="2344228477#97{7&amp;6&gt;82"/>
    
        <!-- MQ Properties -->
        <Environment name="mqHost" type="java.lang.String" value="localhost"/>
        <Environment name="mqUser" type="java.lang.String" value="guest"/>
        <Environment name="mqPass" type="java.lang.String" value="guest"/>
        
        <!-- HTTP Solr Server -->
        <Environment name="solr/serverURL" type="java.lang.String" value="http://localhost:8080/solr/swiftriver"/>
    
        <!-- Location of Solr indexing properties file -->
        <Environment name="solr/indexerProperties" type="java.lang.String" value="/opt/swiftriver/api/indexer.properties" />
    
        <!-- Keys for the indexer properties file -->
        <Environment name="indexer/lastDropIdPropKey" type="java.lang.String" value="indexer.lastDropId" />
        <Environment name="indexer/batchSizePropKey" type="java.lang.String" value="indexer.batchSize" />
        <Environment name="indexer/runInterval" type="java.lang.String" value="30000"/>

        <!-- Default authentication scheme. Possible values are: 
                database
                crowdmapid
                
            'database' is the default 
        -->
        <Environment name="authSchemeName" type="java.lang.String" value="database"/>
        
        <!-- CrowdmapID API URL e.g. https://example.com/ -->
        <Environment name="crowdmapid/serverURL" type="java.lang.String" value="https://crowdmapid.com/api"/>
        <Environment name="crowdmapid/apiKey" type="java.lang.String" value=""/>
        <Environment name="crowdmapid/apiKeyParamName" type="java.lang.String" value="api_secret"/>

        <!-- Mail configuration -->
        <Environment name="mail/host" type="java.lang.String" value="localhost" />
        <Environment name="mail/senderAddress" type="java.lang.String" value="no-reply@swiftriver.dev"/>
        <Environment name="mail/resetPasswordUrl" type="java.lang.String" value="http://swiftriver.dev/login/reset_password"/>
        <Environment name="mail/activateAccountUrl" type="java.lang.String" value="http://swiftriver.dev/login/activate"/>

      </Context>

swiftriver-api.xml properties

ParameterDescription
mqHostThe host running the RabbitMQ server
mqUserUser to connect to RabbitMQ
mqPasswordPassword for the user used to connect to RabbitMQ
solr/serverURLURL of your Solr server
solr/indexerPropertiesLocation of the properties file for the indexer - a background process that periodically updates Solr with the new drops
indexer/lastDropIDPropKeyThe property key that specifies the ID of the last drop to be posted to Solr. This value serves as the reference point for fetching new drops
indexer/batchSizePropKeyThe property key that specifies the maximum number of drops to post to Solr during each run
indexer/runIntervalThe property key that specifies how often (in milliseconds) the indexer should check for new drops and update Solr
authSchemeNameName of the authentication scheme. The possible values are database and crowdmapid
crowdmapid/serverURLURL of the CrowdmapID deployment