Below are instructions for running Fluo in a production environment where Accumulo, Hadoop & Zookeeper are installed and running. If you want to avoid setting up these dependencies, consider using fluo-dev or MiniFluo.
Requirements
Before you install Fluo, you will need the following installed and running on your local machine or cluster:
Obtain a distribution
Before you can install Fluo, you will need to obtain a distribution tarball that works for your environment. Fluo distributions are built for specific releases of Hadoop and Accumulo. If you are using Accumulo 1.6.4 and Hadoop 2.6.3, you can download the latest release. If you need a release for a different environment or one built from the main branch, follow these steps:
First, clone Fluo:
git clone https://github.com/fluo-io/fluo.git
cd fluo/
Optionally, check out a stable tag (if you don’t want to build a release from main):
git checkout 1.0.0-beta-2
Next, build a distribution for your environment. The tarball will be created in
modules/distribution/target.
mvn package -Daccumulo.version=1.6.4 -Dhadoop.version=2.7.1
Install Fluo
When you have a distribution tarball built for your environment, follow these steps to install Fluo.
First, choose a directory with plenty of space and untar the distribution:
tar -xvzf fluo-1.0.0-beta-2-bin.tar.gz
Verify that your distribution has the same versions of Hadoop & Accumulo as your environment:
cd fluo-1.0.0-beta-2
ls lib/hadoop-* lib/accumulo-*
Next, copy the example configuration to the base of your configuration directory to create the default configuration for your Fluo install:
cp conf/examples/* conf/
The default configuration will be used as the base configuration for each new application. Therefore, you should modify fluo.properties for your environment. However, you should not configure any application settings (like observers).
NOTE - All properties that have a default are set with it. Uncomment a property if you want to use a value different than the default. Properties that are unset and uncommented must be set by the user.
Fluo command script
The Fluo command script is located at bin/fluo of your Fluo installation. All Fluo
commands are invoked by this script.
Modify and add the following to your ~/.bashrc if you want to be able to execute the
fluo script from any directory:
export PATH=/path/to/fluo-1.0.0-beta-2/bin:$PATH
Source your .bashrc for the changes to take effect and test the script
source ~/.bashrc
fluo
Running the script without any arguments prints a description of all commands.
./bin/fluo
Configure a Fluo application
You are now ready to configure a Fluo application. Use the command below to create the
configuration necessary for a new application. Feel free to pick a different name (other
than myapp) for your application:
fluo new myapp
This command will create a directory for your application at apps/myapp of your Fluo
install which will contain a conf and lib.
The apps/myapp/conf directory contains a copy of the fluo.properties from your default
configuration. This should be configured for your application:
vim apps/myapp/fluo.properties
When configuring the observer section in fluo.properties, you can configure your instance for the phrasecount application if you have not created your own application. See the phrasecount example for instructions. You can also choose not to configure any observers but your workers will be idle when started.
The apps/myapp/lib directory should contain any observer jars for your application. If
you configured fluo.properties for observers, copy any jars containing these
observer classes to this directory.
Initialize your application
After your application has been configured, use the command below to initialize it:
fluo init myapp
This only needs to be called once and stores configuration in Zookeeper.
Start your application
A Fluo application consists of one oracle process and multiple worker processes. Before starting your application, you can configure the number of worker process in your fluo.properties file.
When you are ready to start your Fluo application on your YARN cluster, run the command below:
fluo start myapp
The start command above will work for a single-node or a large cluster. By using YARN, you do not need to deploy the Fluo binaries to every node on your cluster or start processes on every node.
You can use the following command to check the status of your instance:
fluo status myapp
For more detailed information on the YARN containers running Fluo:
fluo info myapp
You can also use yarn application -list to check the status of your Fluo instance
in YARN. Logs are viewable within YARN.
When you have data in your fluo instance, you can view it using the command fluo scan.
Pipe the output to less using the command fluo scan | less if you want to page
through the data.
Stop your Fluo application
Use the following command to stop your Fluo application:
fluo stop myapp
If stop fails, there is also a kill command.
fluo kill myapp
Tuning Accumulo
Fluo will reread the same data frequently when it checks conditions on
mutations. When Fluo initializes a table it enables data caching to make
this more efficient. However you may need to increase the amount of memory
available for caching in the tserver by increasing tserver.cache.data.size.
Increasing this may require increasing the maximum tserver java heap size in
accumulo-env.sh.
Fluo will run many client threads, will want to ensure the tablet server
has enough threads. Should probably increase the
tserver.server.threads.minimum Accumulo setting.
Using at least Accumulo 1.6.1 is recommended because multiple performance bugs were fixed.
Run locally without YARN
If you do not have YARN set up, you can start the oracle and worker as a local Fluo process using the following commands:
local-fluo start-oracle
local-fluo start-worker
Use the following commands to stop a local Fluo process:
local-fluo stop-worker
local-fluo stop-oracle
In a distributed environment, you will need to deploy and configure a Fluo distribution on every node in your cluster.