Apache Fluo 1.0.0-incubating is the first release of Fluo as an Apache project and the fourth release for the project. Below are resources for this release:

Starting with 1.0.0-incubating, Apache Fluo will follow semver for all future API changes. The API consists of everything under the org.apache.fluo.api package. Code outside of this package can change at any time. If your project is using Fluo code that falls outside of the API, then consider initiating a discussion about adding it to the API.

Significant changes

The 1.0.0-incubating release includes 167 commits that were made since the last release (1.0.0-beta-2). The significant changes are summarized below. For information about changes before this release, see the archive of Fluo releases before becoming an Apache project.

Stabilized API

Starting with this release, Apache Fluo’s API will follow semver. The API is defined as everything under the org.apache.fluo.api Java package. Given that the plan is to support this API for a long time, a lot of API improvements were made since the beta-2 release. The following is a list of significant API changes since beta-2.

  • #772 : Get with default
  • #770 : Use varargs for Columns
  • #768 : Made Obsever and Application configuration consistent
  • #743 : Make data classes final
  • #723 : Removed FluoAdmin.InitOps()
  • #714 : Use Charsequence in API
  • #695 : Removed all 3rd party types from API
  • #680 : Changed package prefix from io.fluo to org.apache
  • #647 : Added Oracle and Worker to API
  • #639 : Improved scanning API
  • #626 : Added operation to get row+column pairs. Further improved by #758.
  • #119 - Consistently offer String in API

Configurable classpath

The previous releases of Fluo contained specific versions of Accumulo, Hadoop, Zookeeper, and other Fluo dependencies. However, the actual dependencies needed should be determined by the existing version of software where Fluo is run. Fluo changed to a model of making its classpath be completely configurable by an administrator. Example classpath configuration and download scripts ship with Fluo to help make setup easier. These script were only tested with specific versions of Hadoop and Accumulo and may not work with all versions. These changes were made in #706 and #687.

New Fluo version command

Added a version command to the Fluo scripts in #779. This makes it easy to automatically build shaded jars with the correct version of Fluo for use in Spark and Map Reduce.

User and historical metrics

Two major improvements related to metrics were made. Support for user metrics was added by #767. This allows observers to easily report application specific metrics. In #635 the example InfluxDB and Grafana configurations were updated to show historical metrics.

Improved transaction processing

Transaction processing throughput was increased by changes in #593 and some subsequent bug fixes. These changes moved commit processing from the threads executing user code to an asynchronous background process. This allows user threads to queue a transaction for commit an immediately start working on another notification. This new process allows many more transactions to be working through the commit process concurrently. Before the change a worker could only be committing up to a few hundred transactions at any time. After the change, a single worker could easily be committing tens of thousands of transactions at a time. These changes increase the latency of individual transactions, which can have some drawbacks #650. For this release changes were made in #654 to use a priority queue for notifications in the worker. The changes in #654 partially solves the lock wait problem identified in #650.

Improved Accumulo scanning

Fluo utilizes server side Accumulo iterators. For this release these iterators were modified to leverage seeking on the sever side. Before these changes the iterators used to sequentially scan data. For popular cells that had a lot versions, this could be slow. These changes were made in #623.

Other changes of interest

  • #668 - Added method to append byte to BytesBuilder
  • #648 - Added documentation to help users if YARN is killing containers
  • #619 #621 - Created Fluo logo

Testing

A two day run of Webindex was started on September 7th using the latest snapshot version of Fluo. This test used 11 EC2 m3.xlarge nodes. The test ran without problem. A good bit before that a 3 day run of Webindex was conducted.