22 Dec 2015
In preparation for a beta 2 release, the stress test was run again on EC2. The test went well outperforming the first stress test and beta-1 stress test.
For this test run, initially ~1 billion random integers were generated and loaded into Fluo via map reduce. After that, 1 million random integers were repeatedly loaded 20 times, sleeping 10 minutes between loads. After everything finished, the test was a success. The number of unique integers computed independently by MapReduce matched the number computed by Fluo. Both computed 1,019,481,332 unique integers.
The test took a total of 7 hours 30 minutes and 30 seconds. Over this time period 61.7 million NodeObserver and 20 million NodeLoader transactions were executed. The average rate of transactions per second for the entire test was 2,968 transactions per second. At the conclusion of the test, the stress table had 3.87 billion entries.
The test was run with the following environment.
An exciting new development in the Fluo eco-system for beta-2 is the utilization of Grafana and InfluxDB to plot metrics. Also metrics configuration was simplified making it possible to report metrics from Map Reduce and Spark. In the plots below we can see metrics from the load transactions executing in Map Reduce. In previous test, this was not visible, being able to see it now is really useful.
Notifications were building up during the test. A better method than sleeping between loads, as mentioned in fluo-io/fluo-stress#30, is still needed.
Before starting the long run, a few short runs loading 1 million few times were done with an empty table.
A long run of webindex will also be run on EC2 before releasing beta-2.
View all posts in the news archive