Hadoop Components Versions in Distros Matrix

Derek Wood is a DevOps Engineer at Cask where he is building tools to manage and operate the next generation of Big Data applications. Prior to Cask, Derek ran large scale distributed systems at Wells Fargo and at Yahoo!, where he was the senior engineering lead for the CORE content personalization platform

The Apache Hadoop ecosystem is always evolving, with the major distributions constantly upgrading their included core Hadoop components.  This can present a challenge when building any application which runs on top of Hadoop.  When developing our open-source application framework, CDAP, we strive to maintain compatibility with all major Hadoop distributions.


Building on our previous reference of supported Linux distributions, the table below breaks down each version of the major Hadoop distributions into their core Hadoop component versions.  We have found this very useful, and hope it will come in handy for your work as well.  All information is culled from documented release notes.

CDH

Avro Hadoop HBase Hive Spark ZooKeeper
5.4.4 avro-1.7.6 hadoop-2.6.0 hbase-1.0.0 hive-1.1.0 spark-1.3.0 zookeeper-3.4.5
5.4.3
5.4.2
5.4.1
5.4.0
5.3.6 hadoop-2.5.0 hbase-0.98.6 hive-0.13.1 spark-1.2.0
5.3.5
5.3.4
5.3.3
5.3.2
5.3.1
5.3.0
5.2.5 spark-1.1.0
5.2.4
5.2.3
5.2.1
5.2.0
5.1.5 avro-1.7.5 hadoop-2.3.0 hbase-0.98.1 hive-0.12.0 spark-1.0.0
5.1.4
5.1.3
5.1.2
5.1.0
5.0.6 hbase-0.96.1.1 spark-0.9.0
5.0.5
5.0.4
5.0.3
5.0.2
5.0.1
5.0.0
Avro Hadoop HBase Hive Spark ZooKeeper
4.7.1 hadoop-2.0.0 hbase-0.94.15 hive-0.10.0 zookeeper-3.4.5
4.7.0
4.6.0
4.5.0 hbase-0.94.6
4.4.0
4.3.2
4.3.1
4.3.0
4.2.2 hbase-0.94.2
4.2.1
4.2.0
4.1.5 hbase-0.92.1 hive-0.9.0 zookeeper-3.4.3
4.1.4
4.1.3
4.1.2
4.1.1
4.1.0
4.0.1 hive-0.8.1
4.0.0

HDP

Avro Hadoop HBase Hive Spark ZooKeeper
2.2.6 hadoop-2.6.0 hbase-0.98.4 hive-0.14.0 spark-1.2.1 zookeeper-3.4.6
2.2.4
2.2.0
2.1.15 avro-1.7.4 hadoop-2.4.0 hbase-0.98.0 hive-0.13.1 zookeeper-3.4.5
2.1.10
2.1.7
2.1.5
2.1.3
2.1.2
2.1.1
2.0 hadoop-2.2.0 hbase-0.96.1 hive-0.12.0

Bigtop

Avro Hadoop HBase Hive Spark ZooKeeper
0.8.0 hadoop-2.4.1.6 hbase-0.98.5.6 hive-0.13.0.6 spark-0.9.1.6 zookeeper-3.4.5.6

 

We’d also like to hear from you: do you stick to the stock Hadoop distributions, or do you upgrade individual components? We will share the results in two weeks on twitter.

 

Do you use stock distro or upgrade/downgrade individual components after setup?

View Results

Loading ... Loading ...

UPDATES

  • JUNE 26th 2015: Added CDH 5.3.5
  • JUNE 17th 2015: Added CDH 5.4.2 and CDH 5.3.4
  • JUNE 30th 2015: Added HDP 2.2.6, HDP 2.1.15 and HDP 2.1.10
  • JULY 2nd 2015: Added CDH 5.4.3
  • JULY 10th 2015: Added CDH 5.4.4
  • AUGUST 5th 2015: Added CDH 5.3.6