SeaweedFS excels on small files and has no issue to store large files. Now it is possible to enable Hadoop jobs to read from and write to SeaweedFS.

Build SeaweedFS Hadoop Client Jar

Or you can download the latest version from MavenCentral

• http://central.maven.org/maven2/com/github/chrislusf/seaweedfs-hadoop2-client/

Test SeaweedFS on Hadoop

Suppose you are getting a new Hadoop installation. Here are the minimum steps to get SeaweedFS to run.

You would need to start a weed filer first, build the seaweedfs-hadoop2-client-xxx.jaror seaweedfs-hadoop3-client-xxx.jar, and do the following:

# create etc/hadoop/mapred-site.xml, just to satisfy hdfs dfs. skip this if the file already exists.
$ echo "<configuration></configuration>" > etc/hadoop/mapred-site.xml
# on hadoop2
$ bin/hdfs dfs -Dfs.defaultFS=seaweedfs://localhost:8888 \
               -Dfs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \
               -libjars ./seaweedfs-hadoop2-client-x.x.x.jar \
               -ls /
# or on hadoop3
$ bin/hdfs dfs -Dfs.defaultFS=seaweedfs://localhost:8888 \
               -Dfs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \
               -ls /

Both reads and writes are working fine.

Installation for Hadoop

• Configure Hadoop to use SeaweedFS in etc/hadoop/conf/core-site.xml. core-site.xml resides on each node in the Hadoop cluster. You must add the same properties to each instance of core-site.xml. There are several properties to modify:
  • fs.seaweedfs.impl: This property defines the Seaweed HCFS implementation classes that are contained in the SeaweedFS HDFS client JAR. It is required.
  • : This property defines the default file system URI to use. It is optional if a path always has prefix seaweedfs://localhost:8888.

• Deploy the SeaweedFS HDFS client jar

# Run the classpath command to get the list of directories in the classpath
$ bin/hadoop classpath
# Copy SeaweedFS HDFS client jar to one of the folders
$ cd ${HADOOP_HOME}
# for hadoop2
$ cp ./seaweedfs-hadoop2-client-x.x.x.jar share/hadoop/common/lib/
# or for hadoop3
$ cp ./seaweedfs-hadoop3-client-x.x.x.jar share/hadoop/common/lib/

Supported HDFS Operations

    bin/hdfs dfs -appendToFile README.txt /weedfs/weedfs.txt
    bin/hdfs dfs -cat /weedfs/weedfs.txt
    bin/hdfs dfs -rm -r /uber
    bin/hdfs dfs -chown -R chris:chris /weedfs
    bin/hdfs dfs -chmod -R 755 /weedfs
    bin/hdfs dfs -copyFromLocal README.txt /weedfs/README.txt.2
    bin/hdfs dfs -copyToLocal /weedfs/README.txt.2 .
    bin/hdfs dfs -count /weedfs/README.txt.2
    bin/hdfs dfs -du -h /weedfs
    bin/hdfs dfs -get /weedfs/weedfs.txt
    bin/hdfs dfs -getfacl /weedfs
    bin/hdfs dfs -getmerge -nl /weedfs w.txt
    bin/hdfs dfs -ls /
    bin/hdfs dfs -mkdir /tmp
    bin/hdfs dfs -mkdir -p /tmp/x/y
    bin/hdfs dfs -moveFromLocal README.txt.2 /tmp/x/
    bin/hdfs dfs -mv /tmp/x/y/README.txt.2 /tmp/x/y/README.txt.3
    bin/hdfs dfs -mv /tmp/x /tmp/z
    bin/hdfs dfs -put README.txt /tmp/z/y/
    bin/hdfs dfs -rm /tmp/z/y/*
    bin/hdfs dfs -rmdir /tmp/z/y
    bin/hdfs dfs -stat /weedfs
    bin/hdfs dfs -tail /weedfs/weedfs.txt
    bin/hdfs dfs -test -f /weedfs/weedfs.txt
    bin/hdfs dfs -text /weedfs/weedfs.txt

Notes

SeaweedFS satisfies the HCFS requirements that the following operations to be atomic, when using MySql/Postgres database transactions.

• Creating a file. If the overwrite parameter is false, the check and creation MUST be atomic.
• Deleting a file.
• Renaming a file.
• Renaming a directory.

• Creating a single directory with mkdir().Among these, except file or directory renaming, the following operations are all atomic for any filer store.

• Creating a file

• Deleting a file
• Creating a single directory with mkdir().

The SeaweedFS hadoop client is a pure java library. There are no native libraries to install if you already have Hadoop running.

One of the headache with complicated Java systems is the jar runtime dependency problem, which is resolved by Go's build time dependency resolution. For this SeaweedFS hadoop client, the required jars are mostly shaded and packaged as one fat jar, so there are no extra jar files needed.

• See if you enabled gRpc security.