The following is implemented in "weed upload" command. For 3rd party clients, here is the spec.

To support large files, SeaweedFS supports these two kinds of files:

  • Chunk File. Each chunk file is actually just normal files to SeaweedFS.
  • Chunk Manifest. A simple json file with the list of all the chunks.

This piece of code shows the json file structure:

When reading Chunk Manifest files, the SeaweedFS will find and send the data file based on the list of ChunkInfo.

SeaweedFS delegates the effort to the client side. The steps are:

  • split large files into chunks
  • upload each file chunk as usual. Save the related info into ChunkInfo struct. Each chunk can be spread onto different volumes, possibly giving faster parallel access.
  • upload the manifest file with mime type "application/json", and add url parameter "cm=true". The FileId to store the manifest file is the entry point of the large file.

Usually we just append large files. Updating a specific chunk of file is almost the same.

  • upload the new file chunks as usual. Save the related info into ChunkInfo struct.
  • update the updated manifest file with mime type "application/json", and add url parameter "cm=true".
  3. # curl -X POST -F "file=@xaa" http://localhost:9333/submit?pretty=yes
  4. {
  5.   "eTag": "809b2add",
  6.   "fid": "6,1b70e99bcd",
  7.   "fileName": "xaa",
  8.   "fileUrl": "10.34.254.62:8080/6,1b70e99bcd",
  9.   "size": 73433
  10. }
  12. # curl -X POST -F "file=@xab" http://localhost:9333/submit?pretty=yes
  13. {
  14.   "eTag": "9c6ca661",
  15.   "fid": "3,1c863b4563",
  16.   "fileName": "xab",
  17.   "fileUrl": "10.34.254.62:8080/3,1c863b4563",
  18.   "size": 73433
  19. }
  21. // get one fid for manifest file
  22. # curl "10.34.254.62:9333/dir/assign?pretty=yes"
  23. {
  24.   "fid": "5,1ea9c7d93e",
  25.   "url": "10.34.254.62:8080",
  26.   "publicUrl": "10.34.254.62:8080",
  27. }
  29. # cat manifest.json 
  30. {
  31.         "name": "sy.jpg",
  32.         "mime": "image/jpeg",
  33.         "size": 146866,
  35.                 "fid": "6,0100711ab7",
  36.                 "offset": 0,
  37.                 "size": 73433
  38.         }, {
  39.                 "fid": "3,1c863b4563",
  40.                 "offset": 73433,
  41.                 "size": 73433
  42.         }]
  43. }
  45. // upload the manifest file
  46. # curl -v -F "file=@manifest.json" "http://10.34.254.62:8080/5,1ea9c7d93e?cm=true&pretty=yes"
  47. *   Trying 10.34.254.62...
  48. * Connected to 10.34.254.62 (10.34.254.62) port 8080 (#0)
  49. > POST /5,1ea9c7d93e?cm=true&pretty=yes HTTP/1.1
  50. > Host: 10.34.254.62:8080
  51. > User-Agent: curl/7.47.0
  52. > Accept: */*
  53. > Content-Length: 418
  54. > Expect: 100-continue
  55. > Content-Type: multipart/form-data; boundary=------------------------a872064c8f40903c
  56. > 
  57. < HTTP/1.1 100 Continue
  58. < HTTP/1.1 201 Created
  59. < Content-Type: application/json
  60. < Etag: "2229f9b4"
  61. < Date: Wed, 15 Jan 2020 03:12:18 GMT
  62. < Content-Length: 66
  63. < 
  64.   "name": "manifest.json",
  65.   "size": 213,
  66.   "eTag": "2229f9b4"
  68. * Connection #0 to host 10.34.254.62 left intact
  70. // download the full file
  71. # curl -v "http://10.34.254.62:8080/5,1ff0fb46c9" -o out.data
  72. *   Trying 10.34.254.62...
  73.   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
  74.                                  Dload  Upload   Total   Spent    Left  Speed
  75.   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 10.34.254.62 (10.34.254.62) port 8080 (#0)
  76. > GET /5,1ff0fb46c9 HTTP/1.1
  77. > Host: 10.34.254.62:8080
  78. > User-Agent: curl/7.47.0
  79. > Accept: */*
  80. > 
  81. < HTTP/1.1 200 OK
  82. < Accept-Ranges: bytes
  83. < Content-Disposition: inline; filename="sy.jpg"
  84. < Content-Length: 146866
  85. < Content-Type: image/jpeg
  86. < Etag: "3e8ef528"
  87. < Last-Modified: Wed, 15 Jan 2020 03:32:47 GMT
  88. < X-File-Store: chunked
  89. < Date: Wed, 15 Jan 2020 03:33:04 GMT
  90. < 
  91. { [7929 bytes data]
  92. 100  143k  100  143k    0     0  47.2M      0 --:--:-- --:--:-- --:--:-- 70.0M
  93. * Connection #0 to host 10.34.254.62 left intact
  95. // check md5 of the downloaded files
  96. # md5sum out.data 
  97. 836eababc392419580641a7b65370e82  out.data
  99. # md5sum sy.jpg 
  100. 836eababc392419580641a7b65370e82  sy.jpg

There are no particular limit in terms of chunk file size. Each chunk size does not need to be the same, even in the same file. The rule of thumb is to just being able to keep the whole chunk file in memory, and not to have too many small chunk files.

The filer server and the FUSE implementation that uses filer server are automatically chunking large files into smaller chunks.

The list of chunks are stored in filer storage, and managed by filer or weed mount client.