"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
Go azure-blobazure-blob-storageazure-filesbackblaze-b2cloud-storagedropboxencryptionftpfuse-filesystemgogolanggoogle-cloud-storagegoogle-driveonedriveopenstack-swiftrclones3sftpsyncwebdav
docs: build overview page from the backend data
Nick Craig-Wooda1522468db33
diff --git a/docs/content/overview.md b/docs/content/overview.mdindex ee3b6ccfc..7d2a3ac20 100644--- a/docs/content/overview.md+++ b/docs/content/overview.md@@ -14,106 +14,7 @@ show through.Here is an overview of the major features of each cloud storage system.-| Name | Hash | ModTime | Case Insensitive | Duplicate Files | MIME Type | Metadata |-| ---------------------------- |:-----------------:|:-------:|:----------------:|:---------------:|:---------:|:--------:|-| 1Fichier | Whirlpool | - | No | Yes | R | - |-| Akamai Netstorage | MD5, SHA256 | R/W | No | No | R | - |-| Amazon S3 (or S3 compatible) | MD5 | R/W | No | No | R/W | RWU |-| Backblaze B2 | SHA1 | R/W | No | No | R/W | - |-| Box | SHA1 | R/W | Yes | No | - | - |-| Citrix ShareFile | MD5 | R/W | Yes | No | - | - |-| Cloudinary | MD5 | R | No | Yes | - | - |-| Drime | - | - | No | No | R/W | - |-| Dropbox | DBHASH ¹ | R | Yes | No | - | - |-| Enterprise File Fabric | - | R/W | Yes | No | R/W | - |-| FileLu Cloud Storage | MD5 | R/W | No | Yes | R | - |-| Filen | Blake3 | R/W | Yes | No | R/W | - |-| Files.com | MD5, CRC32 | DR/W | Yes | No | R | - |-| FTP | - | R/W ¹⁰ | No | No | - | - |-| Gofile | MD5 | DR/W | No | Yes | R | - |-| Google Cloud Storage | MD5 | R/W | No | No | R/W | - |-| Google Drive | MD5, SHA1, SHA256 | DR/W | No | Yes | R/W | DRWU |-| Google Photos | - | - | No | Yes | R | - |-| HDFS | - | R/W | No | No | - | - |-| HiDrive | HiDrive ¹² | R/W | No | No | - | - |-| HTTP | - | R | No | No | R | R |-| iCloud Drive | - | R | No | No | - | - |-| Internet Archive | MD5, SHA1, CRC32 | R/W ¹¹ | No | No | - | RWU |-| Internxt | - | - | No | No | - | - |-| Jottacloud | MD5 | R/W | Yes | No | R | RW |-| Koofr | MD5 | - | Yes | No | - | - |-| Linkbox | - | R | No | No | - | - |-| Mail.ru Cloud | Mailru ⁶ | R/W | Yes | No | - | - |-| Mega | - | - | No | Yes | - | - |-| Memory | MD5 | R/W | No | No | - | - |-| Microsoft Azure Blob Storage | MD5 | R/W | No | No | R/W | - |-| Microsoft Azure Files Storage | MD5 | R/W | Yes | No | R/W | - |-| Microsoft OneDrive | QuickXorHash ⁵ | DR/W | Yes | No | R | DRW |-| OpenDrive | MD5 | R/W | Yes | Partial ⁸ | - | - |-| OpenStack Swift | MD5 | R/W | No | No | R/W | - |-| Oracle Object Storage | MD5 | R/W | No | No | R/W | RU |-| pCloud | MD5, SHA1 ⁷ | R/W | No | No | W | - |-| PikPak | MD5 | R | No | No | R | - |-| Pixeldrain | SHA256 | R/W | No | No | R | RW |-| premiumize.me | - | - | Yes | No | R | - |-| put.io | CRC-32 | R/W | No | Yes | R | - |-| Proton Drive | SHA1 | R/W | No | No | R | - |-| QingStor | MD5 | - ⁹ | No | No | R/W | - |-| Quatrix by Maytech | - | R/W | No | No | - | - |-| Seafile | - | - | No | No | - | - |-| SFTP | MD5, SHA1 ² | DR/W | Depends | No | - | - |-| Shade | - | - | Yes | No | - | - |-| Sia | - | - | No | No | - | - |-| SMB | - | R/W | Yes | No | - | - |-| SugarSync | - | - | No | No | - | - |-| Storj | - | R | No | No | - | - |-| Uloz.to | MD5, SHA256 ¹³ | - | No | Yes | - | - |-| WebDAV | MD5, SHA1 ³ | R ⁴ | Depends | No | - | - |-| Yandex Disk | MD5 | R/W | No | No | R | - |-| Zoho WorkDrive | - | - | No | No | - | - |-| The local filesystem | All | DR/W | Depends | No | - | DRWU |--¹ Dropbox supports [its own custom-hash](https://www.dropbox.com/developers/reference/content-hash).-This is an SHA256 sum of all the 4 MiB block SHA256s.--² SFTP supports checksums if the same login has shell access and-`md5sum` or `sha1sum` as well as `echo` are in the remote's PATH.--³ WebDAV supports hashes when used with Fastmail Files, Owncloud and Nextcloud only.--⁴ WebDAV supports modtimes when used with Fastmail Files, Owncloud and Nextcloud-only.--⁵ [QuickXorHash](https://docs.microsoft.com/en-us/onedrive/developer/code-snippets/quickxorhash)-is Microsoft's own hash.--⁶ Mail.ru uses its own modified SHA1 hash--⁷ pCloud only supports SHA1 (not MD5) in its EU region--⁸ Opendrive does not support creation of duplicate files using-their web client interface or other stock clients, but the underlying-storage platform has been determined to allow duplicate files, and it-is possible to create them with `rclone`. It may be that this is a-mistake or an unsupported feature.--⁹ QingStor does not support SetModTime for objects bigger than 5 GiB.--¹⁰ FTP supports modtimes for the major FTP servers, and also others-if they advertised required protocol extensions. See [this](/ftp/#modification-times)-for more details.--¹¹ Internet Archive requires option `wait_archive` to be set to a non-zero value-for full modtime support.--¹² HiDrive supports [its own custom-hash](https://static.hidrive.com/dev/0001).-It combines SHA1 sums for each 4 KiB block hierarchically to a single-top-level sum.--¹³ Uloz.to provides server-calculated MD5 hash upon file upload. MD5 and SHA256-hashes are client-calculated and stored as metadata fields.+{{< features-table >}}### Hash@@ -509,74 +410,7 @@ See [the metadata docs](/docs/#metadata) for more info.All rclone remotes support a base command set. Other features dependupon backend-specific capabilities.-| Name | Purge | Copy | Move | DirMove | CleanUp | ListR | StreamUpload | MultithreadUpload | LinkSharing | About | EmptyDir |-| ---------------------------- |:-----:|:----:|:----:|:-------:|:-------:|:-----:|:------------:|:------------------|:------------:|:-----:|:--------:|-| 1Fichier | No | Yes | Yes | No | No | No | No | No | Yes | No | Yes |-| Akamai Netstorage | Yes | No | No | No | No | Yes | Yes | No | No | No | Yes |-| Amazon S3 (or S3 compatible) | No | Yes | No | No | Yes | Yes | Yes | Yes | Yes | No | No |-| Backblaze B2 | No | Yes | No | No | Yes | Yes | Yes | Yes | Yes | No | No |-| Box | Yes | Yes | Yes | Yes | Yes | No | Yes | No | Yes | Yes | Yes |-| Citrix ShareFile | Yes | Yes | Yes | Yes | No | No | No | No | No | No | Yes |-| Drime | Yes | Yes | Yes | Yes | No | No | Yes | Yes | No | No | Yes |-| Dropbox | Yes | Yes | Yes | Yes | No | No | Yes | No | Yes | Yes | Yes |-| Cloudinary | No | No | No | No | No | No | Yes | No | No | No | No |-| Enterprise File Fabric | Yes | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes |-| Filen | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes |-| Files.com | Yes | Yes | Yes | Yes | No | No | Yes | No | Yes | No | Yes |-| FTP | No | No | Yes | Yes | No | No | Yes | No | No | No | Yes |-| Gofile | Yes | Yes | Yes | Yes | No | No | Yes | No | Yes | Yes | Yes |-| Google Cloud Storage | Yes | Yes | No | No | No | No | Yes | No | No | No | No |-| Google Drive | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes |-| Google Photos | No | No | No | No | No | No | No | No | No | No | No |-| HDFS | Yes | No | Yes | Yes | No | No | Yes | No | No | Yes | Yes |-| HiDrive | Yes | Yes | Yes | Yes | No | No | Yes | No | No | No | Yes |-| HTTP | No | No | No | No | No | No | No | No | No | No | Yes |-| iCloud Drive | Yes | Yes | Yes | Yes | No | No | No | No | No | No | Yes |-| ImageKit | Yes | No | Yes | No | No | No | No | No | No | No | Yes |-| Internet Archive | No | Yes | No | No | Yes | Yes | No | No | Yes | Yes | No |-| Jottacloud | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes |-| Koofr | Yes | Yes | Yes | Yes | No | No | Yes | No | Yes | Yes | Yes |-| Mail.ru Cloud | Yes | Yes | Yes | Yes | Yes | No | No | No | Yes | Yes | Yes |-| Mega | Yes | No | Yes | Yes | Yes | No | No | No | Yes | Yes | Yes |-| Memory | No | Yes | No | No | No | Yes | Yes | No | No | No | No |-| Microsoft Azure Blob Storage | Yes | Yes | No | No | No | Yes | Yes | Yes | No | No | No |-| Microsoft Azure Files Storage | No | Yes | Yes | Yes | No | No | Yes | Yes | No | Yes | Yes |-| Microsoft OneDrive | Yes | Yes | Yes | Yes | Yes | Yes ⁵ | No | No | Yes | Yes | Yes |-| OpenDrive | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes | Yes |-| OpenStack Swift | Yes ¹ | Yes | No | No | No | Yes | Yes | No | No | Yes | No |-| Oracle Object Storage | No | Yes | No | No | Yes | Yes | Yes | Yes | No | No | No |-| pCloud | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes |-| PikPak | Yes | Yes | Yes | Yes | Yes | No | No | No | Yes | Yes | Yes |-| Pixeldrain | Yes | No | Yes | Yes | No | No | Yes | No | Yes | Yes | Yes |-| premiumize.me | Yes | No | Yes | Yes | No | No | No | No | Yes | Yes | Yes |-| put.io | Yes | No | Yes | Yes | Yes | No | Yes | No | No | Yes | Yes |-| Proton Drive | Yes | No | Yes | Yes | Yes | No | No | No | No | Yes | Yes |-| QingStor | No | Yes | No | No | Yes | Yes | No | No | No | No | No |-| Quatrix by Maytech | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes | Yes |-| Seafile | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes |-| SFTP | No | Yes ⁴| Yes | Yes | No | No | Yes | No | No | Yes | Yes |-| Sia | No | No | No | No | No | No | Yes | No | No | No | Yes |-| SMB | No | No | Yes | Yes | No | No | Yes | Yes | No | No | Yes |-| SugarSync | Yes | Yes | Yes | Yes | No | No | Yes | No | Yes | No | Yes |-| Storj | Yes ² | Yes | Yes | No | No | Yes | Yes | No | Yes | No | No |-| Uloz.to | No | No | Yes | Yes | No | No | No | No | No | No | Yes |-| WebDAV | Yes | Yes | Yes | Yes | No | No | Yes ³ | No | No | Yes | Yes |-| Yandex Disk | Yes | Yes | Yes | Yes | Yes | No | Yes | No | Yes | Yes | Yes |-| Zoho WorkDrive | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes | Yes |-| The local filesystem | No | No | Yes | Yes | No | No | Yes | Yes | No | Yes | Yes |--¹ Note Swift implements this in order to delete directory markers but-it doesn't actually have a quicker way of deleting files other than-deleting them individually.--² Storj implements this efficiently only for entire buckets. If-purging a directory inside a bucket, files are deleted individually.--³ StreamUpload is not supported with Nextcloud--⁴ Use the `--sftp-copy-is-hardlink` flag to enable.--⁵ Use the `--onedrive-delta` flag to enable.+{{< optional-features-table >}}### Purgediff --git a/docs/layouts/partials/bool.html b/docs/layouts/partials/bool.htmlnew file mode 100644index 000000000..0dc9b311b--- /dev/null+++ b/docs/layouts/partials/bool.html@@ -0,0 +1,6 @@+{{ if . }}+ <span class="text-success">✔</span>+{{ else }}+ <span class="text-danger">✖</span>+{{ end }}+diff --git a/docs/layouts/shortcodes/features-table.html b/docs/layouts/shortcodes/features-table.htmlnew file mode 100644index 000000000..494fe9271--- /dev/null+++ b/docs/layouts/shortcodes/features-table.html@@ -0,0 +1,120 @@+<table>+ <thead>+ <tr>+ <th>Name</th>+ <th>Tier</th>+ <th>Hash</th>+ <th>ModTime</th>+ <th>Case Insensitive</th>+ <th>Duplicate Files</th>+ <th>MIME Type</th>+ <th>Metadata</th>+ </tr>+ </thead>+ <tbody>+ {{ range sort site.Data.backends "name" "asc" }}+ {{ if .virtual }}{{ continue }} {{ end }}+ <tr>+ <td style="font-weight: bold;">+ <a href="/{{ .backend | urlize }}">{{ .name }}</a>+ </td>++ {{/* Tier Column */}}+ <td>+ {{ partial "tier.html" (dict "name" .backend "align" false) }}+ </td>++ {{/* Hash Column */}}+ <td>+ {{ $local := in .features "IsLocal" }}+ {{ if $local }}+ <b>ALL</b>+ {{ else }}+ {{ with .hashes }}+ {{ delimit . ", " }}+ {{ else }}+ -+ {{ end }}+ {{ end }}+ </td>++ {{/* ModTime */}}+ <td>+ {{ $d := or (in .features "MkdirMetadata") (in .features "DirSetModTime") }}+ {{ $r := true }} {{/* FIXME don't have a way of reading this from the features */}}+ {{ $w := and (isset . "precision") (le .precision 1000000000) }}+ {{- if $d -}}+ D+ {{- end -}}+ {{ if and $r $w -}}+ R/W+ {{- else if $r -}}+ R+ {{- else if $w -}}+ W+ {{- else -}}+ -+ {{- end -}}+ </td>++ {{/* Case Insensitive */}}+ <td>+ {{ partial "bool.html" (in .features "CaseInsensitive") }}+ </td>++ {{/* Duplicate Files */}}+ <td>+ {{ partial "bool.html" (in .features "DuplicateFiles") }}+ </td>++ {{/* MIME Type Synthesis */}}+ <td>+ {{ $r := in .features "ReadMimeType" }}+ {{ $w := in .features "WriteMimeType" }}+ {{ if and $r $w }}+ R/W+ {{ else if $r }}+ R+ {{ else if $w }}+ W+ {{ else }}+ -+ {{ end }}+ </td>++ {{/* Metadata Synthesis */}}+ <td>+ {{ $meta := "" }}++ {{/* Check Directory Metadata Support */}}+ {{ if or (in .features "ReadDirMetadata") (in .features "WriteDirMetadata") }}+ {{ $meta = "D" }}+ {{ end }}++ {{/* Check System Metadata Read */}}+ {{ if in .features "ReadMetadata" }}+ {{ $meta = print $meta "R" }}+ {{ end }}++ {{/* Check System Metadata Write */}}+ {{ if in .features "WriteMetadata" }}+ {{ $meta = print $meta "W" }}+ {{ end }}++ {{/* Check User Metadata Support */}}+ {{ if in .features "UserMetadata" }}+ {{ $meta = print $meta "U" }}+ {{ end }}++ {{/* Output result or hyphen if empty */}}+ {{ if eq $meta "" }}+ -+ {{ else }}+ {{ $meta }}+ {{ end }}+ </td>++ </tr>+ {{ end }}+ </tbody>+</table>diff --git a/docs/layouts/shortcodes/optional-features-table.html b/docs/layouts/shortcodes/optional-features-table.htmlnew file mode 100644index 000000000..61de1ba38--- /dev/null+++ b/docs/layouts/shortcodes/optional-features-table.html@@ -0,0 +1,40 @@+<table>+ <thead>+ <tr>+ <th align="left">Name</th>+ <th align="center">Purge</th>+ <th align="center">Copy</th>+ <th align="center">Move</th>+ <th align="center">DirMove</th>+ <th align="center">CleanUp</th>+ <th align="center">ListR</th>+ <th align="center">StreamUpload</th>+ <th align="left">MultithreadUpload</th>+ <th align="center">LinkSharing</th>+ <th align="center">About</th>+ <th align="center">EmptyDir</th>+ </tr>+ </thead>+ <tbody>+ {{ range sort site.Data.backends "name" "asc" }}+ {{ if .virtual }}{{ continue }} {{ end }}+ <tr>+ <td align="left" style="font-weight: bold;">+ <a href="/{{ .backend | urlize }}">{{ .name }}</a>+ </td>++ <td align="center">{{ partial "bool.html" (in .features "Purge") }}</td>+ <td align="center">{{ partial "bool.html" (in .features "Copy") }}</td>+ <td align="center">{{ partial "bool.html" (in .features "Move") }}</td>+ <td align="center">{{ partial "bool.html" (in .features "DirMove") }}</td>+ <td align="center">{{ partial "bool.html" (in .features "CleanUp") }}</td>+ <td align="center">{{ partial "bool.html" (in .features "ListR") }}</td>+ <td align="center">{{ partial "bool.html" (in .features "PutStream") }}</td>+ <td align="center">{{ partial "bool.html" (or (in .features "OpenChunkWriter") (in .features "OpenWriterAt")) }}</td>+ <td align="center">{{ partial "bool.html" (in .features "PublicLink") }}</td>+ <td align="center">{{ partial "bool.html" (in .features "About") }}</td>+ <td align="center">{{ partial "bool.html" (in .features "CanHaveEmptyDirectories") }}</td>+ </tr>+ {{ end }}+ </tbody>+</table>