====== rclone und S3 ======

===== Links =====

https://forum.rclone.org/t/ipv6-guide-review-with-google-drive-ec2-s3/43218


===== Config =====

Config Datei (Pfad anzeigen: ''rclone config file''): z.B. /home/$USER/.config/rclone/rclone.conf oder /root/.config/rclone/rclone.conf .

Ceph RGW:
<code>
[MOUNT-NAME]
type = s3
provider = Ceph
env_auth = false
access_key_id = $KEYID
secret_access_key = $ACCESSKEY
region =
endpoint = https://hostname.domain.tld
location_constraint =
acl =
server_side_encryption =
storage_class =
</code>


===== S3 sync =====

ceph osd pool ls

<file>
radosgw-admin bucket list
[
    "pool1",
    "pool2"
]
</file>

<file>
radosgw-admin user list
[
    "test_user",
    "test_user2"
]
</file>


**Ziel**: -> User anlegen: test_user1: <code bash>radosgw-admin user create --uid=test_user1 --email=MAIL@DOMAIN.TLD --display-name=test_user1</code>
<file>
{
    "user_id": "test_user1",
    "display_name": "test_user1",
    "email": "MAIL@DOMAIN.TLD",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "test_user1",
            "access_key": "...",
            "secret_key": "..."
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw"
}
</file>
Ergebnis:

<file>
radosgw-admin user list
[
    "test_user1",
    "test_user2"
]
</file>

bucket anlegen: <code bash>"rclone mkdir ZielS3:s3synctest"</code>


cat ~/.config/rclone/rclone.conf

<file>
[QuelleS3]
type = s3
provider = Ceph
env_auth = false
access_key_id = ...
secret_access_key = ...
region =
endpoint = https://QUELLE.DOMAIN.TLD
location_constraint =
acl =
server_side_encryption =
storage_class =

[ZielS3]
type = s3
provider = Ceph
env_auth = false
access_key_id = ...
secret_access_key = ...
region =
endpoint = https://ZIEL.DOMAIN.TLD
location_constraint =
acl =
server_side_encryption =
storage_class =
</file>

TEST: <code bash>rclone sync -i "QuelleS3:s3synctest" "ZielS3:s3synctest" --dry-run -v</code>
<file>
Ausgabe:
y) Yes, this is OK (default)
n) No, skip this
s) Skip all copy operations with no more questions
!) Do all copy operations with no more questions
q) Exit rclone now.
</file>
-> !

Dateien die im Ziel vorhanden sind, aber nicht in der Quelle:

<file>
rclone: delete "testdatei"?
y) Yes, this is OK (default)
n) No, skip this
s) Skip all delete operations with no more questions
!) Do all delete operations with no more questions
q) Exit rclone now.
y/n/s/!/q> y
2023/11/13 23:41:52 INFO  : testdatei: Deleted
2023/11/13 23:41:52 NOTICE: 
Transferred:  	   55.192 MiB / 55.192 MiB, 100%, 185.291 KiB/s, ETA 0s
Checks:               424 / 424, 100%
Deleted:                1 (files), 0 (dirs)
Transferred:          500 / 500, 100%
Elapsed time:      1m22.1s
</file>

rclone Optimierungen: https://forum.rclone.org/t/rclone-sync-s3-to-s3-runs-for-hours-and-copy-nothing/39687/23
RAM-Bedarf: 1kb per Objekt!


testen (rclone bleibt im Vordergrund:)
rclone mount QuelleS3:s3synctest /tmp/s3synctest -v


Später: S3 Multi sync: https://docs.ceph.com/en/quincy/radosgw/multisite-sync-policy/


==== Shellscript ====


Beispiel mit Ceph rados:
<code bash>
#!/bin/bash

set -e -o pipefail

buckets_no=$(radosgw-admin bucket list | jq length)

echo "number of buckets $buckets_no"

# https://docs.ovh.com/de/storage/object-storage/s3/rclone/

bucket_src="test_quelle"
bucket_dest="test_ziel"


i=1
j=0

until [ $i -gt $buckets_no ]
do
  echo "bucket $i of $buckets_no:"
  bucket_name=$(radosgw-admin bucket list | jq -r "(.[$j])")
  echo "$bucket_name"

  # bucket anlegen?
  echo rclone mkdir "$bucket_dest:$bucket_name"

  echo rclone sync -i "$bucket_src:$bucket_name" "$bucket_dest:$bucket_name" dry-run -vv --checksum --fast-list --s3-list-version 2 --checkers 32 --transfers 16

  let i=$i+1
  let j=$j+1
done
</code>