Backup files with rclone
Backup files to Google drive using Rclone. Before we get right to this, let us break down the keywords first.
Backup
In information technology, a backup, or data backup is a copy of computer data taken and stored elsewhere. That way, it may be used to restore the original after a data loss event. The verb form referring to the process is known as “back up”. Whereas the noun and adjective form for the same process is “backup”.
Related Articles
Backups can be used to recover lost data as a result of data deletion, corruption, and also to recover data from an earlier time.[2] Backups provide a simple form of disaster recovery. However, not all backup systems are able to reconstitute a computer system or other complex configuration. Some of which is a computer cluster, active directory server, or database server.
Source: https://en.wikipedia.org/wiki/Backup
Google Drive
Google Drive is a file storage and synchronization service developed by Google. Launched on April 24, 2012, Google Drive allows users to store files on their servers, synchronize files across devices, and share files. In addition to a website, Google Drive offers apps with offline capabilities for Windows and macOS computers, and Android and iOS smartphones and tablets. Google Drive encompasses Google Docs, Google Sheets, and Google Slides, which are a part of the Google Docs Editors office suite that permits collaborative editing of documents, spreadsheets, presentations, drawings, forms, and more.
Drive offers users 15 GB of free storage through Google One. Google One also offers 100 GB, 200 GB, 2 TB, 10 TB, 20 TB, and 30 TB, offered through optional paid plans.
Rclone
Google drive part of the google suits of applications
Rclone is an open source, multi threaded, command line computer program to manage content on cloud and other high latency storage. Its capabilities include sync, transfer, crypt, cache, union, compress and mount. The rclone website lists fifty supported backends including S3 services and Google Drive.
Descriptions of rclone often carry the strapline Rclone syncs your files to cloud storage. Those prior to 2020 include the alternative Rsync for Cloud Storage. Users have called rclone The Swiss Army Knife of cloud storage.
Rclone is well known for its rclone sync and rclone mount commands. It provides further management functions analogous to those ordinarily used for files on local disks, but which tolerate some intermittent and unreliable service. Rclone file transfers optimises for high latency networks. Rclone is commonly a front-end for media servers such as Plex, Emby or Jellyfin to stream content direct from consumer file storage services
Requirements
Before we begin the process of backing up our files on to google drive with the help of the swish army knife (rclone), we need to get the following fundamentals in place.
- A Google account with google drive access
- Rclone installed on the device from backup files are located
- File to backup available in one specific location
Google Account
Skip this if you have a google acount. All google accounts gives you access to their google drive service.
This particular tutorials concentrate on backing up your files to google drive. Hence, we do need a google account to have access to their drive service. The entire process of creating a google should take less 5 minutes to complete. Follow the create google account link below to have your account created
Create your Google Account
Rclone Installation
Rclone has various installation methods based to the host system. It works on Windows, macOS, linux and FreeBSD operating systems. Installation and download instructions for all supported operating systems are available from the official documentation page. Follow the appropriate instructions for your host system.
Step 1: Mounting Google Drive
We are proceeding with the assumption that all previous requirements have been meet. You have rclone installed on your host computer and your backup file have been prepared. Rclone installation can be confirmed by running the following command in the terminal/command prompt
rclone version
The above command should print the rclone version number, go version (go lang) and host operation system.
Let us proceed with the process to mount Remote ( google drive) as a file system on our host system. The rclone config command provides an interactive wizard that guides through the process of successfully adding a remote as a local file system. Run the rclone config command from your terminal/command prompt
rclone config
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
The first prompt in the rclone config interactive wizard issues a list option to add, rename, copy remote and set configuration password. Enter n to create a new remote . To confirm our selection let us hit the enter button on the keyboard
Next, provide a name to identify the remote. Let us give the name gdrive
name> gdrive
Cloud drive types in rclone
After successfully setting the name parameter need to the select the drive type we want to configure. As captured in the title of this article we need to select the google drive option. Google drive is option number 13 which has a string value of “drive” hence let us type drive into the prompt and hit the enter key as always.
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / 1Fichier
\\ "fichier"
2 / Alias for an existing remote
\\ "alias"
3 / Amazon Drive
\\ "amazon cloud drive"
4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, Tencent COS, etc)
\\ "s3"
5 / Backblaze B2
\\ "b2"
6 / Box
\\ "box"
7 / Cache a remote
\\ "cache"
8 / Citrix Sharefile
\\ "sharefile"
9 / Dropbox
\\ "dropbox"
10 / Encrypt/Decrypt a remote
\\ "crypt"
11 / FTP Connection
\\ "ftp"
12 / Google Cloud Storage (this is not Google Drive)
\\ "google cloud storage"
13 / Google Drive
\\ "drive"
14 / Google Photos
\\ "google photos"
15 / Hubic
\\ "hubic"
16 / In memory object storage system.
\\ "memory"
17 / Jottacloud
\\ "jottacloud"
18 / Koofr
\\ "koofr"
19 / Local Disk
\\ "local"
20 / Mail.ru Cloud
\\ "mailru"
21 / Mega
\\ "mega"
22 / Microsoft Azure Blob Storage
\\ "azureblob"
23 / Microsoft OneDrive
\\ "onedrive"
24 / OpenDrive
\\ "opendrive"
25 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
\\ "swift"
26 / Pcloud
\\ "pcloud"
27 / Put.io
\\ "putio"
28 / QingCloud Object Storage
\\ "qingstor"
29 / SSH/SFTP Connection
\\ "sftp"
30 / Sugarsync
\\ "sugarsync"
31 / Tardigrade Decentralized Cloud Storage
\\ "tardigrade"
32 / Transparently chunk/split large files
\\ "chunker"
33 / Union merges the contents of several upstream fs
\\ "union"
34 / Webdav
\\ "webdav"
35 / Yandex Disk
\\ "yandex"
36 / http Connection
\\ "http"
37 / premiumize.me
\\ "premiumizeme"
38 / seafile
\\ "seafile"
Storage> drive
Continuing from the last step we will need to accept the default for the next two prompt by hitting the enter key twice in regards to the client id and the client secret
** See help for drive backend at: for how to create your own.
If you leave this blank, it will use an internal key which is low performance.
Enter a string value. Press Enter for the default ("").
client_id>
OAuth Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret>
Scope definition is the next step in the configuration wizard. We define the permission or access level the rclone application has on the google drive platform. Full access to all files is what we want hence we need to choose option 1 and proceed by hitting the almighty enter key.
Scope that rclone should use when requesting access from drive.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Full access all files, excluding Application Data Folder.
\\ "drive"
2 / Read-only access to file metadata and file contents.
\\ "drive.readonly"
/ Access to files created by rclone only.
3 | These are visible in the drive website.
| File authorization is revoked when the user deauthorizes the app.
\\ "drive.file"
/ Allows read and write access to the Application Data folder.
4 | This is not visible in the drive website.
\\ "drive.appfolder"
/ Allows read-only access to file metadata but
5 | does not allow any access to read or download file content.
\\ "drive.metadata.readonly"
scope> 1
Root folder provides a base folders for the to be backup files. This should be an existing folder on the google drive platform. If one does not exist we need create one a type the folder name into the root folder prompt. For our example we will use “site_performance_result” as the our backup folder
ID of the root folder
Leave blank normally.
Fill in to access "Computers" folders (see docs), or for rclone to use
a non root folder as its starting point.
Enter a string value. Press Enter for the default ("").
root_folder_id> site_performance_result
Let us go with the default value by hitting the enter
Service Account Credentials JSON file path
Leave blank normally.
Needed only if you want use SA instead of interactive login.
Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
Enter a string value. Press Enter for the default ("").
service_account_file>
Editing the advanced configurations is not relevant to us now therefore we skip it by typing “n” into the prompt to move to the next item on the configuration wizard.
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Next up is the remote configurations, here we authenticate rclone and give it access to our google drive folder. On a headless machine which has no graphical user interface and a browser we might want to use the second option. I personally prefer the headless machine option so I enter n into the prompt and proceed to login
Remote config
Use auto config?
* Say Y if not sure
* Say N if you are working on a remote or headless machine
y) Yes (default)
n) No
y/n> n
Copy and paste link echoed to the command prompt/ terminal into a browser to get the verification codes. In the browser follow the prompt and give the right permission. The verification code will show in the browser , copy the verification code and paste in the terminal/command prompt
This is not team drive hence we do not need to configure the team drive option. Let say no by typing “n” into the prompt to proceed.
Configure this as a team drive?
y) Yes
n) No (default)
y/n> n
Finally let confirm the configuration details by entering “y” into the prompt and hit the enter key for the last time.
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
On the next prompt you should see the drive name given above in the list of remote drives. Let complete the process entering the “q” to quitting the wizard.
Current remotes:
Name Type
==== ====
gdrive drive
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
Following all the aforementioned steps help to successfully install and config google drive remote on rclone. Now that we have rclone in place, it is time to perform the actual backup. The rclone tool and the setup we just concluded is ideal for a headless machine environment which does not have a graphical user interface.
Rclone combines to greater effect with a backup script run in a timely manner by a crontab job. For the purpose of our current example we shall concentrate on setup, configuration and the various backup/ syncronizations commands available to us.
Step 2 : Backup files using rclone
The Second step is to backup our files using the configured remote. Rclone provides a couple option to copy file between source to a remote destination.
The rclone command provides a full list of available commands and a brief description about what they do. Let us run the rclone command in the terminal/command line
For the purposes of our current example we will explore the following commands.
config Enter an interactive configuration session.
copy Copy files from source to dest, skipping already copied.
listremotes List all the remotes in the config file.
sync Make source and dest identical, modifying destination only.
tree List the contents of the remote in a tree like fashion.
List remotes command, as the name suggest is provides a list of all configured remotes in the configuration file. Appending the -l flag list the type of remote. The following options are available to the “listremotes” command
Flags:
-h, --help help for listremotes
--long Show the type as well as names.
eg rclone listremotes --long
Copying files using rclone
To copy files to the remote we use the “rclone copy” command. This command copy files from source to dest, skipping already copied. The command does not transfer unchanged files, testing by size and modification time or MD5SUM. Doesn’t delete files from the destination.
Rclone syncs the contents of the directory not the directory. Only the content of the directory copied to the destination not the directory name and contents.
# rclone copy path-to-source-folder remote-name:remote-path
# Let us asume that there 3 files in our source directory
# path-to-source-folder/file-one.pdf
# path-to-source-folder/file-two.pdf
# path-to-source-folder/file-three.pdf
# Only the files in the source folder will be copied to the remote
# path-to-destination-folder/file-one.pdf
# path-to-destination-folder/file-two.pdf
# path-to-destination-folder/file-three.pdf
# per our configuration we run this code to copy our files to the remote
rclone copy --progress path-to-source-folder gdrive:site_performance_result
# the --progress flag as the name states gives you the progression of the file
# copy to remote
The rclone rync command makes source and destination identical, modifying destination only. The command does not transfer unchanged files, testing by size and modification time or MD5SUM. Rclone updates the destination file to match source, including deleting files if necessary. The source folder or files will always be the single source of truth with regards to the “sync” hence any changes on the remote that does not exist on the source location will be deleted from the destination.
# similar stituation to the copy command applies to the sync command with
# regards to the content of source folder synced rather than the folder
# and files
#example command
rclone sync SOURCE remote:DESTINATION
# per our configuration we run this code to copy our files to the remote
rclone sync --progress path-to-source-folder gdrive:site_performance_result
# the --progress flag as the name states gives you the progression of the file
# copy to remote
Conclusion
Rclone is a powerful command line tool that has many use cases. We only concentrated on a fraction of the use case of the Swiss army knife which is rclone. I encourage you to head over to rlcone documentation page for in-depth explanation on their enormous command list. You can improve your back up script and usage of rclone by reading more and playing with the various commands to get the perfect one for your use-case.
For linux users Dave Mckay has a great step-by-step walk through for setting up and using rclone on linux, give it a read.
Resources for Backup files with rclone
For more reading on rclone here is a list of resources I find very useful
https://gist.github.com/allanphilipbarku/03027025ccbcb7d75548f1589dd8597e
The post Backup files to Google Drive with Rclone appeared first on Anansewaa.
