Added initial version of dump-restore script. by ivanmrsulja · Pull Request #380 · vivo-project/Vitro

ivanmrsulja · 2023-03-17T15:59:25Z

What's new?

This functionality was only available through UI in admin panel, now, it is available through .sh and .bat script so that it can be setup as cron job with minimal configuration.

How should this be tested?

If you are willing to use the script the server has to be running. The CLI arguments are as follows:
bash dumpRestore.sh <action>

Action can be: dump or restore

For script to work, you have to configure parameters in setenv script. Because these parameters rarely change, I figured that someone can just edit few lines of the setenv script so that the command doesn't get cluttered with a lot of arguments:

models="CONTENT"                          # Models to be dumped/restored (CONTENT, CONFIGURATION etc...)
email="email@email.com"                   # Admin account email
password="password"                       # Admin account password

host="http://example:port"                # URL where the VIVO instance is hosted
purge="true"                              # Will the restoration process purge the models before restoring
restoration_files_path="."                # Directory containing the files used for restoration (restore files must be named as their corresponding models >> CONTENT.nq, CONFIGURATION.nq etc...)
dumped_files_path=$restoration_files_path # Directory containing the backed-up files
app_name="vivo"                           # app-name parameter

Interested parties

@chenejac @milospp

…ments.

litvinovg · 2023-03-20T17:02:47Z

I tried to dump data in linux environment, but it didn't work for me.
It prints that dump has been successfully created, but the file CONTENT.nq is empty
I suggest to add error checks and messages in case cookie wasn't returned and check return code of the dump request as well as the file size (not empty).
Also it might be useful to use date in output file name.

curl 7.61.1 (x86_64-redhat-linux-gnu) libcurl/7.61.1 OpenSSL/1.1.1k zlib/1.2.11 brotli/1.0.6 libidn2/2.2.0 libpsl/0.20.2 (+libidn2/2.2.0) libssh/0.9.6/openssl/zlib nghttp2/1.33.0

litvinovg · 2023-03-20T17:31:31Z

home/src/main/resources/scripts/dumpRestore.sh

+loginForm="Log in"
+
+# Log in and get session cookies
+curl --cookie-jar "$session/cookies.txt" -d "loginName=$loginName" -d "loginPassword=$loginPassword" -d "loginForm=$loginForm" $url


I think it should be --data-urlencode for login and password instead of -d as with -d "+" sign is interpreted as a space character

I have improved URL encoding as you advised, thank you for debugging the problem for me.

litvinovg · 2023-03-20T17:35:34Z

home/src/main/resources/scripts/dumpRestore.bat

+set loginForm=Log in
+
+:: Log in and get session cookies
+curl --cookie-jar "%session%\cookies.txt" --create-dirs -d "loginName=%loginName%" -d "loginPassword=%loginPassword%" -d "loginForm=%loginForm%" %url%


I think if you try to use login with + sign it wouldn't work here too. See comment below.

It is still -d here. I haven't tested it on Windows, but maybe we should also use --data-urlencode here?

You are right, I forgot to add it here for username and password. I will change it right now.

litvinovg · 2023-03-20T17:37:10Z

home/src/main/resources/scripts/dumpRestore.bat

+if "%action%" == "dump" (
+    echo Starting dump...
+
+    curl --cookie "%session%\cookies.txt" "%host%/%app_name%/dumpRestore/dump/%models%.nq?which=%models%" -o "%dumped_files_path%\%models%.nq"


Check return code (should be 200, not 302)
If return code is 302 then most likely authorization failed.
I would also check size of returned file. It shouldn't be empty.

I have added the necessary check, as you advised.

litvinovg · 2023-03-20T17:37:53Z

home/src/main/resources/scripts/dumpRestore.sh

+
+if [[ "$action" == "dump" ]]; then
+    echo "Starting dump..."
+    curl --cookie "$session/cookies.txt" "$host/$app_name/dumpRestore/dump/$models.nq?which=$models" -o "$dumped_files_path/$models.nq"


Check return code (should be 200, not 302)
If return code is 302 then most likely authorization failed.
I would also check size of returned file. It shouldn't be empty.

I have added the necessary check, as you advised.

litvinovg

I think login/password escaping should be fixed.
We should also check if dump failed and print some messages to inform user.

ivanmrsulja · 2023-03-21T09:21:48Z

I have improved the URL encoding and added necessary checks, as @litvinovg advised.

chenejac

@ivanmrsulja can we apply similar approach as Tomcat is using with setenv.sh (setenv.bat) file, meaning dumpRestore.bat and dumpRestore.sh might be renamed to vivo-admin.bat and vivo-admin.sh, and there might be additional files example.setenv.sh and example.setenv.bat? All scripts parameters might be specified in those setenv.sh/bat files except action. The files might stay at the script directory in the VIVO code base, but can be copied to VIVO_HOME/bin by using this file - https://github.com/vivo-project/Vitro/blob/main/home/src/main/assembly/home.xml

…figuring functionality on linux.

…tory.

chenejac

@ivanmrsulja it works for me (Windows 10). I have tried to create dump of content graph and to restore. Thanks for the contribution.

hauschke · 2025-01-20T09:49:09Z

In case this is accepted, this solution should be included in https://wiki.lyrasis.org/display/VIVODOC115x/Backup+and+Restore, right?

chenejac · 2025-01-21T07:42:04Z

In case this is accepted, this solution should be included in https://wiki.lyrasis.org/display/VIVODOC115x/Backup+and+Restore, right?

Right. This PR will be considered to be merged once jena is upgraded to 5 (or at least 4), hopefully in VIVO 1.16.x

bkampe

Installed and tested. Works like expected.

ivanmrsulja added 3 commits March 17, 2023 16:38

Added initial dump-restore script.

0f8df35

Changed email and password to get their values from command line argu…

d6c514d

…ments.

Added .bat script for dump-restore functionality.

a9ede4c

chenejac requested review from a user, brianjlowe and litvinovg March 20, 2023 12:32

ivanmrsulja added 2 commits March 20, 2023 15:30

Slight refactor of variable names and initialization.

399bd18

Removed testing value for host variable.

06578b4

litvinovg reviewed Mar 20, 2023

View reviewed changes

litvinovg requested changes Mar 20, 2023

View reviewed changes

Added necessary checks and fixed url encoding.

daee759

ivanmrsulja requested review from litvinovg and removed request for a user and brianjlowe March 21, 2023 09:21

chenejac requested review from a user and brianjlowe March 21, 2023 09:26

ivanmrsulja added 2 commits March 21, 2023 12:43

Added --data-urlencode option when logging in via .bat script.

9b498fb

Renamed variables to match snake-case convention.

6772b47

chenejac linked an issue Apr 6, 2023 that may be closed by this pull request

Making scripts for live backup of TDB vivo-project/VIVO#3819

Open

3 tasks

chenejac self-requested a review April 10, 2023 10:15

chenejac requested changes Apr 10, 2023

View reviewed changes

ivanmrsulja added 4 commits April 11, 2023 14:56

Renamed scripts to vivo-admin. Added example.setenv.sh script for con…

6748039

…figuring functionality on linux.

Added example.setenv.bat configuration script.

cfe9d48

Updated home.xml to copy scripts/ directory into vivo-home/bin/ direc…

5581fd3

…tory.

Fixed bug when comments are parsed as CLI arguments.

d3d8126

ivanmrsulja added 2 commits April 17, 2023 12:56

Fixed bug when performing restoration.

3fbcb57

Removed unneeded quotation marks.

f095b44

chenejac approved these changes Apr 17, 2023

View reviewed changes

bkampe approved these changes Feb 11, 2025

View reviewed changes

Comments

Conversation

ivanmrsulja commented Mar 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's new?

How should this be tested?

Interested parties

Uh oh!

litvinovg commented Mar 20, 2023

Uh oh!

litvinovg Mar 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ivanmrsulja Mar 21, 2023

Choose a reason for hiding this comment

Uh oh!

litvinovg Mar 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

litvinovg Mar 21, 2023

Choose a reason for hiding this comment

Uh oh!

ivanmrsulja Mar 21, 2023

Choose a reason for hiding this comment

Uh oh!

litvinovg Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

ivanmrsulja Mar 21, 2023

Choose a reason for hiding this comment

Uh oh!

litvinovg Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

ivanmrsulja Mar 21, 2023

Choose a reason for hiding this comment

Uh oh!

litvinovg left a comment

Choose a reason for hiding this comment

Uh oh!

ivanmrsulja commented Mar 21, 2023

Uh oh!

chenejac left a comment

Choose a reason for hiding this comment

Uh oh!

chenejac left a comment

Choose a reason for hiding this comment

Uh oh!

hauschke commented Jan 20, 2025

Uh oh!

chenejac commented Jan 21, 2025

Uh oh!

bkampe left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ivanmrsulja commented Mar 17, 2023 •

edited

Loading

litvinovg Mar 20, 2023 •

edited

Loading

litvinovg Mar 20, 2023 •

edited

Loading