Archive files from Wexac to storWis

Owned by Bareket Dassa
Last updated: 11 Jan, 2024
1 min read

  1. login through an access node (i.e. access4.wexac.weizmann.ac.il) with your credentials.

“StorWIS isn’t intended as an HPC storage service, and we are not allowing heavy computation on access servers. So if you need to run a job with data from StorWIS, you need to move/copy from StorWIS to your WEXAC home folder.”

  1. Create your folder in /storwis/Labs/YOUR_LAB/
    i.e. /storwis/Labs/bioservices/

  2. Use this command to archive a directory into a tar file, and to save it in your storwis folder.
    Example:

tar czf - DIRECTORY_NAME |  split --bytes=100G - /storwis/Labs/YOUR_LAB/DIRECTORY_NAME.tar.gz.

Change DIRECTORY_NAME to the directory you need to archive and YOUR_LAB to your lab name.

It will create tar files with the names DIRECTORY_NAME.tar.gz.aa, DIRECTORY_NAME.tar.gz.ab, …
each file will have size of 100GB.

To list the contents of the tar files, run:

cat /storwis/Labs/YOUR_LAB/DIRECTORY_NAME.tar.gz.* | tar tzf -

To extract the folder to the current directory, run:

cat /storwis/Labs/YOUR_LAB/DIRECTORY_NAME.tar.gz.* | tar xzvf -

 

Important notes:

  • archiving must be done from an “access” server

  • Some folders should be archived without “zip” (tar cf instead of tar czf) because they already contain zipped files

  • archiving does not always work well with the bsub command

  • Some file permissions can be problematic for specific files which were created by the sequencing server, or UTAP

 

More info in the Wexac page: https://hpcwiki.weizmann.ac.il/en/home/general/backup_restore