Sharing Project Files on TACC Systems
Last update: June 8, 2023
Collaborators on the same project/allocation often wish to share code, data or other project files with each other, but not necessarily with the world. Users with Educational allocations may wish to have a repository accessible to their students. This page will instruct allocation managers and their delegates how to set up a project workspace that is accessible only to users in the same allocation.
All TACC system users are organized into UNIX "groups", collections of users who typically share the same permissions: read, write, execute or some combination thereof, including none, on a set of files or directories. Groups and group membership are created and assigned by a TACC system administrator upon user account creation. A user may belong to many groups but a file or directory is owned by only one owner and one group. For files and directories to be shared among a collection of users:
- Those users and files must belong to the same UNIX group
- The file or directory's permissions must allow group read or write access
At TACC, users assigned to the same allocated project typically belong to the same UNIX group. This group number will (usually) correspond directly to the project/allocation number. To determine your project's UNIX group number, go to Projects and Allocations via your TACC Dashboard.
Click on the "Project Detail" button to view the group number:
In this case, Test Project XYZ's UNIX group ID (GID) is "G-816631". Therefore, all files to be associated with this project must belong to this group.
The UNIX command
groups displays all groups a user belongs to:
login1$ groups slindsey G-40300 G-80748 G-80906 G-801508 G-803450 G-803454 G-813602 G-816631
The first group listed, in this case
G-40300, is the user's primary or default group,
In the above output, the user
slindsey's primary or default group is G-40300, meaning that any files or directories this user creates will automatically belong to this group. However, Test Project XYZ's UNIX group is G-816631 as determined above.
To display a file's owner and group membership, use the
ls -l command:
login1$ ls -l myfile -rw------- 1 slindsey G-40300 983 Nov 13 10:40 myfile
In the above output the file
myfile is owned by user
slindsey and belongs to the
G-40300 project/group. This file's permissions are set to read and write,
rw, for the owner,
Please consult the UNIX man pages for more information on these commands:
login1$ man groups login1$ man chgrp
It is not possible to make a shared, writable directory under a user's
$HOME directory and the
$SCRATCH file system is subject to periodic purging. Therefore, TACC staff strongly recommends placing all files to be shared in the top level of the user's area of the
/work filesystem, defined in the
$STOCKYARD environment variable. This new shared directory will be accessible only to members of the unix group and by extension the project members.
$STOCKYARD points to the highest level directory you own on Stockyard, TACC's Global Shared File System, which is mounted and available across all TACC systems. Your
$WORK environment variable points to a resource-specific eponymous subdirectory of
$STOCKYARD. For example on Stampede2,
$WORK is defined as
$STOCKYARD/stampede2. All subdirectories are accessible to you on any TACC system where you have an allocation.
$STOCKYARDdirectory accessible, though not readable, to the group members:
login1$ chmod g+X $STOCKYARD
Go to your work directory on Stockyard and create the sub-directory to be shared:
login1$ cd $STOCKYARD; mkdir mysharedirectory
The directory with shared files then needs to belong to the group. This step may be redundant if the directory already belongs to the group:
login1$ chgrp -R G-816631 mysharedirectory
Set the shared directory's ownership and permissions. In this example, permissions will be set to owner:read, write, execute; group:read, execute, and no access to any users outside the group.
a. Set the group id bit. Any new files created in the shared directory will inherit the group ownership:
login1$ chmod g+s mysharedirectory
b. Then edit your
.bashrcand set the umask variable to "027". This ensures that all NEW files created will inherit the proper permissions.
c. Last, set permissions to readable and accessible to group members:
login1$ chmod g+rX mysharedirectory
(optional) Grant write access to the directory. The project manager may wish to grant project collaborators write access to the directory or leave the directory as read-only.
login1$ chmod g+rwX mysharedirectory
Project members will now have read and write, or read-only access to this directory and its contents.
Project managers should remind their users to set their shell's umask variable allowing other users to see their created files.
login1$ set umask 027
Many of these commands can be tricky. If you need assistance in setting up a shared project workspace, please submit a help-desk ticket. See also the following man pages:
login1$ man chmod login1$ man chgrp login1$ man umask
Principal Investigators (PIs) or their delegates can manage project membership through the TACC User Portal under Allocations.