Shell Environment on the Cluster#
The Discovery Shell environment and .bashrc#
Discovery uses a Linux-based Operating System (CentOS), where the Shell program is used to interface with the user. Bash (Bourne Again SHell) is one of the most popular Shell implementations which is the default Shell on Discovery.
The Shell script .bashrc
is used by Bash to initialize your Shell environment. For example, it is typically used to define aliases, functions and load modules. Note that environment variables settings (such as PATH
) generally go in the .bash_profile
or .profile
files.
Your .bashrc
, .bash_profile
and .profile
files live in your $HOME
directory. You can make changes to your .bashrc with a text editor like nano.
Caution
Making edits to your .bashrc
file can result in many issues. Some changes may prevent you from launching apps or executing commands. Modifying your PATH
variable may result in the inability to use basic Shell commands (such as cd
or ls
) if not done correctly.
Before making changes to your .bashrc
file, make a backup of the default .bashrc
file, so you can restore it if necessary.
If you need help with editing .bashrc
, reach out to mailto:rchelp@northeastern.edu or schedule a consultation with
a staff member
who can help suggest edits and troubleshoot any issues you might be having.
About your .bashrc file#
You have a default .bashrc
file in your home directory when your account is created. See the figure below for an example of a default .bashrc
file.
Important
We recommend to keep .bashrc unchanged when using Discovery. You can source environment Shell scripts or load modules directly inside your job instead. This approach can prevent some runtime errors from loading incompatible modules, setting environment variables incorrectly, or from mixing multiple software and Conda environments.
Conda and .bashrc
#
In addition to editing your .bashrc
file as outlined in the example above, programs that you install can also modify your .bashrc file. For example, if you follow the procedure outlined in Working with a Miniconda environment, there may be a section added to your .bashrc
file (if you didn’t use the -b
batch option) that automatically loads your conda environment every time you sign in to Discovery. See the figure below for an example of this:
You should not modify this section in the .bashrc
file directly. If it was changed, remove this section manually using a file editor.
Caution
We recommend removing the conda initialization section from your .bashrc
as it may interfere with the correct startup environment when using Open OnDemand apps. You should always load your Conda environment after your job already started.
If you need help with your .bashrc file or would like it restored to its default, reach out to the RC team at mailto:rchelp@northeastern.edu, and we can provide you with
a new, default .bashrc
file and help troubleshoot issues with the file.
Editing your .bashrc
file#
The basic workflow for editing your .bashrc
file is to sign in to Discovery, go to your $HOME
directory, open the file in a text editor on the command line, make your edits, save the file, sign out of Discovery, then sign back in again. Your changes will take effect when you have signed back in again.
Example procedure for editing your .bashrc
file:
Sign in to Discovery.
(Optional) Type
pwd
to make sure you are in your/home
directory.(Optional) Type
ls -a
to view the contents of your/home
directory, including hidden files. Your .bashrc file is a hidden file (hidden files are preceded by a.
). Using the-a
option withls
displays hidden files.(Recommended) Type
cp .bashrc .bashrc-default
to make a copy of your .bashrc file called.bashrc-default
.Type
nano .bashrc
to open your .bashrc file in the nano text editor.Type the edits that you want to make to your file. In this example, an alias was added to create a shortcut to the user’s
/scratch
space.Save the file and exit the editor.
Sign out of Discovery and sign back in for the changes to take effect.
Sourcing a Shell script example#
A safe alternative to using .bashrc
is to source a Shell script inside your runtime job environment. Below is an example script to load an Anaconda module and source a Conda environment which will then be used inside the Slurm script.
Create a Shell script myenv.bash
:
#!/bin/bash
module load anaconda3/2021.05
module load cuda/11.1
source activate pytorch_env_training
Then, source the Shell script inside your sbatch Slurm script (see Using sbatch):
#!/bin/bash
#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --gres=gpu:1
#SBATCH --time=01:00:00
#SBATCH --job-name=gpu_run
#SBATCH --mem=4GB
#SBATCH --ntasks=1
#SBATCH --output=myjob.%j.out
#SBATCH --error=myjob.%j.err
source myenv.bash
python <myprogram>