August 11, 2020
There will be several upcoming changes made to the Python modules on RCC resources that are meant to improve the ability to manage and locate these modules and provide more consistency between versions. The Python module changes will be applied on September 2nd, 2020. All users of python modules (python, Anaconda3, and Anaconda2) should be aware of these changes and take any necessary steps if their python workflows are affected, prior to the restructure implementation date, in order to avoid any disruptions.
Please see the Python page of the User Guide for information on using Python on RCC resources.
The changes that will be made to the Python modules are enumerated as follows.
-
We will consolidate our naming scheme between the different versions and distributions of Python on the cluster. All Python modules will be named python, rather than having different named modules for PyPi and Anaconda Python, as is done now. To see the available Python modules users should use the python module name. This means users will no longer find modules named Anaconda3 or Anaconda2 on RCC resources. The module suffix for each python module will correspond to the distribution of python and its version. For example the Anaconda3 2020.02 python module has the name python/anaconda-2020.02. The PyPi or intel python version would be named python/pypy-<pyversion> and python/intel-<intel-version>, respectively.
-
Support for Python2 within the module system on the cluster will end. Community support for Python2 has already ended at the beginning of the year. If you still require use of a Python2 module after the restructure date, we can provide guidance on locally installing it for yourself. This does not apply to the non module system python installed in /bin/ which emains version 2.7.5.
-
We will support only two Anaconda releases of Python at any time. Each version will become deprecated after a two year period. The latest version available on RCC resources is python/anaconda-2020.02 which users can begin using immediately. We will support at least one version of intel Python at any given time (current version is python/intel-2020.up1).
-
We will keep an identical copy of a select set of environments in the supported python/anaconda-xxxx versions, so it should be trivial for users to switch to the latest version as it rolls out.
-
Common toolkits or specialty software currently managed through Anaconda environments (such as Tensorflow, Pytorch, and Deeplabcut) will be provided as separate modules of the same name as the software (e.g. module load pytorch), both to make them easier to find and also to make it easier to match versions with dependent libraries (e.g. cuda).
We address specifically some questions that users may have related to these changes below to further clarify how they may or may not affect them.
Q. What if I'm not ready for this change?
If you still need more time to adapt to the new module structure or update your workflows, you can run module load use.deprecated in order to get access to the old python modules. This will work until the end of the year.
Q. Does this affect me if I have my own Anaconda install?
No, if you maintain your own Anaconda install, this doesn't affect you. If you would like the maintain your own Anaconda install and don't know how, we can guide you through this.
Q. Does this affect me if I maintain my own Anaconda environments?
If you have your own local environments, then you should be able to activate it with any version of Anaconda without issue. If you want to be sure, you can always migrate your environment between Anaconda versions with the conda export or conda clone commands. You can find more information about that in our revamped python documentation.
If you have any questions, please direct them to help@rcc.uchicago.edu