Set Up Remote Server for Deep Learning

Install Pyenv

[!tip] Manage multiple versions of Python with pyenv.

Update System Packages

sudo apt update

Install optional dependencies with

sudo apt install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

sudo apt install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python-openssl git

according to OpenAI's gpt-4o-2024-05-13.

Install Pyenv

curl https://pyenv.run | bash

Set Up Environment Variables

if [ -n "$BASH_VERSION" ]; then
  # Load pyenv automatically by appending the following to
  # ~/.bash_profile if it exists, otherwise ~/.profile (for login shells)
  # and ~/.bashrc (for interactive shells) :
  echo -e 'export PYENV_ROOT="$HOME/.pyenv"\n[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
  echo -e 'eval "$(pyenv init -)"' >> ~/.bashrc
  echo "Added pyenv virtualenv-init to ~/.bashrc"
  # Restart your shell for the changes to take effect.
  # Load pyenv-virtualenv automatically by adding the following to ~/.bashrc:
  echo -e 'eval "$(pyenv virtualenv-init -)"' >> ~/.bashrc
  echo "Added pyenv virtualenv-init to ~/.bashrc"
  # Don't forget to restart your shell for the changes to take effect.
  source ~/.bashrc
elif [ -n "$ZSH_VERSION" ]; then
  # Running in Zsh
  echo -e 'export PYENV_ROOT="$HOME/.pyenv"\n[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
  echo -e 'eval "$(pyenv init -)"' >> ~/.zshrc
  echo "Added pyenv virtualenv-init to ~/.zshrc"
  echo -e 'eval "$(pyenv virtualenv-init -)"' >> ~/.zshrc
  echo "Added pyenv virtualenv-init to ~/.zshrc"
  source ~/.zshrc
else
  # Unknown shell
  echo "Unsupported shell."
fi

Install Python & Set Up Virtual Environment with Pyenv

Check available versions

pyenv install --list

Install specific version

pyenv install 3.10.14

Set global version with

pyenv global 3.10.14

or just assign to current shell with

pyenv shell 3.10.14

Install pyenv-virtualenv

git clone https://github.com/pyenv/pyenv-virtualenv.git ~/.pyenv/plugins/pyenv-virtualenv

Update shell configuration:

eval "$(pyenv virtualenv-init -)"

Restart your shell:

exec $SHELL

Create a new virtual environment:

pyenv virtualenv 3.10.14 myenv

Activate the virtual environment:

pyenv activate myenv

Install Packages with pip

install packages with pip using mirrors

For one time use, install packages with mirror address

pip install <package> -i https://pypi.tuna.tsinghua.edu.cn/simple
# pip install <package> -i http://mirrors.aliyun.com/pypi/simple --truste-host mirrors.aliyun.com

For global set up, create file ~/.pip/pip.conf with the following content.

[global]
index-url=https://pypi.tuna.tsinghua.edu.cn/simple
[install]
trusted-host=https://pypi.tuna.tsinghua.edu.cn

Create folder ~/.pip/ if it does not exist.

$ pip config list
global.index-url='https://pypi.tuna.tsinghua.edu.cn/simple'
install.trusted-host='https://pypi.tuna.tsinghua.edu.cn'

[!tip] Some other significant mirrors

中国科学技术大学 : https://pypi.mirrors.ustc.edu.cn/simple

豆瓣：http://pypi.douban.com/simple/

阿里云：http://mirrors.aliyun.com/pypi/simple/

install packages from setup.py

The install_requires list in your setup.py file specifies the core dependencies that will always be installed.
The extras_require dictionary specifies optional dependencies that are only installed if explicitly requested (e.g., pip install -e ".[dev]").

set(
    ......
    extras_require=extras,
    python_requires=">=3.10.9",
    install_requires=install_requires,
)

Python Package Management with Poetry

install poetry and add to PATH

curl -sSL https://install.python-poetry.org | python3 -
echo 'export PATH=$PATH:$HOME/.local/bin' >> ~/.bashrc
source ~/.bashrc

additional setup

I prefer to use virtual environment under the current project directory.

poetry config virtualenvs.in-project true

use poetry config --list to check current settings.

create new project

cd my-project
poetry init

poetry env use python
# poetry env use /Users/caijiaqi/.pyenv/versions/3.10.14/bin/python

[!tip] Note that the python version is determined by python so choose the desired one.

install dependencies

poetry add <package>

Two files, pyproject.toml and poetry.lock, are created and updated accordingly by now. When you use poetry add command, Poetry will automatically do the following three things in this order:

update pyproject.toml
update poetry.lock according to pyproject.toml
update virtual environment according to poetry.lock

[!caution] Do not manually edit poetry.lock and pyproject.toml. If you manually edited pyproject.toml, run poetry lock to update poetry.lock file.

Python 套件管理器——Poetry 完全入門指南

Git Clone Projects for Remote Development

Check out [[git#Clone Private GitHub Repo]].

Run Scripts in the Background

nohup [env key=val] script > output.log 2>&1 &

Set Up Remote Server for Deep Learning

Install Pyenv

Update System Packages

Install Pyenv

Set Up Environment Variables

Install Python & Set Up Virtual Environment with Pyenv

Install Packages with pip

install packages with pip using mirrors

install packages from setup.py

Python Package Management with Poetry

install poetry and add to PATH

additional setup

create new project

install dependencies

read more

Git Clone Projects for Remote Development

Run Scripts in the Background

On this page