My first real continuous deployment

My first real continuous deployment was about three years ago. At that point of time, I didn't associate the term "continuous deployment" with what I built and I knew much less then, than I do now.

The code isn't a success story. The ideas I grew from here are a success story.

The code and process in this aren't great, and there's been some redactions to remove sensitive usernames and passwords. Throughout the code if you see "< description >" then some sensitive piece of information used to be there.

The setup

I had a server on Linode. On this server I was hosting an API and several static websites (HTML + CSS only).

Somewhere along the way I found Linode StackScripts to script out server setup with bash scripts.

Through trial and error, I eventually pieced together something that did what I wanted from scripts I found on their site. The top comments on the script were used to parameterized the script.

#!/bin/bash
#
# Installs a node.js stack backed by PostgreSQL and fronted by nginx.
# <UDF name="user_username" Label="Main user name" />
# <UDF name="user_password" Label="Main user password" />
# <UDF name="USER_EMAIL"  Label="LetsEncrypt Email" />
# <UDF name="user_ssh_key"  Label="Main user RSA SSH key" />
#
# If not deploying to Linode, set these environment variables. All required:
#  - USER_USERNAME: the username you want for your main Linux user
#  - USER_PASSWORD: the password for this ^ user
#  - USER_EMAIL:  The email for LetsEncrypt
#  - USER_SSH_KEY: the contents of an SSH public key. SSH will be locked out of password auth.

LOGFILE=~/stackscript.log
USER_HOMEDIR=/home/$USER_USERNAME

#update to latest everything
apt-get update
apt-get -y upgrade

# set default shell to bash
sed -i '' -e "s|SHELL=/bin/sh|SHELL=/bin/bash|g" /etc/default/useradd;

# setup user
echo "Creating account username $USER_USERNAME" >> $LOGFILE
mkdir -p $USER_HOMEDIR;
useradd -p `openssl passwd -1 $USER_PASSWORD` -d $USER_HOMEDIR $USER_USERNAME
echo "User $USER_USERNAME created" >> $LOGFILE
usermod $USER_USERNAME -aG admin
echo "User $USER_USERNAME added to group admin" >> $LOGFILE
echo "$USER_USERNAME    ALL=(ALL) ALL" >> /etc/sudoers
chmod 400 /etc/sudoers
echo "User $USER_USERNAME added to sudoers" >> $LOGFILE

# install SSH key
echo "Starting SSH Key" >> $LOGFILE
mkdir -p $USER_HOMEDIR/.ssh;
echo "$USER_SSH_KEY" > $USER_HOMEDIR/.ssh/authorized_keys
sed -i '' -e "s/\#PasswordAuthentication yes/PasswordAuthentication no/g" /etc/ssh/sshd_config
sed -i '' -e "s/PermitRootLogin yes/PermitRootLogin no/g" /etc/ssh/sshd_config
chown -R 700 $USER_HOMEDIR/.ssh;
chown 644 $USER_HOMEDIR/.ssh/authorized_keys;
echo "Done SSH Key" >> $LOGFILE
apt-get install -y git-core build-essential autoconf libssl-dev
pushd $USER_HOMEDIR;
echo "Pushed into $PWD" >> $LOGFILE;
mkdir -p packages;
pushd packages;
echo "Pushed into $PWD" >> $LOGFILE;

# restart SSH
service ssh restart

#install ufw
echo "Setting up UFW" >> $LOGFILE
apt-get install ufw
ufw disable
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow https
ufw allow http
ufw enable
echo "Done UFW" >> $LOGFILE

#install NGinx
echo "Setting up Nginx" >> $LOGFILE
apt-get install -y nginx;
service nginx restart;
echo "Done Nginx" >> $LOGFILE

#install git
echo "Installing Git" >> $LOGFILE
apt-get install git

#install nodejs
echo "Installing NodeJS" >> $LOGFILE
curl -sL https://deb.nodesource.com/setup_8.x | sudo bash -
sudo apt-get install nodejs

#setup PM2
echo "Installing PM2" >> $LOGFILE
npm install pm2 -g
echo "Startup PM2" >> $LOGFILE
pm2 startup -u $USER_USERNAME --hp $USER_HOMEDIR

#setup from bitbucket repo
currentDir = $(pwd)
echo "Pulling from bitbucket repo" >> $LOGFILE
cd /opt
git clone https://<redacted username>:<redacted password>@bitbucket.org/<redacted username>/morgemil_server.git
chmod -R +rx morgemil_server  #permissions are the worst!
cd morgemil_server
git checkout master
git config core.filemode false
chmod -R +rx .  #permissions are the worst!

#resume previous directory
cd $currentDir

#save current directory
currentDir = $(pwd)

#install letsencrypt
echo "Installing Letsencrypt" >> $LOGFILE
git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt
cd /opt/letsencrypt
./certbot-auto certonly --agree-tos --no-eff-email --non-interactive --nginx --email $USER_EMAIL -d morgemil.com -d www.morgemil.com
echo '@weekly root cd /opt/letsencrypt && git pull >> /var/log/letsencrypt/letsencrypt-auto-update.log' | sudo tee --append /etc/crontab
echo "Done Letsencrypt" >> $LOGFILE

#resume previous directory
cd $currentDir
echo "Appending bitbucket setup to cron job" >> $LOGFILE
echo '* * * * * root flock -xn /tmp/morgemil_poll.lck -c /opt/morgemil_server/poll.sh' | sudo tee --append /etc/crontab

#cleanup
chown -R $USER_USERNAME:$USER_USERNAME $USER_HOMEDIR;
echo "Done!"  >> $LOGFILE

I think that the last time I updated it was 2017. I knew it wasn't great, but I kept updating it anyway.

The Poll

During setup, the bitbucket repository would be downloaded. In this repository, I had a script called poll.sh and this script would be called by a cronjob once a minute.

#!/bin/bash
cd /opt/morgemil_server
#each repository is reponsible for updating itself indpendently.
./gitcheck.sh 
if [ $? -eq 0 ]; then
    ./setup.sh
fi
./morgemil_install.sh
./olivercoding_install.sh

The polling logic was simple enough.

  1. Change working directories to git repository root.
  2. Refresh the repository.
  3. Call setup.sh if changes occurred in the repository.
  4. Call each website's personal script to do whatever the heck it does.

gitcheck

!/bin/bash

git remote update

UPSTREAM=${1:-'@{u}'}
LOCAL=$(git rev-parse @)
REMOTE=$(git rev-parse "$UPSTREAM")
BASE=$(git merge-base @ "$UPSTREAM")

if [ $LOCAL = $REMOTE ]; then
    echo "Up-to-date"
    exit 1
elif [ $LOCAL = $BASE ]; then
    git reset --hard
    git pull
    chmod -R +rx .
    echo "Need to pull"
    exit 0
elif [ $REMOTE = $BASE ]; then
    git reset --hard
    git pull
    chmod -R +rx .
    echo "Need to push"
    exit 0
else
    git reset --hard
    git pull
    chmod -R +rx .
    echo "Diverged"
    exit 0
fi

Sample website logic

Over time, I found some common patterns and started abstracting logic, such as letsencrypt. My example here is my personal website, this very blog used to run on that server back then.

The install script for this blog was another piece in that patchwork quilt. Copying html files around, copying nginx configuration, and then also calling yet another script to setup letsencrypt if it wasn't already setup.

#!/bin/bash

serverRepo="/opt/morgemil_server/"
olivercodingSourceDir="${serverRepo}olivercoding/placeholder/"
olivercodingTargetDir="/var/www/olivercoding.com/public_html"
olivercodingBaseTargetDir="/var/www/olivercoding.com"

if [ ! -d $olivercodingTargetDir ]; then
    #if directory doesn't exist, setup permissions and user.
    #TODO: remove hardcoded user.
    mkdir -p $olivercodingTargetDir
    chown -R  morgemil:morgemil $olivercodingBaseTargetDir
    chmod 755 /var/www
fi

#REGION ---- SETUP LetsEncrypt
insecureNginxSource="${serverRepo}olivercoding/insecure_olivercoding.com"
secureNginxSource="${serverRepo}olivercoding/secure_olivercoding.com"

${serverRepo}letsencrypt_setup.sh "olivercoding.com" $insecureNginxSource $secureNginxSource
#END REGION ---- SETUP LetsEncrypt

#REGION ---- SETUP olivercoding.com build
#only change if index.html is different.  will add more expansive site later.
diff -q $"${olivercodingTargetDir}/index.html" "${olivercodingSourceDir}index.html"
if [ $? -ne 0 ]; then
    rsync -rv $olivercodingSourceDir $olivercodingTargetDir --delete
    echo "$(date) is the last time the olivercoding repository changed." >> ~/log.txt
fi
#END REGION ---- SETUP olivercoding.com build

Generic letsencrypt setup

#!/bin/bash

websiteName=$1
insecureNginx=$2
secureNginx=$3
wwwWebsiteName="www.${websiteName}"
websitePublicHtml="/var/www/${websiteName}/public_html"
nginxAvailable="/etc/nginx/sites-available/${websiteName}"
nginxEnabled="/etc/nginx/sites-enabled/${websiteName}"
letsencryptLive="/etc/letsencrypt/live/${websiteName}/"

if [ -d $letsencryptLive ]; then

    diff -q $nginxAvailable $secureNginx
    if [ $? -ne 0 ]; then
        #Only copy new server setup and restart server if file is different.
        cp -T $secureNginx $nginxAvailable
        service nginx restart
        echo "$(date) is the last time the ${websiteName} NGinx settings were changed." >> ~/log.txt
    fi
else
    #Remove letsencrypt from crontab for morgemil.com. Little scared about this one.
    #little scared about this one.
    grep -v " $websiteName" /etc/crontab >> /etc/crontab
    cp -T $insecureNginx $nginxAvailable
    ln -s $nginxAvailable $nginxEnabled
    
    indexFile = "${websitePublicHtml}/index.html"
    if [ ! -f $indexFile ]; then
        #dummy file for letsencrypt to find.
        echo "hello">>$indexFile
    fi
    service nginx restart

    #assume that letsencypt successfully exists
    pushd /opt/letsencrypt
    #TODO: refactor email variables: < email redacted >
    ./certbot-auto certonly --agree-tos --no-eff-email --non-interactive --nginx --email < email redacted > -d $websiteName -d $wwwWebsiteName
    echo "@monthly root /opt/letsencrypt/letsencrypt-auto certonly --quiet --standalone --renew-by-default -d $websiteName -d $wwwWebsiteName >> /var/log/letsencrypt/letsencrypt-auto-update.log" | sudo tee --append /etc/crontab    
    popd

    cp -T $secureNginxSource $morgemileNginxDestination
    service nginx restart

    echo "$(date) is the last time the $websiteName LetsEncrypt were changed." >> ~/log.txt
fi

Commentary

I had quite a few websites on this server at one point, and the logic in there got truly heinous. However, I'm proud of this.

I know the security was terrible. I know the programming practices were brittle and fragile. It was a product of ignorance. But it led me to be better.

I could make website changes, add scripts, add install logic, and the polling logic would download changes and make them happen. Though I certainly spent a lot of time SSHing into the server and debugging...

Summary

I've learned a lot since I wrote the above travesty, deployed it, operated it, and then shut it down. It sure is fun to read past code and to remember where I came from.