My first real continuous deployment

My first real continuous deployment was about three years ago. At that point of time, I didn't associate the term "continuous deployment" with what I built and I knew much less then, than I do now.

The code isn't a success story. The ideas I grew from here are a success story.

The code and process in this aren't great, and there's been some redactions to remove sensitive usernames and passwords. Throughout the code if you see "< description >" then some sensitive piece of information used to be there.

The setup

I had a server on Linode. On this server I was hosting an API and several static websites (HTML + CSS only).

Somewhere along the way I found Linode StackScripts to script out server setup with bash scripts.

Through trial and error, I eventually pieced together something that did what I wanted from scripts I found on their site. The top comments on the script were used to parameterized the script.

# Installs a node.js stack backed by PostgreSQL and fronted by nginx.
# <UDF name="user_username" Label="Main user name" />
# <UDF name="user_password" Label="Main user password" />
# <UDF name="USER_EMAIL"  Label="LetsEncrypt Email" />
# <UDF name="user_ssh_key"  Label="Main user RSA SSH key" />
# If not deploying to Linode, set these environment variables. All required:
#  - USER_USERNAME: the username you want for your main Linux user
#  - USER_PASSWORD: the password for this ^ user
#  - USER_EMAIL:  The email for LetsEncrypt
#  - USER_SSH_KEY: the contents of an SSH public key. SSH will be locked out of password auth.


#update to latest everything
apt-get update
apt-get -y upgrade

# set default shell to bash
sed -i '' -e "s|SHELL=/bin/sh|SHELL=/bin/bash|g" /etc/default/useradd;

# setup user
echo "Creating account username $USER_USERNAME" >> $LOGFILE
mkdir -p $USER_HOMEDIR;
useradd -p `openssl passwd -1 $USER_PASSWORD` -d $USER_HOMEDIR $USER_USERNAME
echo "User $USER_USERNAME created" >> $LOGFILE
usermod $USER_USERNAME -aG admin
echo "User $USER_USERNAME added to group admin" >> $LOGFILE
echo "$USER_USERNAME    ALL=(ALL) ALL" >> /etc/sudoers
chmod 400 /etc/sudoers
echo "User $USER_USERNAME added to sudoers" >> $LOGFILE

# install SSH key
echo "Starting SSH Key" >> $LOGFILE
mkdir -p $USER_HOMEDIR/.ssh;
echo "$USER_SSH_KEY" > $USER_HOMEDIR/.ssh/authorized_keys
sed -i '' -e "s/\#PasswordAuthentication yes/PasswordAuthentication no/g" /etc/ssh/sshd_config
sed -i '' -e "s/PermitRootLogin yes/PermitRootLogin no/g" /etc/ssh/sshd_config
chown -R 700 $USER_HOMEDIR/.ssh;
chown 644 $USER_HOMEDIR/.ssh/authorized_keys;
echo "Done SSH Key" >> $LOGFILE
apt-get install -y git-core build-essential autoconf libssl-dev
echo "Pushed into $PWD" >> $LOGFILE;
mkdir -p packages;
pushd packages;
echo "Pushed into $PWD" >> $LOGFILE;

# restart SSH
service ssh restart

#install ufw
echo "Setting up UFW" >> $LOGFILE
apt-get install ufw
ufw disable
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow https
ufw allow http
ufw enable
echo "Done UFW" >> $LOGFILE

#install NGinx
echo "Setting up Nginx" >> $LOGFILE
apt-get install -y nginx;
service nginx restart;
echo "Done Nginx" >> $LOGFILE

#install git
echo "Installing Git" >> $LOGFILE
apt-get install git

#install nodejs
echo "Installing NodeJS" >> $LOGFILE
curl -sL | sudo bash -
sudo apt-get install nodejs

#setup PM2
echo "Installing PM2" >> $LOGFILE
npm install pm2 -g
echo "Startup PM2" >> $LOGFILE
pm2 startup -u $USER_USERNAME --hp $USER_HOMEDIR

#setup from bitbucket repo
currentDir = $(pwd)
echo "Pulling from bitbucket repo" >> $LOGFILE
cd /opt
git clone https://<redacted username>:<redacted password><redacted username>/morgemil_server.git
chmod -R +rx morgemil_server  #permissions are the worst!
cd morgemil_server
git checkout master
git config core.filemode false
chmod -R +rx .  #permissions are the worst!

#resume previous directory
cd $currentDir

#save current directory
currentDir = $(pwd)

#install letsencrypt
echo "Installing Letsencrypt" >> $LOGFILE
git clone /opt/letsencrypt
cd /opt/letsencrypt
./certbot-auto certonly --agree-tos --no-eff-email --non-interactive --nginx --email $USER_EMAIL -d -d
echo '@weekly root cd /opt/letsencrypt && git pull >> /var/log/letsencrypt/letsencrypt-auto-update.log' | sudo tee --append /etc/crontab
echo "Done Letsencrypt" >> $LOGFILE

#resume previous directory
cd $currentDir
echo "Appending bitbucket setup to cron job" >> $LOGFILE
echo '* * * * * root flock -xn /tmp/morgemil_poll.lck -c /opt/morgemil_server/' | sudo tee --append /etc/crontab

echo "Done!"  >> $LOGFILE

I think that the last time I updated it was 2017. I knew it wasn't great, but I kept updating it anyway.

The Poll

During setup, the bitbucket repository would be downloaded. In this repository, I had a script called and this script would be called by a cronjob once a minute.

cd /opt/morgemil_server
#each repository is reponsible for updating itself indpendently.
if [ $? -eq 0 ]; then

The polling logic was simple enough.

  1. Change working directories to git repository root.
  2. Refresh the repository.
  3. Call if changes occurred in the repository.
  4. Call each website's personal script to do whatever the heck it does.



git remote update

LOCAL=$(git rev-parse @)
REMOTE=$(git rev-parse "$UPSTREAM")
BASE=$(git merge-base @ "$UPSTREAM")

if [ $LOCAL = $REMOTE ]; then
    echo "Up-to-date"
    exit 1
elif [ $LOCAL = $BASE ]; then
    git reset --hard
    git pull
    chmod -R +rx .
    echo "Need to pull"
    exit 0
elif [ $REMOTE = $BASE ]; then
    git reset --hard
    git pull
    chmod -R +rx .
    echo "Need to push"
    exit 0
    git reset --hard
    git pull
    chmod -R +rx .
    echo "Diverged"
    exit 0

Sample website logic

Over time, I found some common patterns and started abstracting logic, such as letsencrypt. My example here is my personal website, this very blog used to run on that server back then.

The install script for this blog was another piece in that patchwork quilt. Copying html files around, copying nginx configuration, and then also calling yet another script to setup letsencrypt if it wasn't already setup.



if [ ! -d $olivercodingTargetDir ]; then
    #if directory doesn't exist, setup permissions and user.
    #TODO: remove hardcoded user.
    mkdir -p $olivercodingTargetDir
    chown -R  morgemil:morgemil $olivercodingBaseTargetDir
    chmod 755 /var/www

#REGION ---- SETUP LetsEncrypt

${serverRepo} "" $insecureNginxSource $secureNginxSource
#END REGION ---- SETUP LetsEncrypt

#REGION ---- SETUP build
#only change if index.html is different.  will add more expansive site later.
diff -q $"${olivercodingTargetDir}/index.html" "${olivercodingSourceDir}index.html"
if [ $? -ne 0 ]; then
    rsync -rv $olivercodingSourceDir $olivercodingTargetDir --delete
    echo "$(date) is the last time the olivercoding repository changed." >> ~/log.txt
#END REGION ---- SETUP build

Generic letsencrypt setup



if [ -d $letsencryptLive ]; then

    diff -q $nginxAvailable $secureNginx
    if [ $? -ne 0 ]; then
        #Only copy new server setup and restart server if file is different.
        cp -T $secureNginx $nginxAvailable
        service nginx restart
        echo "$(date) is the last time the ${websiteName} NGinx settings were changed." >> ~/log.txt
    #Remove letsencrypt from crontab for Little scared about this one.
    #little scared about this one.
    grep -v " $websiteName" /etc/crontab >> /etc/crontab
    cp -T $insecureNginx $nginxAvailable
    ln -s $nginxAvailable $nginxEnabled
    indexFile = "${websitePublicHtml}/index.html"
    if [ ! -f $indexFile ]; then
        #dummy file for letsencrypt to find.
        echo "hello">>$indexFile
    service nginx restart

    #assume that letsencypt successfully exists
    pushd /opt/letsencrypt
    #TODO: refactor email variables: < email redacted >
    ./certbot-auto certonly --agree-tos --no-eff-email --non-interactive --nginx --email < email redacted > -d $websiteName -d $wwwWebsiteName
    echo "@monthly root /opt/letsencrypt/letsencrypt-auto certonly --quiet --standalone --renew-by-default -d $websiteName -d $wwwWebsiteName >> /var/log/letsencrypt/letsencrypt-auto-update.log" | sudo tee --append /etc/crontab    

    cp -T $secureNginxSource $morgemileNginxDestination
    service nginx restart

    echo "$(date) is the last time the $websiteName LetsEncrypt were changed." >> ~/log.txt


I had quite a few websites on this server at one point, and the logic in there got truly heinous. However, I'm proud of this.

I know the security was terrible. I know the programming practices were brittle and fragile. It was a product of ignorance. But it led me to be better.

I could make website changes, add scripts, add install logic, and the polling logic would download changes and make them happen. Though I certainly spent a lot of time SSHing into the server and debugging...


I've learned a lot since I wrote the above travesty, deployed it, operated it, and then shut it down. It sure is fun to read past code and to remember where I came from.