Enabling and Forcing HTTPS on nginx server

Getting used to nginx. Just configured nginx to enable HTTPS and also force HTTPS by redirecting HTTP to HTTPS.

1. Get the SSL certs on the server – Navigate to /etc/ssl. Create your own directory with the domain name if you wish to or you could please your SSL Certs with the existing certs and private sub-directories under there. I preferred to create a new directory so it’s easy to manage the certs created by us. Place the following 2 files in the directory, e.g.

/etc/ssl/domain.com/20180521-domain_com.key
/etc/ssl/domain.com/20180521-domain_com.crt

2. Modify the nginx configuration – Navigate to /etc/nginx/sites-available. Open the file redash that is created during the server setup and add the following lines of code so the final file looks something like this (The pieces of code in Bold are the ones that were added).

upstream rd_servers {
  server 127.0.0.1:5000;
}

server {

  server_tokens off;

  listen 80;
  server_name hostname.domain.com;
  return 301 https://$server_name$request_uri;

  access_log /var/log/nginx/rd.access.log;

  gzip on;
  gzip_types *;
  gzip_proxied any;

  location / {
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_pass       http://rd_servers;
  }
}
server {

  server_tokens off;

  listen 443;
  server_name hostname.domain.com;

  ssl on;
  ssl_certificate /etc/ssl/domain.com/20180521-domain_com.crt;
  ssl_certificate_key /etc/ssl/domain.com/20180521-domain_com.key;
  ssl_protocols TLSv1.2;

  access_log /var/log/nginx/rd.access.log;

  gzip on;
  gzip_types *;
  gzip_proxied any;

  location / {
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_pass       http://rd_servers;
  }
}

The following lines of code will redirect the HTTP traffic to HTTPS. Only required if you completely want to disable HTTP.

return 301 https://$server_name$request_uri;
Advertisements

Install Redash server On-Prem on Debian

I followed setup instructions to install the redash server on-prem. There’s a provisioning script to install the redash server on-prem. But as expected, things don’t really work first time on all environments. I used the latest Debian (stretch) v9.2.1 and ran into issues running the provisioning script. Then I had to manually debug and go through each step/package installation at a time and found the following 2 issues.

1. pip installation cryptography didn’t work. I eliminated this step. Remove the following line from the file /opt/redash/current/requirements.txt

#cryptography==1.4

2. The pyOpenSSL package wasn’t compatible so I had to replace the version in requirements_all_ds.txt file under /opt/redash/current/ directory

#pyOpenSSL==0.14
pyOpenSSL==16.2.0


Original Script for reference

#!/bin/bash
#
# This script setups Redash along with supervisor, nginx, PostgreSQL and Redis. It was written to be used on 
# Ubuntu 16.04. Technically it can work with other Ubuntu versions, but you might get non compatible versions
# of PostgreSQL, Redis and maybe some other dependencies.
#
# This script is not idempotent and if it stops in the middle, you can't just run it again. You should either 
# understand what parts of it to exclude or just start over on a new VM (assuming you're using a VM).

set -eu

REDASH_BASE_PATH=/opt/redash
REDASH_BRANCH="${REDASH_BRANCH:-master}" # Default branch/version to master if not specified in REDASH_BRANCH env var
REDASH_VERSION=${REDASH_VERSION-2.0.1.b3080} # Install latest version if not specified in REDASH_VERSION env var
LATEST_URL="https://s3.amazonaws.com/redash-releases/redash.${REDASH_VERSION}.tar.gz"
VERSION_DIR="$REDASH_BASE_PATH/redash.${REDASH_VERSION}"
REDASH_TARBALL=/tmp/redash.tar.gz
FILES_BASE_URL=https://raw.githubusercontent.com/getredash/redash/${REDASH_BRANCH}/setup/ubuntu/files

cd /tmp/

verify_root() {
    # Verify running as root:
    if [ "$(id -u)" != "0" ]; then
        if [ $# -ne 0 ]; then
            echo "Failed running with sudo. Exiting." 1>&2
            exit 1
        fi
        echo "This script must be run as root. Trying to run with sudo."
        sudo bash "$0" --with-sudo
        exit 0
    fi
}

create_redash_user() {
    adduser --system --no-create-home --disabled-login --gecos "" redash
}

install_system_packages() {
    apt-get -y update
    # Base packages
    apt install -y python-pip python-dev nginx curl build-essential pwgen
    # Data sources dependencies:
    apt install -y libffi-dev libssl-dev libmysqlclient-dev libpq-dev freetds-dev libsasl2-dev
    # SAML dependency
    apt install -y xmlsec1
    # Storage servers
    apt install -y postgresql redis-server
    apt install -y supervisor
}

create_directories() {
    mkdir -p $REDASH_BASE_PATH
    chown redash $REDASH_BASE_PATH
    
    # Default config file
    if [ ! -f "$REDASH_BASE_PATH/.env" ]; then
        sudo -u redash wget "$FILES_BASE_URL/env" -O $REDASH_BASE_PATH/.env
    fi

    COOKIE_SECRET=$(pwgen -1s 32)
    echo "export REDASH_COOKIE_SECRET=$COOKIE_SECRET" >> $REDASH_BASE_PATH/.env
}

extract_redash_sources() {
    sudo -u redash wget "$LATEST_URL" -O "$REDASH_TARBALL"
    sudo -u redash mkdir "$VERSION_DIR"
    sudo -u redash tar -C "$VERSION_DIR" -xvf "$REDASH_TARBALL"
    ln -nfs "$VERSION_DIR" $REDASH_BASE_PATH/current
    ln -nfs $REDASH_BASE_PATH/.env $REDASH_BASE_PATH/current/.env
}

install_python_packages() {
    pip install --upgrade pip
    # TODO: venv?
    pip install setproctitle # setproctitle is used by Celery for "pretty" process titles
    pip install -r $REDASH_BASE_PATH/current/requirements.txt
    pip install -r $REDASH_BASE_PATH/current/requirements_all_ds.txt
}

create_database() {
    # Create user and database
    sudo -u postgres createuser redash --no-superuser --no-createdb --no-createrole
    sudo -u postgres createdb redash --owner=redash

    cd $REDASH_BASE_PATH/current
    sudo -u redash bin/run ./manage.py database create_tables
}

setup_supervisor() {
    wget -O /etc/supervisor/conf.d/redash.conf "$FILES_BASE_URL/supervisord.conf"
    service supervisor restart
}

setup_nginx() {
    rm /etc/nginx/sites-enabled/default
    wget -O /etc/nginx/sites-available/redash "$FILES_BASE_URL/nginx_redash_site"
    ln -nfs /etc/nginx/sites-available/redash /etc/nginx/sites-enabled/redash
    service nginx restart
}

verify_root
install_system_packages
create_redash_user
create_directories
extract_redash_sources
install_python_packages
create_database
setup_supervisor
setup_nginx

3. Setup mail server
When you open redash, you’ll get a message notifying that the mail server has not been setup. The setup page doesn’t have much details about how to setup mail configuration and troubleshoot. After some trial and error, I was able to configure the mail server properly.

Test sending email from console. This would throw an error if the configuration/setup is incomplete and notify what’s missing.

cd /opt/redash/current
bin/run ./manage.py send_test_mail

The Setup Page has details about some environment variables that need to be setup for Mail Configuration. But just exporting them on the console didn’t work. There are a couple of places that you can enter these settings – a .env file that stores the environment variables (PREFERRED) and a settings file that reads from the env file. Prefer storing the configuration in the env file so settings stay pristine and is not hard-coded. Here is a detailed list of Environment Variables with descriptions that redash supports and can be configured.

# Setting redash environment variables (PREFERRED)
/opt/redash/current/.env

# Settings file that reads from the Environment Variables.
/opt/redash/current/redash/settings.py

After setting the Environment Variables, do have to restart all processes (not just nginx)

# Restart all processes
sudo supervisorctl restart all

# Restart the Web server
sudo supervisorctl restart redash_server

# Restart Celery workers
sudo supervisorctl restart redash_celery

This maintenance page has information about ongoing maintenance and troubleshooting steps for the redash server.

Convert Array to CSV string for SQL operations

The PHP function implode() can be used to convert an Array to a comma separated string. But if you’d like the string to be a part of an SQL WHERE IN statement, they would need to be enclosed within single quotes. Here’s what you can use to accomplish a comma separated string enclosed within single quotes.

$statusToProcess = array('Pending','Approved');
print implode(",", $statusToProcess); \\ Outputs: Pending,Approved
print "\n";
print "'".implode("','", $statusToProcess)."'"; \\ Outputs: 'Pending','Approved'
print "\n";

jQuery – get nth parent div for the current div

I had this interesting problem of hiding the entire div row from a page. But the div row itself didn’t have an ID. I had an ID of a element down in the hierarchy from that div. e.g. the structure looks something like this:

<div>
    <div>
        <div>
            <div id="StateCode">State Dropdown</div>
        </div>
    </div>
</div>

In order to get to the top-most div from the div with the StateCode ID, this is what I did to get there

$("#StateCode").parents().eq(2);  // "Great-grandfather" div
$('#StateCode').parents().eq(0);  // "Father" div, just stating for the example

And here’s the complete solution to hiding and showing the div based on a condition, in my case it was showing the State/Province only if the country is US/Canada/Australia. The trigger is change of CountryCode field (which is the country dropdown).

$("#CountryCode").change(function(){
    if(['US','CA','AU'].indexOf($("#CountryCode").val()) > -1) {
        $("#StateCode").parents().eq(2).show();
        //$("#StateCode").prop('disabled', false);
    }
    else {
        $("#StateCode").parents().eq(2).hide();
        //$("#StateCode").prop('disabled', true);
    }
});

Configure SmartGit with GitHub

Clone GitHub repository to use Username and Password:
If you’d like to setup SmartGit to use your GitHub username and password to connect and make changes, use the following URL structure to clone the GIT branch locally:

https://username@github.com/username/repo.git

Adding a gitignore file to ignore unwanted files to be committed to the repository:
1. Make sure you commit your existing changes, or you will lose them

git rm -r --cached .
git add .
git commit -m "fixed untracked files"

Adding separate commit authors for separate repositories:
If you have multiple repositories like work and personal and you’d like to separate the username and email to each of these repositories), use the following commands to setup the author configuration to the git commits for specific repositories. Navigate to the git repository root and execute these commands.

git config user.name "User Name"
git config user.email "username@domain.com"