Linux – List only folder/directory

People don’t commonly use this but its pretty helpful when you only want to look at just the directories/folders in the current directory.

ls -ld */
Advertisements

Install Redash server On-Prem on Debian

I followed setup instructions to install the redash server on-prem. There’s a provisioning script to install the redash server on-prem. But as expected, things don’t really work first time on all environments. I used the latest Debian (stretch) v9.2.1 and ran into issues running the provisioning script. Then I had to manually debug and go through each step/package installation at a time and found the following 2 issues.

1. pip installation cryptography didn’t work. I eliminated this step. Remove the following line from the file /opt/redash/current/requirements.txt

#cryptography==1.4

2. The pyOpenSSL package wasn’t compatible so I had to replace the version in requirements_all_ds.txt file under /opt/redash/current/ directory

#pyOpenSSL==0.14
pyOpenSSL==16.2.0


Original Script for reference

#!/bin/bash
#
# This script setups Redash along with supervisor, nginx, PostgreSQL and Redis. It was written to be used on 
# Ubuntu 16.04. Technically it can work with other Ubuntu versions, but you might get non compatible versions
# of PostgreSQL, Redis and maybe some other dependencies.
#
# This script is not idempotent and if it stops in the middle, you can't just run it again. You should either 
# understand what parts of it to exclude or just start over on a new VM (assuming you're using a VM).

set -eu

REDASH_BASE_PATH=/opt/redash
REDASH_BRANCH="${REDASH_BRANCH:-master}" # Default branch/version to master if not specified in REDASH_BRANCH env var
REDASH_VERSION=${REDASH_VERSION-2.0.1.b3080} # Install latest version if not specified in REDASH_VERSION env var
LATEST_URL="https://s3.amazonaws.com/redash-releases/redash.${REDASH_VERSION}.tar.gz"
VERSION_DIR="$REDASH_BASE_PATH/redash.${REDASH_VERSION}"
REDASH_TARBALL=/tmp/redash.tar.gz
FILES_BASE_URL=https://raw.githubusercontent.com/getredash/redash/${REDASH_BRANCH}/setup/ubuntu/files

cd /tmp/

verify_root() {
    # Verify running as root:
    if [ "$(id -u)" != "0" ]; then
        if [ $# -ne 0 ]; then
            echo "Failed running with sudo. Exiting." 1>&2
            exit 1
        fi
        echo "This script must be run as root. Trying to run with sudo."
        sudo bash "$0" --with-sudo
        exit 0
    fi
}

create_redash_user() {
    adduser --system --no-create-home --disabled-login --gecos "" redash
}

install_system_packages() {
    apt-get -y update
    # Base packages
    apt install -y python-pip python-dev nginx curl build-essential pwgen
    # Data sources dependencies:
    apt install -y libffi-dev libssl-dev libmysqlclient-dev libpq-dev freetds-dev libsasl2-dev
    # SAML dependency
    apt install -y xmlsec1
    # Storage servers
    apt install -y postgresql redis-server
    apt install -y supervisor
}

create_directories() {
    mkdir -p $REDASH_BASE_PATH
    chown redash $REDASH_BASE_PATH
    
    # Default config file
    if [ ! -f "$REDASH_BASE_PATH/.env" ]; then
        sudo -u redash wget "$FILES_BASE_URL/env" -O $REDASH_BASE_PATH/.env
    fi

    COOKIE_SECRET=$(pwgen -1s 32)
    echo "export REDASH_COOKIE_SECRET=$COOKIE_SECRET" >> $REDASH_BASE_PATH/.env
}

extract_redash_sources() {
    sudo -u redash wget "$LATEST_URL" -O "$REDASH_TARBALL"
    sudo -u redash mkdir "$VERSION_DIR"
    sudo -u redash tar -C "$VERSION_DIR" -xvf "$REDASH_TARBALL"
    ln -nfs "$VERSION_DIR" $REDASH_BASE_PATH/current
    ln -nfs $REDASH_BASE_PATH/.env $REDASH_BASE_PATH/current/.env
}

install_python_packages() {
    pip install --upgrade pip
    # TODO: venv?
    pip install setproctitle # setproctitle is used by Celery for "pretty" process titles
    pip install -r $REDASH_BASE_PATH/current/requirements.txt
    pip install -r $REDASH_BASE_PATH/current/requirements_all_ds.txt
}

create_database() {
    # Create user and database
    sudo -u postgres createuser redash --no-superuser --no-createdb --no-createrole
    sudo -u postgres createdb redash --owner=redash

    cd $REDASH_BASE_PATH/current
    sudo -u redash bin/run ./manage.py database create_tables
}

setup_supervisor() {
    wget -O /etc/supervisor/conf.d/redash.conf "$FILES_BASE_URL/supervisord.conf"
    service supervisor restart
}

setup_nginx() {
    rm /etc/nginx/sites-enabled/default
    wget -O /etc/nginx/sites-available/redash "$FILES_BASE_URL/nginx_redash_site"
    ln -nfs /etc/nginx/sites-available/redash /etc/nginx/sites-enabled/redash
    service nginx restart
}

verify_root
install_system_packages
create_redash_user
create_directories
extract_redash_sources
install_python_packages
create_database
setup_supervisor
setup_nginx

3. Setup mail server
When you open redash, you’ll get a message notifying that the mail server has not been setup. The setup page doesn’t have much details about how to setup mail configuration and troubleshoot. After some trial and error, I was able to configure the mail server properly.

Test sending email from console. This would throw an error if the configuration/setup is incomplete and notify what’s missing.

cd /opt/redash/current
bin/run ./manage.py send_test_mail

The Setup Page has details about some environment variables that need to be setup for Mail Configuration. But just exporting them on the console didn’t work. There are a couple of places that you can enter these settings – a .env file that stores the environment variables (PREFERRED) and a settings file that reads from the env file. Prefer storing the configuration in the env file so settings stay pristine and is not hard-coded. Here is a detailed list of Environment Variables with descriptions that redash supports and can be configured.

# Setting redash environment variables (PREFERRED)
/opt/redash/current/.env

# Settings file that reads from the Environment Variables.
/opt/redash/current/redash/settings.py

After setting the Environment Variables, do have to restart all processes (not just nginx)

# Restart all processes
sudo supervisorctl restart all

# Restart the Web server
sudo supervisorctl restart redash_server

# Restart Celery workers
sudo supervisorctl restart redash_celery

This maintenance page has information about ongoing maintenance and troubleshooting steps for the redash server.

Installing / Re-Installing Apache, PHP, libapache2-mod-php

The following is done on an Ubuntu machine but hoping it should cover other Debian instances.

I accidentally deleted php7.0.conf and php7.0.load from the mods-available folder. These files come with libapache2-mod-php. I uninstalled the module and installed it again – but the files didn’t appear. A lot of blogs suggested to remove Apache, PHP and libapache2-mod-php altogether and installing it again and it didn’t work. I used remove, purge and everything suggested but didn’t help. Eventually, I had to search for the installed module by:

dpkg --get-selections | grep libapache2-mod-php

I installed the module above as is but the result showed me libapache2-mod-php7.0. So I went ahead and removed and purged the module listed after which I installed the module again (without the 7.0 suffix) and I was able to recover the files again.

apt-get remove --purge libapache2-mod-php7.0
apt-get install libapache2-mod-php

Since I am covering this topic, let’s talk about installing Apache on Debian and the supported PHP packages along with SSL support.

apt-get install apache2
apt-get install php
apt-get install php-curl
apt-get install libapache2-mod-php

Enable SSL on the site:

cd /etc/apache2/sites-enabled
ln -fs ../sites-available/default-ssl.conf 

Finally, add the necessary modules to support PHP/SSL on Apache

cd /etc/apache2/mods-enabled
ln -fs ../mods-available/ssl.conf
ln -fs ../mods-available/ssl.load 
ln -fs ../mods-available/socache_shmcb.load
ln -fs ../mods-available/php7.0.conf
ln -fs ../mods-available/php7.0.load

Do not forget to restart apache with

service apache2 restart

Linux – Adding useful aliases

I prefer to add the aliases to a different file rather than the .bashrc. You can modify the .bashrc file to include the .bash_aliases file:

# Alias definitions.
# You may want to put all your additions into a separate file like
# ~/.bash_aliases, instead of adding them here directly.
# See /usr/share/doc/bash-doc/examples in the bash-doc package.

if [ -f ~/.bash_aliases ]; then
    . ~/.bash_aliases
fi

And then create a new file .bash_aliases in the home directory with the following or your own aliases:

#!/bin/bash
# Adding custom aliases
alias l='ls -lrt'
alias lwc='ls -lrt | wc -l'
alias la='ls -lart'
alias lm='ls -lrt | more'
alias lwc='ls | wc -l'
alias f='find . -name $1'
alias b='cd ..'

Finally, to execute the bash_aliases in your current session without having to logout and login again, execute the following command:

. ~/.bash_aliases

Rename files in a sequential order

My camera shoots images until DSC_9999 and after that it resets to DSC_0001. It messes up the sequence of images shot during a day if the sequence is reset. I found this cool solution which resets them in sequential order starting from 1 and also pads 0’s to the beginning if the number is less than 100.

a=1
for i in *.JPG; do
  new=$(printf "DSC_%04d.JPG" "$a") #04 pad to length of 4
  mv -i -- "$i" "$new"
  let a=a+1
done