Useful arc commands

Arc/Arcanist/Phabricator

To commit a change for review:

arc diff

To push a change to remote branch:

arc land
Advertisements

Configure SmartGit with GitHub

Clone GitHub repository to use Username and Password:
If you’d like to setup SmartGit to use your GitHub username and password to connect and make changes, use the following URL structure to clone the GIT branch locally:

https://username@github.com/username/repo.git

Adding a gitignore file to ignore unwanted files to be committed to the repository:
1. Make sure you commit your existing changes, or you will lose them

git rm -r --cached .
git add .
git commit -m "fixed untracked files"

Adding separate commit authors for separate repositories:
If you have multiple repositories like work and personal and you’d like to separate the username and email to each of these repositories), use the following commands to setup the author configuration to the git commits for specific repositories. Navigate to the git repository root and execute these commands.

git config user.name "User Name"
git config user.email "username@domain.com"

Django – Create temporary CSV file and Email as Attachment

An interesting challenge that I faced today. I wanted to create a CSV file on my Django project (fairly easy, huh) and email it as attachment. My first challenge was to save the file in a temp directory that is OS independent since I do my development on Windows and the application runs on a linux server. I came across this really cool python module called tempfile, but the drawback to me for using this module was that:

  1. It would create these random filenames
  2. I didn’t want to have the recipient see a random filename and think of it as spam
  3. The files would stay in the temp directory until manually cleaned up. Since the files need to be attached after creations, I didn’t want to delete them after creation which is the default behavior for tempfile (s)

So, the solution I found was to use tempfile module to get the location of the temp directory. Then use that location and write a file using standard I/O with a specific name. Every time the job runs, it would overwrite the old file and I won’t have a ton of files sitting in the temp directory. Here’s the complete solution to the problem:

from datetime import datetime, timedelta
import csv
import tempfile, os

file_path = os.path.join(tempfile.gettempdir(), 'expired_deployments.csv')

email_from = "YOUR-EMAIL@DOMAIN.COM"
email_to = ["TO-EMAIL@DOMAIN.COM"]
email_subject = "YOUR EMAIL SUBJECT"
email_body = "YOUR EMAIL BODY..."

# Create CSV file
f = open(file_path, 'wb')
file_writer = csv.writer(f, quotechar='"', quoting=csv.QUOTE_MINIMAL)
# Write headers to CSV file
file_header = ['Product Type', 'Company Name', 'SFID', 'DeploymentID', 'Status', 'License Expiration', 'Support Expiration']
file_writer.writerow(file_header)
file_writer.writerow(['XYZ', 'ABC In.c', 12345, 123, 'Active', '5/31/2017', '12/31/2017'])
f.close()

# Read the file and send as attachment
fr = open(f.name, "rb")
mail = CustomEmailMessage(email_subject, email_body, email_from, to=email_to, bcc=[]) 
mail.attach(os.path.basename(f.name), fr.read(), "application/octet-stream")
mail.send()
fr.close()

Note: The piece of code that sends out the email only works on Django. If you’re looking to send a file using standard python libraries, check out this other article about using smtplib to send out email attachments.

Here’s the custom class CustomEmailMessage that overrides that Django class EmailMessage. This is useful to stop email messages going out to actual users when you’re working on a development system. Must always use CustomEmailMessage to send out email notification and it would take care of things.

from django.core.mail.message import EmailMessage

class CustomEmailMessage(EmailMessage):
    # Over writting the send function so that in development and eval mode, emails are not sent to all.
    def send(self, fail_silently=False):
        """Sends the email message."""
        if settings.ENVIRONMENT == 'development' or settings.ENVIRONMENT == 'eval': # Staging's settings is same as production.
            self.to = [self.from_email,]
        if not self.recipients():
            # Don't bother creating the network connection if there's nobody to
            # send to.
            return 0
        return self.get_connection(fail_silently).send_messages([self])

Python – Export to CSV

Exporting data to a CSV file:

import csv

# Create CSV file
f = open('file_name.csv', 'w')
file_writer = csv.writer(f, quotechar='"', quoting=csv.QUOTE_MINIMAL)

# Write headers to CSV file
volume_header = ['col1', 'col2', 'col3', 'col4', 'col5', 'col6', 'col7']
file_writer.writerow(volume_header)

# Simplifying the code assuming you have an array of data
for i in array:
	file_writer.writerow([i.val1, i.val2, i.val3, i.val4, i.val5, i.val6, i.val7])
f.close()

Python smtplib – Send email with attachments

Here’s how you can send attachments from email using smtplib

#!/usr/lib/python
import smtplib
import mimetypes
from email.mime.multipart import MIMEMultipart
from email import encoders
from email.message import Message
from email.mime.audio import MIMEAudio
from email.mime.base import MIMEBase
from email.mime.image import MIMEImage
from email.mime.text import MIMEText

emailfrom = "FROM-EMAIL@DOMAIN.COM"
emailto = "TO-EMAIL@DOMAIN.COM"
fileToSend = "UPLOAD-FILE.csv"

msg = MIMEMultipart()
msg["From"] = emailfrom
msg["To"] = emailto
msg["Subject"] = "YOUR EMAIL SUBJECT"
msg.preamble = "YOUR EMAIL SUBJECT"

ctype, encoding = mimetypes.guess_type(fileToSend)
if ctype is None or encoding is not None:
    ctype = "application/octet-stream"

maintype, subtype = ctype.split("/", 1)

if maintype == "text":
    print 'text'
    fp = open(fileToSend)
    # Note: we should handle calculating the charset
    attachment = MIMEText(fp.read(), _subtype=subtype)
    fp.close()
elif maintype == "image":
    print 'image'
    fp = open(fileToSend, "rb")
    attachment = MIMEImage(fp.read(), _subtype=subtype)
    fp.close()
elif maintype == "audio":
    print 'audio'
    fp = open(fileToSend, "rb")
    attachment = MIMEAudio(fp.read(), _subtype=subtype)
    fp.close()
else:
    print 'else'
    fp = open(fileToSend, "rb")
    attachment = MIMEBase(maintype, subtype)
    attachment.set_payload(fp.read())
    fp.close()
    encoders.encode_base64(attachment)
attachment.add_header("Content-Disposition", "attachment", filename=fileToSend)
msg.attach(attachment)

server = smtplib.SMTP("EMAIL-SERVER.DOMAIN.COM")
server.set_debuglevel(0)
server.sendmail(emailfrom, emailto, msg.as_string())
server.quit()

GIT: Switching from code branch to dev/master

We recently upgraded to a latest version of Django (1.8). Earlier, we were running Django 1.2 and we created a new GIT branch called django18 that had all the migration changes. After going Live, we planned to stay on the django18 branch for a while so that if anything goes catastrophic, we can revert back to dev/master. But it had been a few months now and we didn’t revert back to dev and master. After I merged the django18 branch with dev, and then merged dev to master – here’s what I did to switch our dev and production server to run dev and master branches respectively. It wasn’t as easy as doing a checkout dev/master and just pulling code. With a lot of code and structural changes, I had to run through a lot of hoops to get this done right:

# Make sure the Django1.8 branch is up to date
git checkout task/django18
git status

# update dev branch
git checkout dev
git pull varun.verma dev

# update master branch
git checkout master
git pull varun.verma master

'''
error: The following untracked working tree files would be overwritten by merge:
	portal/__init__.pyc
	portal/api/__init__.pyc
	portal/api/admin.pyc
	portal/api/management/__init__.pyc
	portal/api/migrations/0001_initial.pyc
	portal/api/migrations/__init__.pyc
	portal/api/models.pyc
	portal/api/urls.pyc
	...
Aborting
'''

# View the the untracked files that will be cleaned. Review that all are .pyc files
git clean -fx --dry-run

# Now remove the untracked files
git clean -fx

# pull master branch again and it will complain about the statuc files as shown below
git pull varun.verma master

'''
From gitserver.domain.com:/var/git/your_repo
 * branch            master     -> FETCH_HEAD
error: The following untracked working tree files would be overwritten by merge:
	portal/static/admin/css/base.css
	portal/static/admin/css/changelists.css
	portal/static/admin/css/dashboard.css
	portal/static/admin/css/forms.css
	portal/static/admin/css/ie.css
	...
Please move or remove them before you can merge.
Aborting
'''

# View the untracked directories
git clean -dfx --dry-run
'''
Would remove portal/bi/management/
Would remove portal/another_app/management/
Would remove portal/static/admin/
'''

# Now get rid of the portal/static directory and build it again later
git clean -dfx

# pull master branch again and it will pull this time and list the additions/deletions/modifications
git pull varun.verma master

# make sure master local and origin are in sync
git status

'''
On branch master
Your branch is up-to-date with 'origin/master'.
'''

# now pull the static files
python manage.py collectstatic --settings= portal.settings_stage

# modify wsgi.py file to use the stage settings file
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "portal.settings_stage")

# restart apache
apache2ctl restart

# You may notice a list of .pyc files that now show as modified. You will have to checkout each of them individually
git checkout portal/__init__.pyc
...