Category Archives: Uncategorized

Programmatically get the ec2 spot price for an instance size

This python script will output the spot price for an instance size passed in via a command line arg. This just looks at the east region but can easily be modified for other aws regions.

import sys, urllib, json
try:
    # Get the jsonp spot price data from amazon
    response = urllib.urlopen("http://spot-price.s3.amazonaws.com/spot.js");
    jsonpstr = response.read();
    data = json.loads(jsonpstr[ jsonpstr.index("(") + 1 : jsonpstr.rindex(")") ])
    # Parse it and retrieve the USD price for the instance size passed in arg[1] i.e. m3.xlarge
    reg=filter(lambda x: x["region"]=="us-east", data["config"]["regions"])
    instance = [ y for x in reg[0]["instanceTypes"] for y in x["sizes"] if y["size"]==sys.argv[1] ]
    print filter(lambda x: x["name"]=="linux", instance[0]["valueColumns"])[0]["prices"]["USD"]
except:
    print 0

My python knowledge is very minimal so any suggested improvements would be appreciated.

Moving / Copying lots of s3 files quickly using gnu parallel

3 Replies

I recently had to copy a large volume of files from one s3 folder to another using s3cmd but found that the process was very slow as it is single threaded. I looked into how to do this in a more parallel manner and discovered GNU Parallel which was the answer to my dreams.

To use it I simply create a list of files that need to be moved and pass them to parallel with the correct copy command :

The -j20 switch tells it to use 20 parallel threads
–halt 1 is so that if one task fails then the remainder are finished and the command fails
{/} is a GNU Parallel specific token which denotes the basename of the line from the file passed to parallel i.e. given s3://bucket/filename.txt {/} returns filename.txt. Note that this requires a recent version of GNU parallel so install the latest stable version from source if necessary.

# filelist.txt just contains a list of s3 files i.e. s3://bucket/filename.ext
# Parallel creates a new thread to handle each line in the file (up to the limit -j)
# Note also that we escape the $ within the command passed to parallel. If we did not 
# escape it then the variable would be treated as a variable in the scope of the 
# calling script rather than within the parallel call.
cat filelist.txt |parallel -j20 --halt 1 "filename={/};s3cmd cp s3://bucket/folder1/$filename s3://bucket/folder2/$filename;"

If you want to perform a more complex task in parallel it can be a bit cumbersome and unreadable to put all the command line. To get around this we can just use a bash function.

#!/bin/bash
function dosomethingabitmorecomplex {
    echo "Doing something with arg $1"
    sleep 10
    echo "finished doing something with arg $1"
}

# Since parallel creates subshells for each of its threads we need to 
# export the function to ensure it can be accessed by the subshell
export -f dosomethingabitmorecomplex 
# testfile.txt just contains lines of text
# the {} token represents the line passed to parallel from the text file
parallel -j20 "dosomethingabitmorecomplex {}" < testfile.txt

A simple script to list s3 bucket sizes

Mounting a Windows 7 Share in Linux

A simple “treesize” shell script for Linux

26 Replies

One of my favorite pieces of software on windows is a little app called treesize free by Jam Software. It basically gives you a simple list of how much disk space each directory is taking up. This is really useful when you are trying to work out where all the space on your 500gig disk is gone.

I am always finding myself looking for similar piece of software for linux which can be run simply from the command line, but alas none exists so i decided to create a simple shell script to do a similar job, and here it is :

#/bin/sh
du -k --max-depth=1 | sort -nr | awk '
     BEGIN {
        split("KB,MB,GB,TB", Units, ",");
     }
     {
        u = 1;
        while ($1 >= 1024) {
           $1 = $1 / 1024;
           u += 1
        }
        $1 = sprintf("%.1f %s", $1, Units[u]);
        print $0;
     }
    '

Just put this code into a file /bin/treesize and make it executable. Then any system user can get a list of directory sizes within a directory by just running treesize from any directory.

Pro Tip : PHP has built in functions for reading and writing csv files

2 Replies

Just discovered this and I am mighty impressed. It seems that in PHP 5.1 and higher there are the functions fgetcsv() and fputcsv(). They make it pretty simple to read and write a csv file.

Here is a simple example which reads a csv file, adds a couple of fields and writes it to a new file.

Introducing 33mail.com – Unlimited Disposable Email Addresses

1 Reply

I’ve just launched my latest project 33mail.com. Its a solution to the ever increasing problem of spam and the fact that you really can’t trust anyone with your email address anymore. The system is pretty simple, you sign up to 33mail with a username, say testing123, and you give 33mail your real email address, you can then create alias addresses using 33mail, such as blog.aclarke.eu@testing123dhy66s.kjjss33djjhsda2mail.com which will then be forwarded onto your real address. The kicker is that at any time you can kill the alias addresses you create, and because you can create unlimited addresses you can give a different one every time you need one.

The site is built with codeigniter and the email server backend is pure java.

I’d love to hear any feedback on the site design, functionality and the service as a whole.

Laser card custom payment module for Magento

11 Replies

While helping my brother out with bringing his sports shop online, I discovered Magento which is a pretty kick ass piece of php. One thing that Magento doesn’t have at the moment is a payment module for Irish Laser Debit cards, so I decided to write my own.

Some resources that I found useful were this and this, The first being from the Magento wiki and the second from a forum post.

The process is pretty straight forward once you get your head around all the config files, there are basically 10 files needed for the module (Coolbreeze is just the package name I gave the modules akin to the Mage group in magento) :

app/code/local/Coolbreese/LaserPayment/Block/Form.php
the controller for displaying the input form in the store and admin

app/code/local/Coolbreese/LaserPayment/Block/Info.php
the controller for displaying the card details in the store and admin

app/code/local/Coolbreese/LaserPayment/etc/config.xml
config stuff

app/code/local/Coolbreese/LaserPayment/etc/system.xml
more config stuff

app/code/local/Coolbreese/LaserPayment/Model/Laser.php
where the validation etc happens

app/design/adminhtml/default/default/template/laserpayment/form.phtml
the form template for the admin interface

app/design/adminhtml/default/default/template/laserpayment/info.phtml
the details display template for the admin interface

app/design/frontend/default/default/template/laserpayment/form.phtml
form display in the store

app/design/frontend/default/default/template/laserpayment/info.phtml
details display in the store

app/etc/modules/Coolbreeze_LaserPayment.xml
the main config file which tells magento that this module exist.

Here for your viewing pleasure is the module code to download.

To install it in a Magento installation just copy the app folder into the magento root and all the files will be put in the right place. Then go to admin -> System -> Configuration -> Payment Methods and enable the module.

Some cavaets :

I take no responsibility for breaking anything whatsoever so install this at your own risk or better yet look at the code and improve it.

The Laser number validation is pretty simple at the moment with no modulus checks. Something like this would probably be useful also as there is no client side validation just server side which works ok but is not ideal.

I’d love to hear feedback from anyone who uses this or finds it useful.

Windows 7 first impressions

Crop and resize an image using MiniMagick