Author Archives: andrew

Programmatically get the ec2 spot price for an instance size

1 Reply

This python script will output the spot price for an instance size passed in via a command line arg. This just looks at the east region but can easily be modified for other aws regions.

import sys, urllib, json
try:
    # Get the jsonp spot price data from amazon
    response = urllib.urlopen("http://spot-price.s3.amazonaws.com/spot.js");
    jsonpstr = response.read();
    data = json.loads(jsonpstr[ jsonpstr.index("(") + 1 : jsonpstr.rindex(")") ])
    # Parse it and retrieve the USD price for the instance size passed in arg[1] i.e. m3.xlarge
    reg=filter(lambda x: x["region"]=="us-east", data["config"]["regions"])
    instance = [ y for x in reg[0]["instanceTypes"] for y in x["sizes"] if y["size"]==sys.argv[1] ]
    print filter(lambda x: x["name"]=="linux", instance[0]["valueColumns"])[0]["prices"]["USD"]
except:
    print 0

My python knowledge is very minimal so any suggested improvements would be appreciated.

Moving / Copying lots of s3 files quickly using gnu parallel

3 Replies

I recently had to copy a large volume of files from one s3 folder to another using s3cmd but found that the process was very slow as it is single threaded. I looked into how to do this in a more parallel manner and discovered GNU Parallel which was the answer to my dreams.

To use it I simply create a list of files that need to be moved and pass them to parallel with the correct copy command :

  • The -j20 switch tells it to use 20 parallel threads
  • –halt 1 is so that if one task fails then the remainder are finished and the command fails
  • {/} is a GNU Parallel specific token which denotes the basename of the line from the file passed to parallel i.e. given s3://bucket/filename.txt {/} returns filename.txt. Note that this requires a recent version of GNU parallel so install the latest stable version from source if necessary.
# filelist.txt just contains a list of s3 files i.e. s3://bucket/filename.ext
# Parallel creates a new thread to handle each line in the file (up to the limit -j)
# Note also that we escape the $ within the command passed to parallel. If we did not 
# escape it then the variable would be treated as a variable in the scope of the 
# calling script rather than within the parallel call.
cat filelist.txt |parallel -j20 --halt 1 "filename={/};s3cmd cp s3://bucket/folder1/$filename s3://bucket/folder2/$filename;"

If you want to perform a more complex task in parallel it can be a bit cumbersome and unreadable to put all the command line. To get around this we can just use a bash function.

#!/bin/bash
function dosomethingabitmorecomplex {
    echo "Doing something with arg $1"
    sleep 10
    echo "finished doing something with arg $1"
}

# Since parallel creates subshells for each of its threads we need to 
# export the function to ensure it can be accessed by the subshell
export -f dosomethingabitmorecomplex 
# testfile.txt just contains lines of text
# the {} token represents the line passed to parallel from the text file
parallel -j20 "dosomethingabitmorecomplex {}" < testfile.txt

A simple script to list s3 bucket sizes

Leave a reply

This script uses the excellent s3cmd to provide a list of all buckets in an account along with their sizes rounded down to the nearest GB MB and Byte. Amazingly there doesnt seem to be an easy way to do this within the s3 web interface.

#!/bin/sh
buckets=`s3cmd -c .yours3cfgfile ls | awk '{FS=" ";print $3}'`
for bucket in $buckets
do
size=`s3cmd -c .yours3cfgfile du "$bucket" |awk '{FS=" ";print $1}'`
sizemb=`expr $size / \( 1024 \* 1024 \)`
sizegb=`expr $sizemb / 1024`
echo "$bucket ${sizegb} GB ${sizemb} MB ${size} bytes"
done

Hope someone finds this useful.

Mounting a Windows 7 Share in Linux

Leave a reply

I run an Ubuntu VM using VMware on my windows 7 machine and I find it very useful to be able to access files on the windows machine from inside of linux. This is actually quite easy to achieve and you can be up and running in about five minutes if nothing goes wrong.

First, in windows, share the folder that you are trying to access from your linux box. Make sure that the user you want to grant access to has read and write permissions (by default this the the current admin user so most likely you wont have to worry about this)

Now create an .smbcredentials file somewhere on your linux machine. I just used :

/home/andrew/.smbcredentials

but you can put it wherever floats your boat.

Now edit the .smbcredentials file and add the username and password for the windows machine in this file in this format :

username=andrew
password=andrewspassword

Next you need to create the entry in your /etc/fstab which will mount the directory on startup.
Add this line to your /etc/fstab file (filling in your details where necessary) :

///dev /mnt/ smbfs iocharset=utf8,credentials=/path/to/.smbcredentials,uid=1000 0 0

Next you need to create the mount point and make sure you can access it as a non root user :

sudo mkdir /mnt/
chmod -R 775 /mnt/
chown -R :root /mnt/

Finally ensure that smbfs is installed on your system. In Ubuntu just use :

sudo apt-get install smbfs

Now just mount it

sudo mount -a

If you get an error like this :

mount: wrong fs type, bad option, bad superblock on ⁄⁄mywindows⁄myshare,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might....

You probably havent installed smbfs correctly so run :

sudo apt-get install smbfs

again.

A simple “treesize” shell script for Linux

26 Replies

One of my favorite pieces of software on windows is a little app called treesize free by Jam Software. It basically gives you a simple list of how much disk space each directory is taking up. This is really useful when you are trying to work out where all the space on your 500gig disk is gone.

I am always finding myself looking for similar piece of software for linux which can be run simply from the command line, but alas none exists so i decided to create a simple shell script to do a similar job, and here it is : 

#/bin/sh
du -k --max-depth=1 | sort -nr | awk '
     BEGIN {
        split("KB,MB,GB,TB", Units, ",");
     }
     {
        u = 1;
        while ($1 >= 1024) {
           $1 = $1 / 1024;
           u += 1
        }
        $1 = sprintf("%.1f %s", $1, Units[u]);
        print $0;
     }
    '

Just put this code into a file /bin/treesize and make it executable. Then any system user can get a list of directory sizes within a directory by just running treesize from any directory.

Introducing 33mail.com – Unlimited Disposable Email Addresses

1 Reply

I’ve just launched my latest project 33mail.com. Its a solution to the ever increasing problem of spam and the fact that you really can’t trust anyone with your email address anymore. The system is pretty simple, you sign up to 33mail with a username, say testing123, and you give 33mail your real email address, you can then create alias addresses using 33mail, such as blog.aclarke.eu@testing123dhy66s.kjjss33djjhsda2mail.com which will then be forwarded onto your real address. The kicker is that at any time you can kill the alias addresses you create, and because you can create unlimited addresses you can give a different one every time you need one.

The site is built with codeigniter and the email server backend is pure java.

I’d love to hear any feedback on the site design, functionality and the service as a whole.

Laser card custom payment module for Magento

11 Replies

While helping my brother out with bringing his sports shop online, I discovered Magento which is a pretty kick ass piece of php. One thing that Magento doesn’t have at the moment is a payment module for Irish Laser Debit cards, so I decided to write my own.

Some resources that I found useful were this and this, The first being from the Magento wiki and the second from a forum post.

The process is pretty straight forward once you get your head around all the config files, there are basically 10 files needed for the module (Coolbreeze is just the package name I gave the modules akin to the Mage group in magento) :

app/code/local/Coolbreese/LaserPayment/Block/Form.php
the controller for displaying the input form in the store and admin

app/code/local/Coolbreese/LaserPayment/Block/Info.php
the controller for displaying the card details in the store and admin

app/code/local/Coolbreese/LaserPayment/etc/config.xml
config stuff

app/code/local/Coolbreese/LaserPayment/etc/system.xml
more config stuff

app/code/local/Coolbreese/LaserPayment/Model/Laser.php
where the validation etc happens

app/design/adminhtml/default/default/template/laserpayment/form.phtml
the form template for the admin interface

app/design/adminhtml/default/default/template/laserpayment/info.phtml
the details display template for the admin interface

app/design/frontend/default/default/template/laserpayment/form.phtml
form display in the store

app/design/frontend/default/default/template/laserpayment/info.phtml
details display in the store

app/etc/modules/Coolbreeze_LaserPayment.xml
the main config file which tells magento that this module exist.

Here for your viewing pleasure is the module code to download.

To install it in a Magento installation just copy the app folder into the magento root and all the files will be put in the right place. Then go to admin -> System -> Configuration -> Payment Methods and enable the module.

Some cavaets :

I take no responsibility for breaking anything whatsoever so install this at your own risk or better yet look at the code and improve it.

The Laser number validation is pretty simple at the moment with no modulus checks. Something like this would probably be useful also as there is no client side validation just server side which works ok but is not ideal.

I’d love to hear feedback from anyone who uses this or finds it useful.

Windows 7 first impressions

Leave a reply

The first surprising thing about the Windows 7 beta is that it works great running in VMware, so if you have a relativly new vesion of VMware installed then it is well worth installing the beta to have a look.

The UI is distinctly vista like with the the round start buton and start menu, and similar dialogs, but for some reason it doesnt give off that hateful vibe that Vista does (having installed vista and immediatly removed it reverting back to trusty XP, Im no fan).

The beta comes with some nifty little apps like sticky notes and a great snipping tool for taking screen shots along with the usual games, calculator and of course paint (which is a nicely updated version in the vein of office 2007). It also comes with an IE 8 beta which may be of interest to some.

Overall i like the feel of Windows 7 and after seeing this beta I have high hopes of eventually upgrading my Windows XP machine to it.

So Microsoft, please dont f*$k it up before release.

Crop and resize an image using MiniMagick

3 Replies

During my dealings with twitter avatars (see here) I discovered how to resize and crop an image using MiniMagick. This is useful when dealing with twitter avatars as the large size ones that the twitter api provides are not always square and the smaller ones may be too small for some applications. Below is the simple function : 

def resize_and_crop(image, size)         
  if image.width  image.height 
    remove = ((image.width - image.height)/2).round
    image.shave("#{remove}x0")
  end
  image.resize("#{size}x#{size}")
  return image
end