Cheetah textmate bundle

Stack 2.0 - Mongodb, node.js, redis, ios

Node.js seems to be the viable backend for scalable chat and game services

Mongodb seems to be the best tool for storing information on any backend game or mobile apps

When you compare this to what was available in 2005 , postgres/mysql and twisted/ejabberd - this is definitely a quantum improvement and 10gen definitely has made Mongodb the defacto backend replacing cassandra and many other backends

I wonder where redis fits into this ecosystem, i might presume it would be good as a caching option?

Jinja2 is good for templating, usage of jinja2 is enhanced by snippets in textmate

iOS is the winner among the smart phone oses

Apache mod_fcgid is the way to go for webpy

1. automatically spawns more processes based on traffic - lighttpd automatically the spawns the max number of processes even when no traffic - which is not useful if your max-procs is 100

2. usually need to set PYTHONPATH or append to sys.path inside code.py manually

import os,sys

sys.path.append( os.path.dirname(os.path.abspath(__file__)))

3. This is how you hook app.run for webpy

if __name__ == '__main__':

if apache:

web.wsgi.runwsgi = lambda func, addr=None: web.wsgi.runfcgi(func, addr)

app.run(session_mw)

else:app.run(session_mw)

The link to the related Webpy cookbook is here

Python Accessing Caller **locals() from callee method using sys._getframe

This is my first attempt at memcaching html page using cheetah

since cheetah render needs locals() i use getCallerInfo() to get the locals() and send to memcached

let me know if it is possible to better do this

notice getCallerInfo

utils.py

@log_time_func

def renderpage(key, htmlfile, deleteafter=3600):

from globaldb import mc

try:page = mc.get(key)

except:

page=None

clogger.info('except error mc.get '+ key)

if not page:

clogger.info(key+ ' rendering cheetah page')

terms = getCallerInfo(1)

#print terms

page = str(web.render(htmlfile, asTemplate=True, terms=terms))

try:mc.set(key, page, deleteafter)

except:

clogger.info('except error mc.set '+ key)

return page

@log_time_func

@memcachethis

def mcrenderpage(key, htmlfile, deleteafter=3600):

terms = getCallerInfo(2)

#print terms

return str(web.render(htmlfile, asTemplate=True, terms=terms))

def getCallerInfo(decorators=0):

'''returns locals of caller using frame.optional pass number of decorators\nFrom Dig deep into python internals http://www.devx.com/opensource/Article/31593/1954'''

f = sys._getframe(2+decorators)

args = inspect.getargvalues(f)

return args[3]

Usage

key=facebookstuff.APP_NAME+'newstart'+str(uid)

return utils.renderpage(key, 'pick.html')

Facebook sytle Notifications in couchdb

Goal: To prepare a facebook style notification using couchdb
schema
class Notification(Document):
unread = BooleanField(default=True)
activity = TextField()
appname = TextField()
image = TextField()
type = TextField(default='notification')
fromuser = TextField()
touser = TextField()
link = TextField()
message = TextField()
created = DateTimeField(default=datetime.datetime.now())
time =TimeField(default=datetime.datetime.now())
date = DateField(default=datetime.date.today()

def savenotifications(fromuser, tousers, link, message, activity, image, appname, *args, **kwargs):
for touser in tousers:
if fromuser == touser:continue
notidoc = Notification( fromuser = fromuser,touser=touser, link=link, message=message, activity=activity, image = image, appname = appname)
notidoc.store(fbcouchdb)
for touser in tousers:
r=fbcouchdb.view('_view/pop/unread',key=str(touser), count=1)
for i in r:i

Bold: I even try to prepare the views when some one gets a new notifications by precalling the view unread ahead of time when a new notification for that user is inserted inorder to cause forced view regeneration


def getnotifications(self, all=False):
try:
from couchmodel import fbcouchdb
if not all:
return [ (x.id, x.value) for x in fbcouchdb.view('_view/pop/unre
ad',key=str(self.uid), count=5) ]
else:
return [ (x.id, x.value) for x in fbcouchdb.view('_view/pop/noti
fications',key=str(self.uid)) ]
except:return[]


Problem:but whenever i do getnotification it is really slow.. And the page doesnt open forever. Anyways to make couchdb usable with this db? the situation is that notifications are constantly inserted into the db all the time as things happen to the user in real time. is this not a good use case for couchdb

Using couchdb to build a EMAIL messaging solution with threaded inline replies

Couchdb based Messaging system
The goal is to build an anonymous inbox with inline threaded replies and count of number of items in the inbox and sent. It works well, the only part that is hard is that the views are pretty slow the first time they are constructed

Schema

class Anonmessage(Document):
 subject = TextField()
 read = BooleanField()
 fromread = BooleanField()
 toread = BooleanField()
 msg = TextField()
 background = TextField()
 type = TextField()
 reply = TextField()
 fromuid = LongField()
 replycount = IntegerField()
 touid = LongField()
 created = DateTimeField(default=datetime.datetime.now())
 time =TimeField(default=datetime.datetime.now())
 date = DateField(default=datetime.date.today())

Permanent view for inbox:
function(d){ if(d.type != 'reply'){ emit(d.touid, d); if(d.touid != d.fromuid){ if (d.replycount>0) emit(d.fromuid, d); } } }

Count of number of messages in USER inbox
map:function(d){
if(d.type != 'reply'){
emit(d.touid, 1);
if(d.touid != d.fromuid){
if (d.replycount>0)
emit(d.fromuid, 1);
}
}
}

reduce:function(keys, values) { return sum(values)}

Count of Sent items
map:function(doc) { emit(doc.fromuid, 1) }
reduce:function(keys, values) { return sum(values)}

View to get replies
map:function(d) { if (d.type=="reply" ) emit(d.reply,d); }

Usage from python

inbox_messages = [ (x.id, x.value) for x in anondb.view('_view/truthbox/inbox',key=z.uid) ]

Ten Reasons why Couchdb is better than Mysql

1. No schema - schema less db, which means you can develop at the speed of your thought, you dont need to do a db update everytime you add a column
   
2. everything is over http, simple http, get post, put, delete requests which means works with varnish/squid out of the box (i prefer varnish because it is clean and simple)
3. Attachments - you can store file attachments for eg., your greeting card to grandma can carry images, music, flash
4. Map Reduce - no more sql queries, use amazingly scalable map-reduce based views. Views once saved are lighting fast
5. Sexy Futon javascript interface, comes with a cool js interface for displaying and editing data
6. javascript server using mozilla spidermonkey to construct views - means no need php in flash
   
7. zero config replication - work from home with no internet
8. python couchdb library
9. bulk updates, deletes - you can store 100000 docs in one post request
10. Each couchdb document is just a simple JSON compatible doc, no cruft just simple
11. (Bonus) Uses Erlang, which means it is scalable for multicore multiprocessor machines
12. (Extra Bonus) Low memory requirement Takes 150MB compared to 8Gig taken by Mysql for a similar db setup
13. similar to zodb, but much more cleaner and intuitive
14. Extremely friendly community and developers - Damien, Jchris, paul davis, noah slater, chris anderson, Jan
    

Python Api for last.fm

Pylast Python Api for last.fm

The best torrent search engine

I accidentally discovered this new torrent search engine... It is very clean, googley and really fast.. When I say blazingly fast... Incredibly high speeds... .02 seconds



Do give it a spin

Recommendations : Highly Recommended

python datetime guide

3. Dates and Times
Introduction

#-----------------------------
#introduction
# There are three common ways of manipulating dates in Python
# mxDateTime - a popular third-party module (not discussed here)
# time - a fairly low-level standard library module
# datetime - a new library module for Python 2.3 and used for most of these samples
# (I will use full names to show which module they are in, but you can also use
# from datetime import datetime, timedelta and so on for convenience)

import time
import datetime

print "Today is day", time.localtime()[7], "of the current year"
# Today is day 218 of the current year

today = datetime.date.today()
print "Today is day", today.timetuple()[7], "of ", today.year
# Today is day 218 of 2003

print "Today is day", today.strftime("%j"), "of the current year"
# Today is day 218 of the current year


Finding Today's Date

#-----------------------------
# Finding todays date

today = datetime.date.today()
print "The date is", today
#=> The date is 2003-08-06

# the function strftime() (string-format time) produces nice formatting
# All codes are detailed at http://www.python.org/doc/current/lib/module-time.html
print t.strftime("four-digit year: %Y, two-digit year: %y, month: %m, day: %d")
#=> four-digit year: 2003, two-digit year: 03, month: 08, day: 06

Converting DMYHMS to Epoch Seconds

#-----------------------------
# Converting DMYHMS to Epoch Seconds
# To work with Epoch Seconds, you need to use the time module

# For the local timezone
t = datetime.datetime.now()
print "Epoch Seconds:", time.mktime(t.timetuple())
#=> Epoch Seconds: 1060199000.0

# For UTC
t = datetime.datetime.utcnow()
print "Epoch Seconds:", time.mktime(t.timetuple())
#=> Epoch Seconds: 1060195503.0

Converting Epoch Seconds to DMYHMS

#-----------------------------
# Converting Epoch Seconds to DMYHMS

now = datetime.datetime.fromtimestamp(EpochSeconds)
#or use datetime.datetime.utcfromtimestamp()
print now
#=> datetime.datetime(2003, 8, 6, 20, 43, 20)
print now.ctime()
#=> Wed Aug 6 20:43:20 2003

# or with the time module
oldtimetuple = time.localtime(EpochSeconds)
# oldtimetuple contains (year, month, day, hour, minute, second, weekday, yearday, daylightSavingAdjustment)
print oldtimetuple
#=> (2003, 8, 6, 20, 43, 20, 2, 218, 1)

Adding to or Subtracting from a Date

#-----------------------------
# Adding to or Subtracting from a Date
# Use the rather nice datetime.timedelta objects

now = datetime.date(2003, 8, 6)
difference1 = datetime.timedelta(days=1)
difference2 = datetime.timedelta(weeks=-2)

print "One day in the future is:", now + difference1
#=> One day in the future is: 2003-08-07

print "Two weeks in the past is:", now + difference2
#=> Two weeks in the past is: 2003-07-23

print datetime.date(2003, 8, 6) - datetime.date(2000, 8, 6)
#=> 1095 days, 0:00:00

#-----------------------------
birthtime = datetime.datetime(1973, 01, 18, 3, 45, 50) # 1973-01-18 03:45:50

interval = datetime.timedelta(seconds=5, minutes=17, hours=2, days=55)
then = birthtime + interval

print "Then is", then.ctime()
#=> Then is Wed Mar 14 06:02:55 1973

print "Then is", then.strftime("%A %B %d %I:%M:%S %p %Y")
#=> Then is Wednesday March 14 06:02:55 AM 1973

#-----------------------------
when = datetime.datetime(1973, 1, 18) + datetime.timedelta(days=55)
print "Nat was 55 days old on:", when.strftime("%m/%d/%Y").lstrip("0")
#=> Nat was 55 days old on: 3/14/1973

Difference of Two Dates

#-----------------------------
# Dates produce timedeltas when subtracted.

diff = date2 - date1
diff = datetime.date(year1, month1, day1) - datetime.date(year2, month2, day2)
#-----------------------------

bree = datetime.datetime(1981, 6, 16, 4, 35, 25)
nat = datetime.datetime(1973, 1, 18, 3, 45, 50)

difference = bree - nat
print "There were", difference, "minutes between Nat and Bree"
#=> There were 3071 days, 0:49:35 between Nat and Bree

weeks, days = divmod(difference.days, 7)

minutes, seconds = divmod(difference.seconds, 60)
hours, minutes = divmod(minutes, 60)

print "%d weeks, %d days, %d:%d:%d" % (weeks, days, hours, minutes, seconds)
#=> 438 weeks, 5 days, 0:49:35

#-----------------------------
print "There were", difference.days, "days between Bree and Nat."
#=> There were 3071 days between bree and nat

Day in a Week/Month/Year or Week Number

#-----------------------------
# Day in a Week/Month/Year or Week Number

when = datetime.date(1981, 6, 16)

print "16/6/1981 was:"
print when.strftime("Day %w of the week (a %A). Day %d of the month (%B).")
print when.strftime("Day %j of the year (%Y), in week %W of the year.")

#=> 16/6/1981 was:
#=> Day 2 of the week (a Tuesday). Day 16 of the month (June).
#=> Day 167 of the year (1981), in week 24 of the year.

Parsing Dates and Times from Strings

#-----------------------------
# Parsing Dates and Times from Strings

time.strptime("Tue Jun 16 20:18:03 1981")
# (1981, 6, 16, 20, 18, 3, 1, 167, -1)

time.strptime("16/6/1981", "%d/%m/%Y")
# (1981, 6, 16, 0, 0, 0, 1, 167, -1)
# strptime() can use any of the formatting codes from time.strftime()

# The easiest way to convert this to a datetime seems to be;
now = datetime.datetime(*time.strptime("16/6/1981", "%d/%m/%Y")[0:5])
# the '*' operator unpacks the tuple, producing the argument list.

Printing a Date

#-----------------------------
# Printing a Date
# Use datetime.strftime() - see helpfiles in distro or at python.org

print datetime.datetime.now().strftime("The date is %A (%a) %d/%m/%Y")
#=> The date is Friday (Fri) 08/08/2003

High-Resolution Timers

#-----------------------------
# High Resolution Timers

t1 = time.clock()
# Do Stuff Here
t2 = time.clock()
print t2 - t1

# 2.27236813618
# Accuracy will depend on platform and OS,
# but time.clock() uses the most accurate timer it can

time.clock(); time.clock()
# 174485.51365466841
# 174485.55702610247

#-----------------------------
# Also useful;
import timeit
code = '[x for x in range(10) if x % 2 == 0]'
eval(code)
# [0, 2, 4, 6, 8]

t = timeit.Timer(code)
print "10,000 repeats of that code takes:", t.timeit(10000), "seconds"
print "1,000,000 repeats of that code takes:", t.timeit(), "seconds"

# 10,000 repeats of that code takes: 0.128238644856 seconds
# 1,000,000 repeats of that code takes: 12.5396490336 seconds

#-----------------------------
import timeit
code = 'import random; l = random.sample(xrange(10000000), 1000); l.sort()'
t = timeit.Timer(code)

print "Create a list of a thousand random numbers. Sort the list. Repeated a thousand times."
print "Average Time:", t.timeit(1000) / 1000
# Time taken: 5.24391507859

Short Sleeps

#-----------------------------
# Short Sleeps

seconds = 3.1
time.sleep(seconds)
print "boo"

Program: hopdelta

#-----------------------------
# Program HopDelta
# Save a raw email to disk and run "python hopdelta.py FILE"
# and it will process the headers and show the time taken
# for each server hop (nb: if server times are wrong, negative dates
# might appear in the output).

import datetime, email, email.Utils
import os, sys, time

def extract_date(hop):
# According to RFC822, the date will be prefixed with
# a semi-colon, and is the last part of a received
# header.
date_string = hop[hop.find(';')+2:]
date_string = date_string.strip()
time_tuple = email.Utils.parsedate(date_string)

# convert time_tuple to datetime
EpochSeconds = time.mktime(time_tuple)
dt = datetime.datetime.fromtimestamp(EpochSeconds)
return dt

def process(filename):
# Main email file processing
# read the headers and process them
f = file(filename, 'rb')
msg = email.message_from_file(f)

hops = msg.get_all('received')

# in reverse order, get the server(s) and date/time involved
hops.reverse()
results = []
for hop in hops:
hop = hop.lower()

if hop.startswith('by'): # 'Received: by' line
sender = "start"
receiver = hop[3:hop.find(' ',3)]
date = extract_date(hop)

else: # 'Received: from' line
sender = hop[5:hop.find(' ',5)]
by = hop.find('by ')+3
receiver = hop[by:hop.find(' ', by)]
date = extract_date(hop)

results.append((sender, receiver, date))
output(results)

def output(results):
print "Sender, Recipient, Time, Delta"
print
previous_dt = delta = 0
for (sender, receiver, date) in results:
if previous_dt:
delta = date - previous_dt

print "%s, %s, %s, %s" % (sender,
receiver,
date.strftime("%Y/%d/%m %H:%M:%S"),
delta)
print
previous_dt = date

def main():
# Perform some basic argument checking
if len(sys.argv) != 2:
print "Usage: mailhop.py FILENAME"

else:
filename = sys.argv[1]
if os.path.isfile(filename):
process(filename)
else:
print filename, "doesn't seem to be a valid file."

if __name__ == '__main__':
main()

svn deps

Cheetah
Psycopg2
PIL
mx egenix
sudo python setup.py build --skip install
aspell.so

How to update to Fedora 7

This is how I updated my FC6 to F7:

wget ftp://download.fedora.redhat.com/pub/fedora/linux/releases/7/Fedora/i386/os/Fedora/fedora-release-7-3.noarch.rpm
wget ftp://download.fedora.redhat.com/pub/fedora/linux/releases/7/Fedora/i386/os/Fedora/fedora-release-notes-7.0.0-1.noarch.rpm
rpm -U fedora-release-7-3.noarch.rpm fedora-release-notes-7.0.0-1.noarch.rpm
yum update

In theory that’s the only thing you need to do, in practice it’s not that easy. I had some dependency issues with pidgin and liferea so I just removed them:

rpm -e pidgin liferea

Then yum complained about libexif being “not signed”. I tried to edit /etc/yum.conf and setting gpgcheck=0, that didn’t work, yum was still checking it. So I had to manually set gpgcheck=0 in all the repos on /etc/yum.repos.d/. Then it worked.

That’s it!

Adding a fast random row to a table in postgresql

ALTER TABLE posts ADD myrand DOUBLE PRECISION;UPDATE posts SET myrand = RANDOM(); CREATE INDEX myrand_posts ON posts(myrand,id); ANALYZE VERBOSE posts;
ALTER TABLE posts ALTER myrand
SET DEFAULT RANDOM();
ALTER TABLE posts ALTER myrand
SET NOT NULL;
--SELECT * FROM posts WHERE myrand >= (SELECT RANDOM() OFFSET 0) ORDER BY myrand ASC LIMIT 1;

Boolean column in a table in postgres

CREATE INDEX polls_active_idx ON polls(done) WHERE done='t';

how to get auto align css centering ala mybloglog and facebook

body { width:99%;max-width:1024px; margin:auto;text-align:center; }
#wrap { margin: auto; width:85%; text-align:left; }
#main { float:left; width:74%; margin-top:1%; }
#sidebar { float:right;max-width: 25%;width:25%;min-width: 5%; padding:3px; }
#footer { background:#cc9; clear:both; }

tech tools

Python

postgres

Amazon S3

webpy.org

Beautiful soup

lighttpd or nginx

vim and emacs

Determining who's blocking who in Postgres

Determining who's blocking who in Postgres
If you have databases like I do with lots of concurrent queries, you can sometime run into situations where you issue a query and it just hangs there blocked. Or, more likely somebody or something issues a query and then comes calling when it doesn't seem to be doing anything.

Of course, you have the handy pg_stat_activity and pg_locks views at your disposal, but when it comes to determining exactly which queries are blocking which others and on what table, querying those alone is a tedious way to get the answer. What you really need is a query that sums it all up, in one neat and tidy bundle. Well, my friends here is such a query:

SELECT
bl.procpid as blocked_pid,
bl.usename as user,
bl.current_query as blocked_query,
bl.query_start,
relname as blocked_on ,
lq.procpid as blocking_pid,
lq.usename as user,
lq.current_query as blocking_query,
lq.query_start,
pgl2.mode as lock_type
FROM pg_stat_activity bl, pg_locks pgl1,
pg_stat_activity lq, pg_locks pgl2, pg_class
WHERE bl.procpid = pgl1.pid
AND not pgl1.granted
AND pg_class.oid = pgl1.relation
AND pgl2.relation = pgl1.relation
AND pgl2.granted
AND lq.procpid = pgl2.pid;

In extended mode (\x) psql returns something along these lines for this query:

blocked_pid | 21418
user | sueuser
blocked_query | insert values ('foo', 'bar', 'baz')
into extremely_large_table;
query_start | 2007-02-13 15:14:06.77606-08
blocked_on | extremely_large_table
blocking_pid | 21417
user | joeuser
blocking_query | delete from extremely_large_table;
query_start | 2007-02-13 14:45:34.637675-08
lock_type | AccessExclusiveLock

python spell checker

How to Write a Spelling Corrector
In the past week, two friends (Dean and Bill) independently told me they were amazed at how Google does spelling correction so well and quickly. Type in a search like [speling] and Google comes back in 0.1 seconds or so with Did you mean: spelling. What surprised me is that I thought Dean and Bill, being highly accomplished engineers and mathematicians, would have good intuitions about statistical language processing problems such as spelling correction. But they didn't, and come to think of it, there's no reason they should. I figured they and many others could benefit from an explanation, and so on the plane back from my trip I wrote a toy spelling corrector, which I now share.

Let's get right to it. I figured that in less than a plane flight, and in less than a page of code, I could write a spelling corrector that achieves 80 or 90% accuracy at a rate of at least 10 words per second. And in fact, here, in 20 lines of Python 2.5 code, is the complete spelling corrector:

import re, string, collections

def words(text): return re.findall('[a-z]+', text.lower())

def train(features):
model = collections.defaultdict(lambda: 1)
for f in features:
model[f] += 1
return model

NWORDS = train(words(file('Documents/holmes.txt').read()))

def edits1(word):
n = len(word)
return set([word[0:i]+word[i+1:] for i in range(n)] + ## deletion
[word[0:i]+word[i+1]+word[i]+word[i+2:] for i in range(n-1)] + ## transposition
[word[0:i]+c+word[i+1:] for i in range(n) for c in string.lowercase] + ## alteration
[word[0:i]+c+word[i:] for i in range(n+1) for c in string.lowercase]) ## insertion

def known_edits2(word):
return set(e2 for e1 in edits1(word) for e2 in edits1(e1) if e2 in NWORDS)

def known(words): return set(w for w in words if w in NWORDS)

def correct(word):
return max(known([word]) or known(edits1(word)) or known_edits2(word) or [word],
key=lambda w: NWORDS[w])

This defines the function correct, which takes a word as input and returns a likely correction of that word. For example:

>>> correct('speling')
'spelling'
>>> correct('korrecter')
'corrector'

mencoder convert to flash

mencoder pasta_maken_311205.avi -o video.flv -of lavf -ovc lavc -oac lavc -lavcopts vcodec=flv:vbitrate=500:autoaspect:mbd=2:mv0:trell:v4mv:cbp:last_pred=3:predia=2:dia=2:precmp=2:cmp=2:subcmp=2:preme=2:turbo:acodec=mp3:abitrate=56 -vf scale=320:240 -srate 22050 -af lavcresample=22050

thread control clean

An example of an idiom for controling threads

Doug Fort
http://www.dougfort.net
"""

import threading

class TestThread(threading.Thread):
"""
A sample thread class
"""

def __init__(self):
"""
Constructor, setting initial variables
"""
self._stopevent = threading.Event()
self._sleepperiod = 1.0

threading.Thread.__init__(self, name="TestThread")

def run(self):
"""
overload of threading.thread.run()
main control loop
"""
print "%s starts" % (self.getName(),)

count = 0
while not self._stopevent.isSet():
count += 1
print "loop %d" % (count,)
self._stopevent.wait(self._sleepperiod)

print "%s ends" % (self.getName(),)

def join(self,timeout=None):
"""
Stop the thread
"""
self._stopevent.set()
threading.Thread.join(self, timeout)

if __name__ == "__main__":
testthread = TestThread()
testthread.start()

import time
time.sleep(10.0)

testthread.join()