InternetSchminternet

Advice: scaling to 1 Billion pages view per month

Matt.stone — Wed, 09 Jul 2008 22:38:39 -0700

Here is a great article describing how Jim Meyer, Manager of Linked In's Light Engineering Development scaled his facebook application, Bumpersticker, to over 1 Billion page views per month.

In a nutshell, his three design guidlines are :

Talking to spindles is bad - talking to file systems and databases has to be minimized
Dynamic content is your enemy. Anything you can turn into static, including periodic re-caching should be removed.
Push everything you possibly can to the client to reduce the amount of traffic going over the network no matter where the client is.

Thanks to the guys at RailsEnvy for the tip.

Building code quality into the culture of a development team.

Matt.stone — Wed, 09 Jul 2008 22:05:29 -0700

I like to build code quality in the culture of development team. Being proud of the code quality is something that I want my developers to aspire to. I actively manage towards this by focusing on shipping high quality production code; not on technical issues [I'm pretty agnostic when it comes to technology].

The three management tools I use to achieve this are:

Agile development methodologies
Building code quality into programmer KPIs
Make programmer KPI measurements public

Agile Development Methodologies

By this I mean:

Small milestones of production quality code & all stakeholders having visibility of project progress
Programmer commits own tests with code
At least a daily build, or preferably a build with every commit & the team is notified of a commit that breaks the build; ie: cruisecontrol
Daily team meetings [ie:scrum] that update on progress

Stake holders should be able to see an up to date view of project progress & quality. Issues should be identified early & remedial action can be taken before a small issue becomes a large issue.

I like to have a stated code testing/quality assurance strategy. With Tickex, we had commercial pressures to ship code very fast, & change often so I settled up on the follow [its tailored for a Ruby on Rails project]:

Standardise environments [development, test, stage, production]. Developers can use their preferred tools, but commited code must work on the standardised environment.
Aim for 100% Rspec Model coverage
Cruisecontrol runs tests on every commit & notifies team if build is broken
Utilise Selenium for user land testing, rather than testing controller actions.
Performance testing using Hpricot & Selenium Grid - - to script concurrent user sessions
Manual tests for muliple browsers [ok as long as the feature set is not massive]

This is a balances providing the high level of testing coverage with the ability to rapidly turn around features. As a result of this, we have experienced zero downtime from code issues, and very few bugs have made it into production [& most of those have been specification issues, rather than code].

Build Code Quality into Programmer KPIs
Every environment is different, but here are some guidelines. Each metric on its own is pretty much worthless, but the whole will build a picture.

Quantative KPIs

lines of code
milestones met or features delivered
bugs in code
number of times commits break build

Note: lines of code is a bit controversial as on its own its a pretty meaningless measure [a good programmer will do something in many lines less than a bad programmer, but will also complete more features]. I have a couple of MBA type CEO's who have really struggled with this. Patterns should be investigated though.

Qualitative KPIs

Peer reviews - both quality of code & as a team member
Customer/Stakeholder feedback - satisfaction with work
Test coverage of code
Team fit

Make programmer KPI measurements public
Making these measure available public, on a wiki, & in regular team meetings is intended to build a team spirit. Peer pressure can be a better motivator than agressive management.

I don't favour an agressive leadership style. Fear is one motivitor,but I don't believe it leads to habits being formed & long term behaviourial change. I think that respect of a manager & peer pressure lead to long lasting habits being formed. This is not to say be a push over, think hard but fair; If someone is not performing, I put the ownus upon the team to sort the issue quickly, then act accordingly and decisively if needed. The message will be seen that code quality cannot be compromised.

This philosophy is independent of team size or technology. Whether a small ruby/php team or a large .Net or Java team it does not matter to me. What matters is predictably shipping high quality production code that meets customer requirements.

Google now hosting Ajax libraries - w00t!

Matt.stone — Mon, 07 Jul 2008 06:52:38 -0700

Google's new Ajax libaries API is a great way to help offload bandwdith & speed up your sites performance. Google hosts the most popular open source Javascript libraries on their content distribution network architecture. By using the Google AJAX API Loader's google.load() method you can access:

Here is a great article describing how to use this in Rails, although it will soon be in plugin form I believe.

Your chance to influence Google's web spam team

Matt.stone — Mon, 07 Jul 2008 06:38:07 -0700

Matt Cutts has a thread on his blog asking for what Google's web spam team should work on next. It makes for very interesting reading.

Topics of note:
- Adwords & Blogspot spam will probably soon be targeted
- Link farms and paid links are still working for people & Google can't seem to get onto of this.

Hopefully Google & Matt Cutts will start communicating back to the community a little better after this process.

Web designers rejoice!

Matt.stone — Mon, 07 Jul 2008 06:12:13 -0700

Google can now index flash

Well, kind of, sort of.. but its definitely better than it was.

Binary content such as images & FLV's will probably always remain invisible to the google spiders.

I think the real bummer is that if I build anything with the two most interesting client side technologies, flash & javascript - it won't get spidered & I won't get search engine traffic.

HTTP Basic Authentication

Matt.stone — Wed, 25 Jun 2008 01:19:52 -0700

At Tickex we have an API that is accessed via HTTP Basic Authentication.

Here are a few links for developers to read a little bit more about this.

c# dotnet
java
ruby - it's just one line using open_uri - open("http://www.example.com", :auth_info => [user, pwd]) do |f|
PHP

Ruby is awesome for this kind of stuff - a bit high in CPU usage, but your done in a few minutes.

IE & XML MIME Type

Matt.stone — Wed, 25 Jun 2008 01:15:31 -0700

It's been awhile since I used IE & had forgotten that it does not handle MIME Type XML very well. IE will treat an XML file as file to be downloaded; which if fine as you can save the file & look at in in notepad or a programming editor. It did cause me some confusion as I told it a client that it would render fine in their browser :-(

Try this url in Firefox, Safari, or Opera & they will actually render the data - Firefox is best as it keeps the XML formatting.

Google appengine fetch basic authentication

Matt.stone — Fri, 20 Jun 2008 07:33:10 -0700

Google's documentation for the appengine is a bit basic for those not to familiar with Python, like me.

If your building a mashup, your going to want to use the fetch routine to pull down data from a web service. This is one of those vaguely documented features.

Here is how it's done.

import base64
from google.appengine.api import urlfetch

url = "www.target_webservice.com"
username = "your_username"
password = "your_password"
base64string = base64.encodestring('%s:%s' % (username, password))
headers = {'Authorization': "Basic %s" % base64string}
result = urlfetch.fetch(url, headers=headers)

if result.status_code == 200:
all is good
else:
something went wrong

Why are there no rails CMS like Joomla?

Matt.stone — Fri, 20 Jun 2008 01:44:27 -0700

If you compare the open source Content Management Systems available in rails, with those availble in PHP, Java, Python [ie: Django], and even dotnet, I think that you'll be disappointed by the feature sets of the rails
software.

Although lovdbyless & the community engine plugin, whilst not strictly being CMS's provide a social network/blog straight out of the box with a feature set that exceeds the PHP CMS's - for the social networking side of things.

This is not such a big surprise. Firstly the majority of the open source CMS's [ie: Joomla & Django] are quite mature compared to the rails options. Secondly, and much more importantly [having battled with Joomla quite a bit] .. I think the Rails way would be to build it yourself.

Restful authentication, Restful scaffolding, layouts, plugins, & the mentioned above make it possible to build something quite quickly.

Building it yourself means your not constrained by what someone else thinks you needs. The flipside is you need to have the chops to develop what you want. Just as importantly, you need to know what you want.

On the ruby on rails home page there is a screencast of someone building a blog in 15 minutes. Its not very pretty & its not something I would ever put into production use, but you get the idea of rapid development.

rake & the test database

Matt.stone — Fri, 20 Jun 2008 01:26:50 -0700

Gentle reminder:

Never, ever point your test environment at your production db.

Here are the comments from database.yml

# Warning: The database defined as 'test' will be erased and
# re-generated from your development database when you run 'rake'.
# Do not set this db to the same as development or production.

enough said.