Coders, Musicians and Cooks

In a light hearted Twitter exchange yesterday I asked why so many coders also played guitar. Mark Seemann suggested that there was also a high correlation with cooking.

How about a survey? Mark and I have over 4000 twitter followers between us so I was sure we could get a few of them to click a few radio buttons, with no promise of reward, but with the warm feeling that they were helping to progress the anthropological understanding of the coding tribe. So I created a survey and waited for the results to stream in. Of course, as Rob Pickering pointed out, this kind of self selecting survey is very unscientific, but fun nonetheless.

What have we learnt?

1. It is ridiculously easy to create an online survey using Google Drive. Just click ‘new form’, type in a few questions, and Bob’s yer uncle. It automatically builds a nice summary graphic on the fly as the survey progresses (see below).
2. Most of Mark and I’s followers are C# coders. 81%. That’s not really surprising.
3. More coders cook than play musical instruments: 85% verses 53%. That’s not really surprising either. I expect if you took a survey of the general population you’d find quite a high percentage who cook verses those who play.
4. Guitar is by far the most popular instrument among coders. 30% play guitar. This confirms my hunch, but doesn’t answer my question – why it’s such a high percentage.

Here is the Google Drive graphic of the results.

Fractured Product Syndrome

How do you sell your software product? Do you fork the source code of your previous customer, modify it a little and deploy it as its own instance with its own database? Maybe each customer even has their own server? Their own set of servers?

You are suffering from the Fractured Product Syndrome.

Let me tell you a story.

Colin and Alex have a little web development consultancy, called Coaltech (get it?) they are pretty handy with ASP.NET and SQL Server and already had a respectable list of satisfied clients when a local car hire company, Easy Wheels, asked them if they could build them a car booking system. It would keep a list of Easy Wheels’ cars and allow customers to view them and book them online.

Colin and Alex signed a contract with Easy Wheels and got busy. Their “just do it” attitude to getting features done meant that the code wasn’t particularly beautiful, but it did the job and after a couple of months work the system went live. Easy Wheels were very happy with their new system, so happy in fact that they started to tell other independent car rental companies about it. A few months later Colin and Alex got a call from Hire My Ride. They loved what Coaltech had done for Easy Wheels and wanted a similar system. Of course it would have to be redesigned to fit in with Hire My Ride’s branding, but the basic functionality was pretty much identical. Colin and Alex did the obvious thing, they took a copy of the Easy Wheels code and tweaked it for Hire My Ride. Of course they charged the same bespoke software price for Hire My Ride’s system.

Soon a third and fourth car hire company had asked Coaltech for their ‘system’. Each time Colin and Alex took a copy of the last customer’s code – it made sense to take the most recent code because they inevitably added a few refinements with each customer – and then they altered it to match the new customer’s requirements. Before long they had arranged a deal with another web agency to take over all their non-car-hire work and decided to concentrate full time on the ‘system’. It needed a name too, they couldn’t carry on calling it ‘Easy Wheels’ and they certainly couldn’t market it like that. Soon it became ‘Coaltech Cars’ with a new marketing website. They also found that they couldn’t keep up with demand with just the two of them doing customer implementations. Each new customer took around six weeks of development work with the inevitable to-and-fro of slightly different requirements and design changes. To help meet demand they started to hire developers. First Jez, then Dave, then Neville. They all modified the software in slightly different ways, but it didn’t seem to matter at the time.

Fast forward five years. Coaltech now has 50 employees and around 50 customers. It’s fair to say that they are the leading vendor of independent car rental management systems. The majority of their staff are developers, although they also have a small sales, HR and customer relationship management team. You would have thought that Colin and Alex would be happy to have turned their little web development company into such a success, but instead life just seemed to be one long headache. Although the company had grown, it always seemed hard to turn a profit. As it was customers often balked at the price of the system. The same bug would turn up again and again for different customers. There never seemed to be enough time to fix them all. Even though they might have delivered a new feature for one client, it always seemed to take just as long to implement it for another; sometimes longer depending on what code they’d been forked from. The small team that looked after the servers were in constant fire fighting mode and got very upset when anyone suggested ‘upgrading’ any of the clients -  it always meant bugs, downtime and screaming customers. And then the government changed the Value Added Tax rate. Colin had to cancel his holiday and they lost two of their best developers after several weeks of late nights and no weekends while they updated and redeployed 50 separate web applications.

The end for Coaltech cars came slowly and painfully. Alex had the first hint of it when he was made aware of a little company called RentBoy. They had a little software-as-a-service product for small car hire companies. To use it you entered your credit card number, a logo, a few other details, and you were good to go. They weren’t any immediate competition for Coaltech Cars, having only a small subset of the features, but they soon captured the low end of the market, the one or two man band car companies that had never been able to afford Coaltech’s sign up or licensing fees.

But then the features started coming thick and fast and soon Coaltech found they were loosing out to RentBoy when bidding for clients. Colin found an article on the High Scalability blog about RentBoy’s architecture. They had a single scalable application that served all their customers, one place to fix bugs, one point of scalability and maintenance. They practiced continuous deployment allowing them to quickly build and release new features for all their customers. The company had four employees and already more customers than Coaltech. They charged about a tenth of Coaltech’s fees.

Coaltech’s new client revenues soon dried up. They’d always made a certain amount of money from sign-up fees. Too late they realised that they had to start shedding staff, always a painful thing for a small closely knit company. The debts mounted until the bank would no longer support them, and before long they had to declare bankruptcy. Luckily Colin and Alex managed to get jobs as project managers in enterprise development teams.

The moral of the story? Try to avoid Fractured Product Syndrome if you possibly can. Although simply forking the source code for each new customer appears by far the easiest thing to do at the start, it simply doesn’t scale. Start thinking about how to build multi-tenanted software-as-a-service long before you get to where Coaltech got to. Learn to say ‘no’ to customers if you have to. It’s far better to have a high number of low value customers than smaller numbers of higher value ones on a differentiated platform. It’s much easier for a low-value volume software provider to move into the high-value space than for a high-value provider to move down.

Learn to recognise Fractured Product Syndrome and address it before it gets serious!

EasyNetQ on .NET Rocks!

Last week I had the pleasure of being interviewed by Carl Franklin and Richard Campbell for a .NET Rocks episode on RabbitMQ and EasyNetQ. It was terrific fun and a real honour to be invited on the show. I’ve been listening to .NET Rocks since it started in 2002 so you can imagine how excited I was. Carl and Richard are seasoned pros when it comes to podcasting, and the awesome ninja editing skills they posses turned my rather hesitant and rambling answers into something that almost sounded coherent.
You can listen to the show on the link below:
http://www.dotnetrocks.com/default.aspx?ShowNum=848
Now Richard, about that Tesla …

EasyNetQ in Africa

Anthony Moloney got in touch with me recently. He’s using EasyNetQ, my simple .NET API for RabbitMQ, with his team in Kenya. Here’s an email he sent me:

Hi Mike,
Further to the brief twitter exchange today about using EasyNetQ on our Kenyan project. We started using EasyNetQ back in early November and I kept meaning to drop you a line to thank you for all your good work.
Virtual City are based in Nairobi and supply mobile solutions to the supply chain and agribusiness industry in Africa. African solutions for African problems. I got involved with them about 2 years ago to help them improve the quality of their products and I have been working on and off with them since then. Its been a bit of a journey we are getting there.
We have a number of client applications including android and wpf working in an online/offline mode over mobile networks. We need to process large amounts of incoming commands from these applications. These commands are also routed via the server to other client apps.
The application had originally used MVC and SQL server to synchronously process and store the commands but we were running into severe performance problems. We looked at various MQ solutions and decided to use RabbitMQ, WebApi & Mongo to improve processing throughput. While researching a .Net API for RabbitMQ I noticed that you had created the EasyNetQ API.
EasyNetQ greatly simplifies interacting with RabbitMQ and providing your needs are not too complicated you really don't need to know too much about the guts of RabbitMQ. We replaced the existing server setup in about a week. The use of RabbitMQ has greatly increased the scalability of the product and allows us to either scale up or scale out.
We are also using the EasyNetQ management API for monitoring queue activity on our customer services dashboard.
Kind Regards
Anthony Moloney

One of the great rewards of running an open source project is hearing about the fascinating ways that it’s used around the world. I really like that it’s an ‘African solution for African problems’ and built by a Kenyan development team. It’s also interesting that they’ve used OSS projects like RabbitMQ and Mongo alongside .NET. It reminds me of the Stack Overflow architecture, a .NET core surrounded by OSS infrastructure.

How to Write Scalable Services

I’ve spent the last five years implementing and thinking about service oriented architectures. One of the core benefits of a service oriented approach is the promise of greatly enhanced scalability and redundancy. But to realise these benefits we have to write our services to be ‘scalable’. What does this mean?

There are two fundamental ways we can scale software: 'Vertically' or 'horizontally'.

• Vertical Scaling addresses the scalability of a single instance of the service. A simple way to scale most software is simply to run it on a more powerful machine; one with a faster processor or more memory. We can also look for performance improvements in the way we write the code itself. An excellent example of company using this approach is LMAX. However, there are many drawbacks to the vertical scaling approach. Firstly the costs are rarely linear; ever more powerful hardware tends to be exponentially more expensive and the costs (and constraints) of building sophisticated performance optimised software are also considerable. Indeed premature performance optimisation often leads to overly complex software that's hard to reason about and therefore more prone to defects and high maintenance costs. Most importantly, vertical scaling does not address redundantcy; vertically scaling an application just turns a small single point of failure into a large single point of failure.

• Horizontal Scaling. Here we run multiple instances of the application rather than focussing on the performance of a single instance. This has the advantage of being linearly scalable; rather than buying a bigger, more expensive box, we just buy more copies of the same cheap box. With the right architectural design, this approach can scale massively. Indeed it's the approach taken by almost all of largest internet scale companies: Facebook, Google, Twitter etc.. Horizontal Scaling also introduces redundancy; the loss of a single node need not impact the system as a whole. For these reasons, horizontal scaling is the preferred approach to building scalable, redundant systems.

So, the fundamental approach to building scalable systems is to compose them of horizontally scaled services. In order to do this we need to follow a few basic principles:

• Stateless. Any services that stores state across an interaction with another service is hard to scale. For example, a web service that stores in-memory session state between requests requires a sophisticated session-aware load balancer. A stateless service, by contrast, only requires simple round-robin load balancing. For a web application (or service) you should avoid using session state or any static or application level variables.

• Coarse Grained API. To be stateless, a service should expose an API that exposes operations as a single interaction. A chatty API, where one sets up some data, asks for some transition, and then reads off some results, implies statefulness by its design. The service would need to identify a session and then maintain information about that session between successive calls. Instead a single call, or message, to the service should encapsulate all the information that the service requires to complete the operation.

• Idempotent. Much scalable infrastructure is a trade-off between competing constraints. Delivery guarantees are one of these. For various reasons it's is far simpler to guarantee 'at least once' delivery than 'exactly once'. If you can make your software tolerant of multiple deliveries of the same message it will be easier to scale.

• Embrace Failure. Arrays of services are redundant if the system as a whole can survive the loss of a single node. You should design your services and infrastructure to expect and survive failure. Consider implementing a Chaos Monkey that randomly kills processes. If you start by expecting your services to fail, you'll be prepared when they inevitably do.

• Avoid instance specific configuration. A scalable service should be designed in such a way that it doesn't need to know about other instances of itself, or have to identify itself as a specific instance. I shouldn't need to have to configure one instance any differently than another. This would include communication mechanisms that require messages to be addressed to a specific instance of the service, or some non-convention based way that the service was required to identify itself. Instead we should rely on infrastructure (load-balancers, pub-sub messaging etc.) to manage the communication between arrays of services.

• Simple automated deployment. Have a service that can scale is no advantage if we can't deploy it when we are close to capacity. A scalable system must have automated processes to deploy new instances of services as the need arises.

• Monitoring. We need to know when services are close to capacity so that we can add additional service instances. Monitoring is usually an infrastructure concern; we should be monitoring CPU, network, and memory usage and have alerts in place to warn us when these pass certain trigger points. Sometimes it's worth introducing application specific alerts when some internal trigger is reached, such as the number of items in an in-memory queue, for example.

• KISS - Keep It Small and Simple. This is good advice for any software project, but is especially pertinent to building scalable resilient systems. Large monolithic codebases are hard to reason about, hard to monitor, and hard to scale. Building your system out of many small pieces makes it easy to address those pieces independently. Design your system so that each service has only one purpose and is decoupled from the operations of other services. Have your services communicate using non-proprietary open standards to avoid vendor lock-in and allow for a heterogeneous platform. JSON over HTTP, for example, is an excellent choice for intra-service communication. Every platform has HTTP and JSON libraries and there is abundant off-the-shelf infrastructure (proxies, load-balancers, caches) that can be used to help your system scale.

This post just gives a few pointers to building scalable systems, for far more detailed examples and case studies I can't recommend the High Scalability Blog enough. The Dodgy Coder blog has a very nice summary of some of the High Scalability case studies here.

Visual Studio: Multiple Startup Projects

I’ve been using Visual Studio for over 10 years, but still keep on learning new tricks. That’s because I learn things very slowly dear reader! Today’s ‘wow, I didn’t know you could do that moment’, was finding out that it’s possible to launch multiple startup projects when one hits F5 – or Control-F5 in this case.

My scenario is that I’m writing a little spike for a distributed application. Each of my services is implemented as a console app. During development, I want to run integration tests where the services all talk to each other, so before running the tests I want to run all the services. Now I’m used to right-clicking on a project and choosing ‘Set as startup project’, but you can’t select multiple projects this way and it’s very tedious to launch multiple projects by going to each one in turn, setting it as the startup project, and then hitting ctrl-F5. What I didn’t know is that you can right click on the Solution node, select ‘Set Startup Projects’ and you get this dialogue:

You can then select multiple startup projects and choose any number of them to ‘Start without debugging’.

Now I can hit ctrl-F5 and all my little services start up. Wonderful.

Converting Between Unix Time And DateTime

Unix time is the time value used in Unix based operating systems and is often exposed by Unix based APIs. To convert it to, or from, a .NET System.DateTime simply calculate the number of seconds since the Unix Epoch, midnight on the 1st January 1970. I’ve created a little class you can use to do just that. Note that the Unix Epoch is UTC, so you should always convert your local time to UTC before doing the calculation.

public class UnixTime
{
    private static readonly DateTime epoch = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);

    public static long FromDateTime(DateTime dateTime)
    {
        if (dateTime.Kind == DateTimeKind.Utc)
        {
            return (long)dateTime.Subtract(epoch).TotalSeconds;
        }
        throw new ArgumentException("Input DateTime must be UTC");
    }

    public static DateTime ToDateTime(long unixTime)
    {
        return epoch.AddSeconds(unixTime);
    }

    public static long Now
    {
        get
        {
            return FromDateTime(DateTime.UtcNow);
        }
    }
}

You can convert from Unix time to a UTC DateTime like this:

var calculatedCurrentTime = UnixTime.ToDateTime(currentUnixTime);

Convert to Unix time from a UTC DateTime like this:

var calculatedUnixTime = UnixTime.FromDateTime(myDateTimeValue);

And get the current time as a UTC time value like this:

Console.Out.WriteLine(UnixTime.Now);

 

As an interesting aside, the 32 bit signed integer used in older Unix systems will overflow at 14 minutes past 3 o’clock and 7 seconds on the morning of 19th January 2038 and interpret the date as 1901. I shall come out of retirement and spend happy well paid hours as a ‘unix time consultant’. 64 bit systems will overflow in the year 292,277,026,596.

A Geek Christmas Quiz–The Answers!

Thanks for everyone who had a go at my Geek Christmas Quiz. The response was fantastic with both Iain Holder and Rob Pickering sending me emails of their answers. I’m pretty sure neither of them Googled any of the questions, since their scores weren’t spectacular :)

So, now the post you’ve all been waiting for with such anticipation… the answers!

Computers

1. G.N.U stands for GNU is Not Unix. A recursive acronym, how geeky is that?
2. The A in ARM originally stood for ‘Acorn’ as in Acorn Risc Machine. Yes, I know it stands for ‘Advanced’ now, but the question said ‘originally’.
3. TCP stands for Transmission Control Protocol.
4. Paul Allen founded Microsoft with Bill  Gates. I’ve just finished reading his memoirs ‘Ideas Man’. Hard work!
5. F2 (hex) is 15(F) * 16 + 2 = 242.  1111 0010 (binary)
6. Windows ME was based on the Balmer Peak theory of software development.
7. The first programmer was Ada Lovelace. Yes yes, I know that’s contentious, but I made up this quiz, so there!
8. UNIX time started in 1970 (1st January to be exact). I know this because I just had to write a System.DateTime to UNIX time converter.
9. SGI, the mostly defunct computer maker. You get a mark for Silicon Graphics International (or Inc).
10. Here’s monadic ‘Bind’ in C#: M<B> Bind<A,B>(M<A> a, Func<A, M<B>> func)

Name That Geek!

1. Bill Gates – Co-founder of Microsoft with Paul Allen.
2. Tim Berners-Lee – Creator of the World Wide Web.
3. Larry Ellison – Founder of Oracle. Lives in a Samurai House (how geeky is that?)
4. Linus Torvalds – Creator of Linux.
5. Alan Turing – Mathematician and computer scientist. Described the Turing Machine. Helped save the free world from the Nazis.
6. Steve Jobs – Founded Apple, NeXT and Pixar.
7. Natalie Portman – Actress and self confessed geek.

Science

1. The four ‘letters’ of DNA are C, G, T and A. If you know the actual names of the nucleotides (guanine, adenine, thymine, and cytosine), give yourself a bonus point – you really are a DNA geek!
2. The ‘c’ in E = mc² is a constant, the speed of light.
3. The next number in the Fibonacci sequence 1 1 2 3 5 8 is 13 (5 + 8).
4. C₈H₁₀N₄0₂ is the chemical formula for caffeine.
5. According to Wikipedia, Australopithecus, the early hominid, became extinct around 2 million years ago.
6. You would not find an electron in an atomic nucleus.
7. Nitrogen is the most common gas in the Earth’s atmosphere.
8. The formula for Ohm’s Law is I = V/R (current = voltage / resistance).
9. A piece of paper that, when folded in half, maintains the ratio between the length of its sides, has sides with a length ratio of 1.618, ‘the golden ratio’. Did you know that the ratio between successive Fibonacci sequence numbers also tends to the golden ratio? Maths is awesome!
10. The closest living land mammal to the cetaceans (including whales) is the Hippopotamus.

Space

1. The second (and third) stage of the Apollo Saturn V moon rocket was powered by five J2 rocket engines.
2. Saturn’s largest moon is Titan. Also the only moon in the solar system (other than our own) that a spaceship has landed on.
3. You would experience 1/6th of the Earth’s gravity on the moon. Or there about.
4. This question proved most contentious. The answer is false, there is nowhere in space that has no gravity. Astronauts are weightless because they are in free-fall. Gravity itself is a property of space.
5. A Geosynchronous spaceship has an orbital period of 24 hours. So it appears to be stationary to a ground observer.
6. The furthest planet from the sun is Neptune. Far fewer people know this, than know that Pluto used to be the furthest planet from the sun. Actually Pluto was only the furthest for part of it’s, irregular, orbit.
7. There are currently 6 people aboard the International Space Station.
8. According to Google (yes, I know) there are 13,000 earth satellites.
9. Prospero was the only satellite built and launched by the UK. It was launched by stealth after the programme had been cancelled, that’s the way we do things in the UK.
10. The second man on the moon was Buzz Aldrin. He’s never forgiven NASA.

Name That Spaceship!

In this round, give yourself a point if you can name the film or TV series the fictional spacecraft appeared in.

1. Red Dwarf. Sorry, you probably have to be British to get this one.
2. Space 1999. Sorry, you really have to be British and over 40 to get this one … or a major TV space geek.
3. Voyager. Difficult, interplanetary probes all look similar.
4. Apollo Lunar Excursion Module (LEM). You can have a point for ‘Lunar Module’, but no, you don’t get a point for ‘Apollo’. Call yourself a geek?
5. Skylab. The first US space station, made out of old Apollo parts. Not many people get this one. A read a whole book about it, that’s how much of a space geek I am.
6. Darth Vader’s TIE fighter. You can have a point for ‘TIE Fighter’. You can’t have a point for ‘Star Wars’. Yes yes, I know I’m contradicting myself, but, come on, every geek should know this.
7. Curiosity. No, no points for ‘Mars Rover’.
8. 2001 A Space Odyssey. Even I don’t know what the ship is called.
9. Soyuz. It’s been used by the Russians to travel into space since 1966. 46 years! It’s almost as old as me. Odd, when space travel is so synonymous with high-technology, that much of the hardware is actually ancient.

Geek Culture

1. ‘Spooky’ Mulder was the agent in the X-Files, played by actor David Duchovny.
2. Kiki is the trainee witch in ‘Kiki’s Delivery Service’, one of my favourite anime movies by the outstanding Studio Ghibli.
3. The actual quote: “Humans are a disease, a cancer of this planet.” by Agent Smith. You can have a point for Virus or Cancer too. Thanks Chris for the link and clarification.
4. Spiderman of course!
5. “It’s a Banana” Kryten, of Red Dwarf, learns to lie.
6. My wife, who is Japanese, translates ‘Otaku’ as ‘geek’. Literally it means ‘you’ and is used to describe someone with obsessive interests. An appropriate question for a geek quiz I think.
7. The name R2D2  apparently came about when Lucas heard someone ask for Reel 2 Dialog Track 2 in the abbreviated form ‘R-2-D-2’. Later it was said to stand for Second Generation Robotic Droid Series 2, you can have a point for either.
8. Clarke’s 3rd law states: “Any sufficiently advanced technology is indistinguishable from magic.”
9. African or European? From Monty Python’s Holy Grail.
10. Open the pod bay doors please HAL. 2001 A Space Odyssey. Or on acid here.

So there you are. I hope you enjoyed it, and maybe even learnt a little. I certainly did. I might even do it again next year.

A very Merry Christmas to you all!

A Geek Christmas Quiz

God rest ye merry gentlemen! Welcome to my 2012 Geek Christmas Quiz. Every Friday morning at 15below we have a ‘DevEd’ session. Usually this is a presentation about some interesting tech, or a new way we want to do something at the company, but today I thought I would try to gauge the true geekiness of our development team with a quiz. The winners, and therefore crowned top geeks, were Toby and Linda who got a total of 32 points. See if you can do better dear reader.

You get one point for each correct answer. The quiz is split into six sections:  Computers, ‘Name That Geek’, Science, Space, ‘Name That Spaceship’, and Geek Culture.

Update: The answers are here.

Computers

1. What does G.N.U. stand for?
2. What did the A in ARM originally stand for?
3. What does TCP stand for?
4. Who founded Microsoft with Bill Gates?
5. What is F2 (hexadecimal) in decimal?
6. Which operating system's development was based on the 'Balmer Peak'?
7. Who was the first programmer?
8. What year does UNIX time start?
9. What did SGI stand for?
10. Write down the type signature of the Monadic Bind method.

Name that Geek

Science

1. What are the four letters of DNA?
2. What does the 'c' in E = mc2 stand for?
3. What is the next number in this sequence: 1 1 2 3 5 8 _ ?
4. What is C8 H10 N4 02 ?
5. When did Australopithecus become extinct? (in millions of years ago)
6. Which of the following would you not expect to find in an atomic nucleus (electron, neutron, proton)
7. What is the most common gas in the Earth's atmosphere?
8. Write the formula for Ohm's law.
9. If, after you fold a piece of paper in half, the ratio between its longest side and its shortest side is the same, what is that ratio?
10. What living land mammal is the closest evolutionary relative to Whales? (cetaceans)

Space

1. What rocket engine powered the 2nd stage of the Saturn V?
2. What is Saturn's largest moon?
3. What fraction of the Earth's gravity would you experience on the moon?
4. Astronauts are weightless in space because there is no gravity. true or false?
5. What is the orbital period of a geosynchronous satellite?
6. What is the furthest planet from the sun? (now that Pluto has been demoted)
7. How many people are currently living aboard the ISS?
8. To the nearest thousand, how many satellites are currently orbiting the earth?
9. What was the name of the only satellite launched by the UK?
10. Who was the second man on the moon?

Name that spaceship

Geek Culture

1. Who was Spooky Mulder?
2. Was Kiki a trainee witch or an evil princes?
3. "Humans are a _____" (Agent Smith)
4. Who is Peter Parker?
5. "It's a b... It's a b... It's a small, off-duty Czechoslovakian traffic warden!" What is it really?
6. What does 'Otaku' (Japanese) mean?
7. What does R2D2 stand for?
8. What is Clarke's 3rd law?
9. What is the air speed velocity of an unladen swallow?
10. Open ___ ___ ___ ____ please H.A.L

My Super Simple Node Twitter Re-Tweeter

I’ve been having a lot of fun writing a little ‘re-tweeter’ this morning. We basically want to monitor our user stream and then re-tweet any status with a particular hash tag. I thought this would be an excellent little project for node, and indeed it proved to be extremely easy to do. I used the node-twitter library which worked fine for what I wanted to do.

If you want to use this code, you’ll need to do the following:

First you’ll need to go to https://dev.twitter.com/apps and register a new app. You can then copy and paste your consumer key, consumer secret, access token and access token secret into the ‘xxx’ fields.

Next install node-twitter with npm:

npm install twitter

Then just run the code with node (I’m a poet and I didn’t know it):

node twitter-retweeter.js

Here’s the code in the twitter-retweeter.js file:

var util = require('util');
var twitter = require('twitter');

var twit = new twitter({
    consumer_key: 'xxx',
    consumer_secret: 'xxx',
    access_token_key: 'xxx',
    access_token_secret: 'xxx'
});

var hashtag = '#iloveprog'

function write(data) {
    if ( typeof data === 'string') {
        console.log(data);
    }
    else if (data.text && data.user && data.user.screen_name) {
        console.log(data.user.screen_name + ": " + data.text);
        testForHashtag(data);
    }
    else if (data.delete) {
        console.log('DELETE');
    }
    else if (data.message) {
        console.log('ERROR' + data.message);
    }
    else {
        console.log(util.inspect(data));
    }
}

function testForHashtag(data) {
    if(data.retweeted) return;
    if(data.text.indexOf(hashtag) != -1) {
        twit.retweetStatus(data.id_str, function(){
            console.log('retweet callback');
        });
    }
}

function reconnect() {
    setTimeout(startStreaming, 1000);
}

function startStreaming() {
    twit.stream('user', function(stream) {
        console.log('starting stream');
        stream.on('data', write);
        stream.on('end', reconnect)
    });
}

startStreaming();

console.log('lisening for tweets');

It’s all really straight forward. The startStreaming function kicks of the callback on the twitter user stream. Each time an event occurs it calls the write function which checks for the given hashtag and then retweets the status if there’s a match.

Lovely!