Alex Schultz: APIs

del.icio.us API working PHP code for Https v1

So I was very frustrated when all of a sudden my del.icio.us api tag cloud stopped working, since I think it is a super swank piece of kit. I managed today to find out why it stopped working... del.icio.us have a highly secret blog which I (being a total muppet) didn't know about. On this blog they warned everyone their del.icio.us app would break if they didn't wake up and smell the SSL. So today I put together an SSL application for V1 of the del.icio.us https api using PHP and I wanted to share it here in case anyone else ended up having the same issues as me (maybe the friendly natural search monkeys will spider it for you to find).

So calling this API is hyper simple. First you need to define your user name and password:

$dusername = "foo";
$dpassword = "bar";

Then you need to create your URL to call (the example here is for the get tags call which I use in my del.icio.us tag cloud ):

$api = "api.del.icio.us/v1"

$apicall = "https://$dusername:$dpassword@$api/tags/get"

Then you need to throw together your CURL call to the API which I find is needed to connect via CSS and also send the user agent that del.icio.us requires from you

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$apicall);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_USERAGENT, 'alexschultz');
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
$xml = curl_exec ($ch);
curl_close ($ch);

Now you should find that your API content is contained in the $xml variable and you can enjoy translating it via the XSL tool of your choice (mine is sablotron because I am a masochist :p )

Paying for community created content

Postbubble has an awesome article today about digg, netscape, social news and remuneration for community drive content. I think this is a really interesting area since there is inherent value in content created by the community but there is also value in giving people a soap box. In the UK speaker's corner is a really popular location (wierdly in my opinion). It is a location where people stand and speak their view on anything and everything and thousands of tourists come to watch. The speakers draw an audience which they enjoy and hyde park, london transport and local shops benefit from the dollars this audience brings. Everyone is happy and no one pays for the content... it's great.

At the moment digg, amazon reviews, wikipedia, the venerable dmoz and even webmasterworld run on a similar basis to speaker's corner... I like the sound of my own voice, I love having an audience hence I contribute. Even my own cocktail recipes site runs on this basis. There are however a new breed of sites out there that are starting to pay people to create content and this is a really interesting development since all else being equal I'd prefer to be paid for hearing my voice heard. Some examples are:

Google Video: Last night I was chatting with one of the PMs at Google on this project at a party, I am now thinking of switching my videos from YouTube to Google, they get >100k impressions a month (generated from my site) and at Google video there is a real chance I will get paid for those eyeballs.

forums.digitalpoint.com: Shawn Hogan's site has a revenue sharing forum... if you start a thread you will get a share of the revenue Shawn makes from people looking at that thread (plus you can link off the site and a few other points). Digital point has come from nowhere to be a significant enough player to challenge Brett Tabke's webmasterworld.com

Netscape Navigators and About.com guides: Digg is under threat from both these models in my opinion. Just a few people kick off most of the stories on Digg and now Netscape is paying a few people to do the same for them. If About.com get their arses in gear about web2.0 they have a long standing and highly profitable history of paying for content, arguably about.com are the most experienced company on the web today in terms of paying for content. So keep on innovating digg old buddy and find a way to keep your diggers happy or they might switch.

The old addage goes there is no better price than free and from the point of view of digg, youtube, flickr and so on that is still true but as a contributor, you know what? I want to get paid.

Cocktailmaking.co.uk v3.0 Launched

w00t it is finally out of the door, for almost 3months now I have been playing around with how to launch a strong upgrade to my existing cocktail making site. I have been told at times that the design is grade school, it doesn't work in firefox, it isn't very intuitive, it's ugly and so on and so forth. All the while it's been unattractive visits have climbed and I topped 1/4million page views last month but I agree that users have had an unpleasant site to visit.

v1.0 launched in 2001 and simply consisted of a few hundred hard coded cocktails I had dug up from all over the place. It was ok but wasn't scalable for me to add cocktails and was rapidly falling behind sites like webtender and barnone. In 2003 therefore I upgraded the site to be fully user generated with approvals by me and this was release v2.0 which I was really proud of. More and more visitors came to the site and I found I had given too much freedom in the database for users to add all kinds of rubbish... this I still haven't fixed (although I cleaned up the database a few times this year) but I have plans of how to do it.

For this release, I have totally changed the appearance of the site, the cocktail list is pulled in a new way with far fewer lines of code and most of the key pages are now cached and not pulled live from the database to make them more scalable. The site also works again in firefox which is a good idea but the key thing is the code is now clean, self contained and fully rewritten. I have an architecture and platform I can finally expand and build on with new modules, this is what v3.0 means and over the next 3months I am going to start adding some really exciting user functionality including membership of the site, site emails, the ability to store your bar and more... this is gonna be an amazingly exciting phase and I can't wait to get my teeth into it.

Now off to bed, it's been a really long hard day of coding and as with any new launch bits of the site fell over and died and I've had to get them back up and running with fast programming (I refused to roll back). Now that is done I am tired... night night!

mySQL Left Outer Join

So I don't want to disappoint my friend who describes this site as everything geek... Earlier this week I was trying to work out an efficient way to find which cocktails in my cocktail making database had a certain set of ingredients and no others. i.e. tell me what you have in your bar and I will tell you what drinks can be made with a subset of those ingredients.

The solution to pulling this data turned out to be what is called a Left Outer Join and is supported in mySQL 5.0 (and possibly below this but I don't have a lower version to check).

The best way to describe what a left outer join does is an example so let's name two tables: age and beauty (the names of the trial eights in my college boatclub when I was captain).

Age Name Sex Brad Pitt Male Mick Jagger Male Jen Anniston Female Mo Mowlam Female

Beauty Name Sex Brad Pitt Male Josh Hartnett Male Jen Anniston Female Shakira Female

I may well want to find those people who are beautiful but not old and if would be great to have an effective way to do this without doing some complex IS NOT IN statement.

So what I could do is as follows:

SELECT Beauty.Name AS Name_b, Beauty.Sex AS Sex_b, Age.Name AS Name_a, Age.Sex AS Sex_a FROM Beauty LEFT OUTER JOIN Age ON Beauty.Name = Age.Name

Which would give me the following result:

Name_b	Sex_b	Name_a	Sex_a
Brad Pitt	Male	Brad Pitt	Male
Josh Hartnett	Male	Null	Null
Jen Anniston	Female	Jen Anniston	Female
Shakira	Female	Null	Null

However I would like to just see young beautiful people and so I want to get rid of Brad and Jen (too old). In order to do that I simply select those rows where the Name_a variable is null.

SELECT Beauty.Name AS Name_b, Beauty.Sex AS Sex_b, Age.Name AS Name_a, Age.Sex AS Sex_a FROM Beauty LEFT OUTER JOIN Age ON Beauty.Name = Age.Name WHERE Name_a IS NULL

Job now done the output is as follows and I have gone from a list of old people and a list of beautiful people of all ages to a list which just contains young beautiful people... now off to Google!

Name_b	Sex_b	Name_a	Sex_a
Josh Hartnett	Male	Null	Null
Shakira	Female	Null	Null

Moving With Your PC to the US

Moving to the US was a big pain for me in one particular regard... the frequency and voltage of US electricity was not what my computer needed and so I couldn't get my old UK computer working in the US. You can imagine as a web and API geek living without my PC was a painful experience until I bought a new Dell.

I then lived without my computer and in particular my hard drive for 8months but earlier this week a colleague suggested to me that you can get an IDE to USB connector for your old harddrive. I searched for this on eBay and then found one of the Yahoo! adverts on an eBay search results page which I clicked on, found the item I wanted and went back to eBay (finding the item cheaper on the site even including postage). I finally got the connection today which was an ion external harddrive casing and IDE to USB 2.0 connection.

It is up and running and quietly humming away on my desk... I now have my university music collection (2000 songs) and all my university photos again which has made me a very very happy boy this evening! So if you move countries and find your computer isn't working (even with a power adapter) don't fear, you can always recover the harddrive content using a IDE to USB 2.0 external hardrive connection (potentially from ion).

Releasing an API can make you more money

So facebook have released an API earlier this week (surprise surprise I'd already signed up within 6hrs of launch, more to come), I just can't see the revenue upside for them in this esp. given their generous call limits. This confusing facebook API is a neat segway into the flipside of the post I made earlier this week saying natural search APIs are poorly supported due to the lack of revenue upside. For me the power of a lot of sites is in the platform they offer and the fact that this platform is their revenue base. APIs that enable use of a platform to make the API owner more money are the best supported tools out there. Let's look at 2 examples: the eBay API, and the typepad API.

Starting with the first last I am writing this post through the typepad api using the great windows live blogging tool and you know what? Typepad simply don't care since I pay them for using their platform. Switching costs are pretty low for me on blogs. I have written my own blog software based off a database which I still own and works pretty well, I can use wordpress or blogger too and Typepad allows me to download all my listings in a datafile. My domain name is actually hosted by namesco (who I love but that's another story) and just DNS mapped to this namespace (easy to change), basically apart from a few small technical hurdles I could move my blog over night. I don't however because the typepad system is great and they keep on making it easier for me to use them with the blogging tool as an extension of their platform through the API being a key example. Offering out their API is helping to keep me a loyal customer by allowing anyone to develop for me the tools I need/want and they must know this since the API is free.

eBay's API is awesome. I love it and I keep saying it (much to the embarrassment of my colleagues). Without the eBay API such companies as channeladvisor and marketworks probably wouldn't exist and the new startup I read about today certainly would not. The beauty of this API is simple, eBay makes money wherever and however users buy and list so long as they use the eBay platform to do it. There are more and more examples of how eBay is cropping up all over the web wherever people might partake in e-commerce. This API strengthens the core platform and hence is totally free to use up to 1.5million calls a day if you certify (which is also free).

I am a passionate fan of forward thinking corporations/companies who use API's to their advantage. I think it makes the most sense when the companies have a really useful platform and they make revenue from that platform detached from their pure website. In summary if you want to make money from an API for your really useful website make sure it's users interacting with the platform that's valuable for you and not visitors to your website.

I really wanted to shout about the Adwords and AdSense API's here today and Yahoo's upcoming Panama but there just wasn't enough room. Another post coming soon.

Great Analytics are Crucial to Website Success

Google Analytics just received (on the day they opened up their tool to everyone) a round drubbing from one of my favorite blogs seomoz. The drubbing was somewhat fair and very true of all pixel/image based tracking tools. The pixel doesn't load everytime, cookies used are deleted/read, sessions are lost and on very big sites they often have to sample which is just irritating to me.

I agree that having analytics that aren't perfect is annoying. I look at my server logs as well as my google analytics and addfreestats accounts (I used to use Hitbox when I was first on the web and they were years ahead of their time) and despite the fact that pixel tracker based tools do have some inaccuracies I will happily stand up and say that all these stats tools are similarly informative to me on user behaviour as log file analysis if not more so because of their beautiful graphical representation of what is going on.

For me data is a means to an end... for a deep dive into user behaviour you should study server logs, perhaps build your own deeply insightful inhouse tool which ties activity to user and even follows their general patterns allowing you to change content to closely suit that user. For big trends affecting your site, the major entry routes, the massive exit routes, the most popular content and pointless content that no one visits and should be culled I believe the free analytics software offered by google and addfreestats should be a useful port of call.

Ripe for Regulation

I spent a little time thinking about the whole aol research fiasco where they released search records which although the user names were removed and replaced with unique number ids could be used to identify people from their search logs. A friend said to me he was really hoping that this would not cause the internet to become overly regulated and stop companies like AOL/Google/Yahoo/MSN from saving user related data since analysing and improving services based on this is a key part of making better, more relevant services.

I certainly agree with him about the fears of regulation strangling research and the benefits of data to innovation. Strangling research is always a valid fear with increased regulation. I however am hoping quite the reverse to my friend. I hope this fiasco does bring tighter regulation of the search engines and those with user data. To explain why I want to draw a few parallels to the real world and internet.

In the media in Germany any flattery of the Nazi party is banned, sale of Nazi memorabilia is banned (inc. online), Mein Kampf cannot be bought in a shop and using the Nazi salute will wind you up in jail. On Google.de you could however search for "why hitler was great" and see tonnes of neo-nazi results from outside DE turn up. Incitement to racial hatred is an important law in the UK and any newspaper or political group suspected of doing so can face harsh fines and jail sentences for anyone directly responsible. Again a search for any racial hatred topic online will turn up tonnes of related material.

Search engines are portals to this content. Without the oxygen of publicity movements often die but in the age of the internet anyone can type in their personal prejudice or hatred and quickly find sites with information about it and communities of people with similar mindsets. These sites are then further fuelled by getting visitors and those visitors can build out a community around the site. Television, radio and even newspapers are regulated to control access to this sort of information and the publicity that fuels and fans hatred. Perhaps search engines should too. Often the bulk of those with thuggish hatred are not well educated and I suspect (based on typical web-browsing behaviour) placing even placing simple obstacles to finding the data they need in their way (like hiding it in search engines) will stop them in their tracks.

I understand esp. in the web community where the pace of development is so fast right now and the opportunity to make millions still so huge that the idea of regulation isn't popular. A number of the searches I saw on the AOL logs convinced me it is and should be coming. Questions about murdering your wife and people searching for perverted pictures of children closely followed by looking for local schools suggest to me that AOL/Google/Yahoo/MSN have a responsibility to understand better what they are making available and to who and the government should stop worrying about whether to allow people to make more money from their "tubes" and more about what's passing through them.

Why do search companies offer these Map APIs?

Now maps is a really interesting area for looking at the offerings of the large web companies, i.e. Google, Yahoo and MSN (not eBay who don't offer anything in this area but do offer APIs which produce cool maps mashups with eBay listings).

I truly don't understand the strategy here from these big internet companies. They offer out their maps API for free and get thousands of users developing and adding maps to their site showing high traffic roads, apartments for rent, used cars for sale, real estate for sale and a whole bunch more. These resources developed by developers are fab for me as a user but I simply don't understand the value they offer for the search giants. Are they intending to monetize with adverts in future, do they just want to stay friendly with geeks, who knows?

I do have one suggestion for why it's good that the search engines are offering these APIs out, perhaps they are trying to capture the future of search. Offering out map's APIs for free gives them a huge R&D network. Let's say they have 1000 developers each using their maps (a gross understatement but hey ho) and each of those spends a 3 weeks of late nights working on their cool mashup (that would comfortably add up to 40hrs a week for me as a geek who loves this stuff :) ) that gives them 120 000 hours of free developer time from people near the cutting edge and not constrained by corporate concerns. Even at just a $10 an hour wage for engineers at their company that would be $1.2million of free R&D without factoring in all the savings by not actually employing someone (a really expensive task). The numbers I have given above are all low balls, I could comfortably be 2orders of magnitude low in that $1.2million!

Certainly geo-location and local search are important and will be of bigger and bigger importance as time goes on.

1. Being seen as the provider of choice thanks to all the geeks using and exposing your product
2. having 1000 applications in test to see which ones take off and you can copy, improve upon and release with huge traffic driven from your user base is actually quite a good idea.

Heaven forfend but potentially these big search companies aren't just having a random altruistic brain fart and pointless API arms race but are being very canny about capturing a future search channel.

In my next post in this series (there may be other posts inbetween) I am going to chat about paid/free user Platform APIs. Specifically eBay, del.icio.us and Flickr but potentially more.

Microsoft Windows Live Writer Download

Great job from microsoft today in providing a simply awesome bit of desktop kit. I love being able to work on the web without going through numerous layers of authentication and the usual vagueries of a browser. With the new windows live writer I can post to this blog from my desktop! Great job microsoft keep on creating good work like this. Download it now.

A final point is that the live homepage and widgets are great, the live search is looking actually very competitive in terms of user experience and quality listings. I think Microsoft are seriously starting to get their act together. The next 5yrs is gonna be fun and on a totally personal note (since I have mates working in classifieds) I am going to be really interested to see the classifieds offering when it comes out!