Apps
Contact
Seminars
RSS FeedDecember 8th, 2010
AdWords Program facing “technical difficulties”?
As I cover in the Organizational Efficiency seminar, we must value both the idea and its implementation. Google owes its success to at-the-time-innovative idea (pagerank and crowd sourcing of relevance etc) as much as it does to the engineering marvels of its later years when it was the only search engine that was able to keep up with the publishing speed of the world. At the end of the day (read 2002), it wasn’t its innovative algorithm, but the engineering that stops other starts ups from building search engines. The idea provided the spark, and the engineering/implementation provided the ongoing fuel.
So, how is it possible that Google is letting its prized possession (97% of revenue) go through having “technical difficulties”? This is the 3rd time this month (we are only 8 days into it) that as I try to navigate from one tab to another, AdWords just barfs and gives me this “down for maintenance” junk. (And by the way, don’t you think this fake “performing maintenance” at 10 AM EST, actually sounds worse than the truthful “technical difficulties” message?)
Anyhoo, technical difficulties are what they are. But I am a computer scientist damn it, and I am sure I can figure out what did I do to make AdWords suddenly go into the maintenance mode.
So, this is what I did:
I clicked on Billing, and tried to enter my credit card. That is it! I tried that one more time, and thankfully, Adwords was back from its maintenance and allowed me to enter the card information, but when I click “Save”, it gives this message: “Authorization failed: Invalid credit card details”. Hmm, something is not right. The card works at Micro Center bricks and mortar store. It works at microcenter.com. It works at Amazon.com. But, Google, wouldn’t take it. What could it be? This “help article” may have a clue: http://adwords.google.com/support/aw/bin/topic.py?hl=en&topic=29244. So, is it true that Google just doesn’t take credit cards? Seems a bit ludicrous, I am sure there is more to the story than meets the eye. In either case, it would be a major faux pas for Google in case it doesn’t really accept credit cards, but allows the users to try to enter them anyway.
As my good friend Bob would put it – there is something fishy in here. (Image courtesy – Bob’s friend cbhuster.)
November 5th, 2010
Applying Wilson Score to Ad Variations in Google AdWords
In Google AdWords (and other ads programs), you can create multiple ad variations, and then Google, in its infinite wisdom, will show the “better performing” one. You can also use this to conduct simple A/B experiments and compare different versions of landing pages, etc. However, every time you “tweak” an ad or a landing page, it is considered a new version, so its statistics get reset. If you create a new ad or landing page, of course, you also start with blank slate of statistics. The question then becomes: How can you use the low volume of impressions and clicks for various ad variations/landing pages to estimate their performance?
Let us take an example:
Ad 1: 1900 impressions, 8 clicks (0.42% CTR)
Ad 2: 285 impressions, 1 click (0.35% CTR)
Obviously, at this point, Ad 1 seems to be dominating Ad 2, and chances are Google will continue to show that ad more. However, comparing CTRs at low volumes is problematic due to statistical significance of the data involved (for example, one extra click on Ad 2 would propel its CTR to higher than that of Ad 1).
I don’t know how Google handles it, but one reasonable way to handle this would be by using Wilson score, which in this context can be defined as:
where:
- CTR’ is the Wilson score based CTR,
- B is the base CTR rate that we would assume about an ad without showing that ad even once (kind of the default rate),
- z^2 is a measure of how confident we want to be, [For 90% confidence, z^2 is approximately 2.6.] and
- N is the number of impressions of the ad.
Finding the base CTR rate
This aspect is a bit more interesting. If we consider all online ads, then perhaps B = 0.1 may be a reasonable number (in either case Google knows this number). However, the average click through rate is highly dependent on the category of ads. So it is much more practical to simply take the sum of clicks and divide by sum of impressions. (Although, do note a clear implication of doing this – the CTR’ of your ad with highest CTR will by definition be pushed down, and the CTR’ of your ad with lowest CTR will by definition be pushed up.)
Some Results
Using B=0.1 (10% CTR being the norm), and z^2 = 2.6, and using the Wilson score based CTR formula above, we obtain:
Ad 1: 1900 impressions, 8 clicks (0.42% CTR), CTR’ = 0.43%
Ad 2: 285 impressions, 1 click (0.35% CTR), CTR’ = 0.44%
So, in this particular case, it does appear that Ad 2 is actually the better performing ad than Ad 1, or at least to say that there is no statistically significant difference between performance of Ad 1 and Ad 2.
However, if we use B as the average value of CTR for these two ads, then the difference between CTR and CTR’ vanishes almost entirely.
Using B=0.004119 (total number of clicks / total impressions for these two ads), and z^2 = 2.6, and using the Wilson score based CTR formula above, we obtain:
Ad 1: 1900 impressions, 8 clicks (0.42% CTR), CTR’ = 0.42%
Ad 2: 285 impressions, 1 click (0.35% CTR), CTR’ = 0.35%
Further Observations
- If B value is used that is the average click through rate, then difference between CTR and CTR’ for most ads is very minor.
- Also, it is easy to see that after a few thousand impressions, CTR’ and CTR are virtually the same (as they should be). So, this calculation is of interest only in the beginning of an ad launch, or after an ad has been modified.
So, after all this, we reach the perhaps anti-climactic conclusion that applying Wilson score to ad variations in Google AdWords may be unnecessary, and CTR may be as good a measure as any of an ad performance. Null hypothesis holds.
November 4th, 2010
Google’s AdWords Seminars (for $299) are rather silly
Now this email from Google is just silly.
Join us in Washington D.C. for the following:
- AdWords 101 on Dec 6: Discover how to organize your AdWords campaigns, and craft keywords and ads that give you the most bang for your buck.
- AdWords 201 on Dec 7: Take your account to the next level. Learn how to turn clicks into customers by identifying keywords that really work, and by improving those that don’t.
Register now
1-day package: $299 and earn $50 in free AdWords advertising
2-day package: $549 and earn $100 in free AdWords advertising
So, you essentially have to pay $299 to Google so you can learn how to pay more money to Google later?
Especially that $50 in free AdWords advertising is just plainly stupid. (Google is forgetting the KISS principle.)
September 25th, 2010
User Interface, Google and Ben Adam
User Interface bugs exist everywhere, even on a minimal page such as Google’s sign in page. They continue to linger simply because they are not blocker errors. Pages work, your queries get processed, your reports get generated, your changes get saved – so no problem. So a “bug” isn’t even an appropriate word for UI issues. Rather, a “user inconvenience”, or a “usability snag” is a more appropriate word.
Consider Google’s sign in page. The page is largely empty. Even if you are a Ben Adam, and you have an intuitive email
address, as you type in the email address, you can’t see it all, since the text field only shows a total of 17 characters. Excluding “@gmail.com”, that leaves a total of 7 letters. My first name itself has 8 letters.
Knowing how quickly Google churns out new releases*, such a minor inconvenience could be fixed if it was a little bit more major.
* [As for Google's releases, are we at Google 3.0 now?. Let us see - Google 1.0 was the original page rank, Google 2.0 was the universal search, Google 3.0 is the instant universal search. Did I miss one?]
August 24th, 2010
Google’s universal search gives non-deterministic answers? (Perhaps due to MapReduce?)
One of the innovations at Google was the launch of the universal search a couple of years ago. While it was considered a drastic change by outsiders (something that can fundamentally change the user experience), the search giant was able to roll it out without much fuss, and pretty much all users are very familiar with it by now. You search for “Elvis” and you can see books about Elvis, blog posts, images, regular web pages, all interspersed using that magical ranking that made the search engine the king (no pun intended).
However, sometimes the universal search gives different results just a few second apart. Here, consider the first try for RYN:
Google Search RYN - Try 1
Now, let us try the same search again:
Google Search RYN - Try 2
So, sometimes the stock results are shown, and sometimes not. You can try this behavior yourself, by clicking on this search a few times: RYN. I can’t say how many times you might have to try it, but chances are, you will be able to replicate this behavior easily.
Now, the universal search likely uses the MapReduce paradigm (I am entering purely speculative mode here, so be forewarned.) Say the map function of a search term returns a list of search categories (which are then say farmed out to worker machines to process). Some worker machines may or may not return to the master in time, and in the reduce phase, the master may be only putting together the results that it received in time (and ordering it using the search results rank and such).
So, in case the worker does not return the results for the search term from the “finance” search category in time, the master is not able to include those results.
Also, by repeatedly trying out the search, I observe that that this non-deterministic behavior manifests mostly for finance and image search categories. One would imagine that Google has a check in place that if the “core webpage” search category worker has not returned, the results are not considered valid, and master must wait for that.
That is enough speculation for a day, so I guess I will just wait until someone with REAL knowledge of how Google’s universal search can enlighten everyone on this matter.
Oh, and btw, the original MapReduce paper does address non-determinism, but only as:
When the map and/or reduce operators are nondeterministic, we provide weaker but still reasonable semantics. In the presence of non-deterministic operators, the output of a particular reduce task R1 is equivalent to the output for R1 produced by a sequential execution of the non-deterministic program.
Clearly, this non-determinism is not directly related to the end users non-determinism that we refer to here.