http://benchatelain.com/2009/03/05/scraping-app-store-rankings-around-the-world/
It’s been driving me crazy how slow iTunes can be when browsing through categories and switching between different countries to view the ranking for our Full Screen Web Browser in all the App Stores worldwide. I’ve had Erica Sadun’s article on Scraping AppStore Reviews in the back of my mind but never had the time to dig into it. Well, tonight I played around with this approach, using curl to pull back iTunes content for the top paid utility apps, which is accomplished with the following command line:
curl -s -A “iTunes/4.2 (Macintosh; U; PPC Mac OS X 10.2″ -H “X-Apple-Store-Front: $storeID-1″ ‘http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewTop?id=$categoryID&popId=$popId’
The $storeID variable is the iTunes store number for the country in question (143441 for the US). Just as you’d suspect, $categoryID is the category id your app is in while $popId is either 27 (free) or 30 (paid).
I wrapped this in a perl script which uses a regex to parse out the rank, app ID and app name from the XML that is returned. The list is filtered for the app ID in question and a loop iterates over all the App Stores around the world. Here is a sample of the output from this script:
Full Screen Web Browser Ranking in Category Top Overall:
New Zealand: 100
Finland: 44
Nederland: 77
Denmark: 67
Poland: 79
Ireland: 41
Malaysia: 80
Full Screen Web Browser Ranking in Category Utilities:
New Zealand: 9
Portugal: 38
France: 69
Czech Republic: 34
Lebanon: 67
Thailand: 86
United Arab Emirates: 63
China: 14
Schweitz/Suisse: 25
Singapore: 15
Slovenia: 15
United Kingdom: 20
Finland: 4
Saudi Arabia: 90
Brazil: 73
Nederland: 7
Chile: 27
Osterreich: 30
Turkey: 44
Mexico: 70
Hungary: 25
Croatia: 35
Denmark: 9
India: 26
Russia: 56
Sweden: 27
Italia: 44
Greece: 64
Deutschland: 29
Romania: 35
Espana: 89
United States: 21
Phillipines: 36
South Africa: 50
Canada: 23
Belgium: 67
Australia: 39
Poland: 10
Kuwait: 35
Korea: 24
Japan: 52
Argentina: 28
Taiwan: 14
Luxembourg: 23
Norway: 44
Ireland: 2
Malaysia: 8
Costa Rica: 16
Hong Kong: 39
Slovakia: 28
Israel: 24
To use this for your own app, you’ll want to edit the following variables at the top of the script:
- $appName - just used for display
- $appID - the numeric app ID from iTunes Connect
- $categoryName - the string name for your category from the %categories hash
- $popId - change to 27 if you have a free app (way down in fetchTop100InCategory sub)
I’ve taken the liberty of building the list of categories and IDs but haven’t tested any besides Utilities so the categoryIDs may not all be correct. If you’re curious how I got these, Control+Click on the arrow which takes to to the top paid/free apps section and select Copy iTunes Store URL. Note, that the link to the category from the main iTunes page uses different IDs. The IDs in this script are specifically for the viewTop servlet.
And without further ado, here is the complete script:
Update: Pastie is apparently down today. So here’s an inline copy of the script:
#! /usr/bin/perl
# Autofetch category rank
package itunesRank;
use warnings;
use strict;
# FSWB
my $appName = "Full Screen Web Browser";
my $appID = 303154925;
# my $categoryName = "Top Overall";
my $categoryName = "Utilities";
my %categories = (
"Top Overall" => 25204,
"Books" => 25470,
"Business" => 25148,
"Education" => 25156,
"Entertainment" => 25164,
"Finance" => 25172,
"Healthcare & Fitness" => 25188,
"Lifestyle" => 25196,
"Medical" => 26321,
"Music" => 25212,
"Navigation" => 25220,
"News" => 25228,
"Photography" => 25236,
"Productivity" => 25244,
"Reference" => 25252,
"Social Networking" => 25260,
"Sports" => 25268,
"Travel" => 25276,
"Utilities" => 25284,
"Weather" => 25292,
"All Games" => 25180,
"Games/Action" => 26341,
"Games/Adventure" => 26351,
"Games/Arcade" => 26361,
"Games/Board" => 26371,
"Games/Card" => 26381,
"Games/Casino" => 26341,
"Games/Dice" => 26341,
"Games/Educational" => 26411,
"Games/Family" => 26421,
"Games/Kids" => 26431,
"Games/Music" => 26441,
"Games/Puzzle" => 26451,
"Games/Racing" => 26461,
"Games/Role Playing"=> 26471,
"Games/Simulation" => 26481,
"Games/Sports" => 26491,
"Games/Strategy" => 26501,
"Games/Trivia" => 26511,
"Games/Word" => 26521,
);
# print "$appName Ranking in Category $categoryName:\n";
# getAppRankingInCategoryForUS($appID, $categories{$categoryName});
print "$appName Ranking in Category Top Overall:\n";
getAppRankingInCategoryForWorld($appID, $categories{"Top Overall"});
print "$appName Ranking in Category $categoryName:\n";
getAppRankingInCategoryForWorld($appID, $categories{$categoryName});
#
# Subroutines
#
sub getAppRankingInCategoryForUS {
my ($appID, $categoryID) = @_;
my ($country, $storeID);
$country = "United States";
$storeID = 143441;
# $categoryID = 25284;
fetchAppCategoryRankForCountry($appID, $categoryID, $country, $storeID);
}
sub getAppRankingInCategoryForWorld {
my ($appID, $categoryID) = @_;
my %appStore = (
143441 => "United States",
143505 => "Argentina",
143460 => "Australia",
143446 => "Belgium",
143503 => "Brazil",
143455 => "Canada",
143483 => "Chile",
143465 => "China",
143501 => "Colombia",
143495 => "Costa Rica",
143494 => "Croatia",
143489 => "Czech Republic",
143458 => "Denmark",
143443 => "Deutschland",
143506 => "El Salvador",
143454 => "Espana",
143447 => "Finland",
143442 => "France",
143448 => "Greece",
143504 => "Guatemala",
143463 => "Hong Kong",
143482 => "Hungary",
143467 => "India",
143476 => "Indonesia",
143449 => "Ireland",
143491 => "Israel",
143450 => "Italia",
143466 => "Korea",
143493 => "Kuwait",
143497 => "Lebanon",
143451 => "Luxembourg",
143473 => "Malaysia",
143468 => "Mexico",
143452 => "Nederland",
143461 => "New Zealand",
143457 => "Norway",
143445 => "Osterreich",
143477 => "Pakistan",
143485 => "Panama",
143507 => "Peru",
143474 => "Phillipines",
143478 => "Poland",
143453 => "Portugal",
143498 => "Qatar",
143487 => "Romania",
143469 => "Russia",
143479 => "Saudi Arabia",
143459 => "Schweitz/Suisse",
143464 => "Singapore",
143496 => "Slovakia",
143499 => "Slovenia",
143472 => "South Africa",
143486 => "Sri Lanka",
143456 => "Sweden",
143470 => "Taiwan",
143475 => "Thailand",
143480 => "Turkey",
143481 => "United Arab Emirates",
143444 => "United Kingdom",
143502 => "Venezuela",
143471 => "Vietnam",
143462 => "Japan"
);
while (my ($storeID, $country) = each(%appStore)) {
# print "$storeID\n";
# my $country = $appStores{$storeID};
fetchAppCategoryRankForCountry($appID, $categoryID, $country, $storeID);
}
}
sub fetchAppCategoryRankForCountry {
my ($myAppID, $categoryID, $country, $storeID) = @_;
my ($rank, $found);
my @top100= fetchTop100InCategory($storeID, $categoryID);
for (my $i = 0; $i < $#top100; $i += 3) {
$rank = $top100[$i];
my $appID = $top100[$i+1];
my $title = $top100[$i+2];
# print $rank . " " . $title . " " . $appID . "\n";
if ($appID == $myAppID) {
$found = 1;
# print $rank . " " . $title . "\n";
last;
}
}
if ($found) {
# $rank = "n/a";
print "$country: $rank\n";
}
}
sub fetchTop100InCategory {
my($storeID, $categoryID) = @_;
# 30 == paid
# 27 == free
my $popId = 30;
my $fetchcmd = qq{curl -s -A "iTunes/4.2 (Macintosh; U; PPC Mac OS X 10.2" -H "X-Apple-Store-Front: $storeID-1" 'http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewTop?id=$categoryID&popId=$popId' };
# Might need to pipe through these...
# | gunzip
# | xmllint --format -
my $doc = `$fetchcmd`;
# DEBUG: read from local file
# my $filename = '/Users/phatblat/tmp/itunes_utilties_top100.xml';
# open (FILE, "<", $filename) or die "Failed to read file $filename : $! \n";
# my $doc;
# {
# local $/;
# $doc = ;
# }
# close(FILE);
my @top100;
while ($doc =~ m#(\d+)\.\.*?viewSoftware\?id=(\d+).*?draggingName="([^"]+)"#gs) {
# 1 => rank
# 2 => app ID
# 3 => app name
# print $1 . " " . $3 . " " . $2 . "\n";
push(@top100, ($1, $2, $3));
}
return @top100;
}
Copyright © 1999-2017, swindler. All rights reserved.
367,611 visitor ( 1999.1.8-2004.5.26 ), 2,405,771 ( -2017.01.31)