At the Forge

eBay's Web services API allows programs to search through on-line auctions, but only if the programmer doesn't get too frustrated first.

During the past few months, we have looked at the Web services offered by two of the largest companies on the Internet, Amazon and Google. Each of these companies has enormous databases at the core of their businesses. By opening up some of that data to the public via Web services APIs, they have made it possible for outside developers to create new and interesting applications. No longer must we write “screen scraper” programs that parse the HTML produced by Amazon or Google. Now, we can write a program that requests precisely the data we want and receives it in exactly the format we need.

Another major on-line contender is eBay, and its database of on-line sales might well be the largest ever assembled. eBay began solely as a site for on-line auctions, but it has moved far beyond that in recent years—with a fixed-price subsidiary (Half.com), fixed-price sales on the main eBay site (Buy it now) and third-party “stores” where people sell a variety of goods for a fixed or variable price.

For several years, eBay has run a developer program for programmers interested in tapping into its database. However, until recently, this developer program required that developers pay in order to participate. From a business perspective, it might have initially seemed foolish for eBay to give away access to its sales database, particularly when the developer program clearly costs money to set up and maintain. Whether it was due to pressure from Amazon and Google, or from individual developers, or if eBay simply decided that it would benefit from additional publicity and outside developers, eBay dropped fees—making it possible for everyone to try this service.

This month, we look at several aspects of the eBay Web services API. The API is too rich and extensive to discuss fully, so we look at the functionality that I believe most people will be interested in using—namely, that which lets you search through existing eBay auctions for items that are of interest. By the end of this article, you should understand how the API works, how to write programs that use REST to search through eBay's database and how to use that information for personal and business needs.

The idea behind Web services is quite simple. Instead of treating an HTTP transaction as a request for an HTML document, why not think of it as a remote procedure call? The HTTP request then becomes a method for invoking a procedure on a remote server, with the URL indicating which method should be invoked and the HTTP response containing the result of the call. In nearly all cases, the response is an XML document, allowing for the invoked procedure to return a complex data structure.

There are at least three different styles for invoking a Web service, and eBay supports all of them. SOAP is perhaps the most sophisticated method, using XML in both the request and the response, but it is also the most complex and the most likely to run into cross-platform incompatibilities. This is partly because SOAP has tried to standardize all of the possible method calls, data types and scenarios that might be needed—leading to a somewhat bloated specification and many places where vendors disagree on how best to adhere to the specification.

eBay also supports invoking Web services with what it calls the XML API. Because SOAP also consists of XML, I find this terminology to be a bit confusing, but Amazon also describes things in this way. So, until someone creates a useful acronym or name, we're stuck with it. APIs based on XML are basically stripped-down versions of SOAP, without much of the overhead associated with it, such as namespaces and highly specified methods to marshal complex data structures. eBay says it is possible to use either XML or SOAP to access the full functionality of its Web services.

If I had to choose between SOAP and XML, I usually would use XML. But eBay provides a third interface, which is more limited than the SOAP and XML APIs, but far easier to work with. This third option is known as REST (short for representational state transfer), and anyone familiar with URLs should immediately understand how it works. The parameters are passed in the URL, using the standard name=value syntax. A REST invocation thus looks like http://www.example.com/method?param1=value1&param2=value2.

REST invocations are useful only for searching through eBay's catalogs. If you want to monitor sales, adjust your shopping basket, add listings to your store or even send messages to sellers and buyers, you must use the XML or SOAP API. The size of the API documentation says it all: eBay's REST documentation is 29 pages long, and documentation for the SOAP and XML APIs is more than 1,600 pages long in each case.

Because the application we are building is supposed to search only through existing offers, rather than add a new item for sale, we can get away with using the REST API. The REST API makes it easier to jump right in, and it provides all of the functionality with less programming overhead.

Before you can use eBay's Web services, you first must register. I'll now say something I've never had to say before in the history of writing this column: I cannot guarantee the directions I provide here will work. I had an extremely difficult time registering with eBay's developer system, despite spending hours trying to do so, and I worry that many readers of this column will face similar challenges.

The first confusing issue is the fact that eBay has several different computer systems, each with its own user database. The first (www.ebay.com) is the main, regular eBay system, on which you already have a user name and password if you ever have bought or sold something on eBay.

A second system (www.sandbox.ebay.com), known as the sandbox, is where eBay developers can test their applications without having to use up their monthly quota of requests (described in greater detail below) and without having to risk damaging a working on-line store. You can do anything you want inside of the sandbox, including create new users (to simulate interactions with those users), but the database is separate from the main eBay site.

Finally, there is the eBay developer site (developer.ebay.com), which allows access to the APIs, certification of applications and documentation. Access to this site requires a third user name and password.

I suggest that aspiring eBay developers register with all three of these sites—starting with the main eBay site, continuing with the developer site and ending with the sandbox. Technically speaking, you don't need to register with the sandbox if your applications will be used only on the production eBay system. However, I found enough places in which URLs mistakenly took me to the sandbox, rather than the developer site, so obtaining a sandbox login would be a prudent move. Was I sent to the sandbox because it's the same as the developer site? Because of a bug? Because of something that went wrong in my configuration? I wish I could say; I spent a great deal of time trying to figure it out and am simply trying to avoid pain for readers of this article.

Part of the confusion is that the sandbox looks and feels exactly like the regular eBay site. This is largely a good thing, except that it means the only way to distinguish between the sandbox and the usual eBay site is by looking at the URL. Even confirmation e-mail messages from the sandbox were identical to e-mailed notices from the production eBay site.

Once you have all three logins, you need to generate a set of production keys: a developer ID, an application ID and a certificate ID. These IDs uniquely identify you and your application, although the role of each key is not obvious to me. (The eBay documentation indicates that each application has its own key, but I could not figure out how to generate a new set of keys for a separate application.) Each developer may have only one set of such production keys. Although the term application ID implies that there would be a separate key for each application you create, this does not seem to be the case.

If you are going to use eBay's production system, then you need to certify your application. There are two levels of certification. One, known as self-certification, allows you to make up to 10,000 requests to eBay's servers per month. Self-certification, as its name implies, requires that you fill out a short Web-based form describing your application. Upon submitting the form to eBay's server, you are sent an e-mail indicating that your application has been self-certified. This e-mail message contains a link to a URL from which you can pick up your production keys, as well as a code that you must enter to retrieve those keys.

Using this confirmation code, you then return to the eBay developer site, where you enter it. This results in the generation of your three production keys: the devID, appID and certID (sometimes referred to in the documentation as AuthCert).

If you are planning to use XML or SOAP, this is the end of the certification process; your application will need to send these IDs in the HTTP request headers. But we are using REST, which is supposed to simplify things—and although our actual method invocations eventually will be simpler than the XML and SOAP alternatives, we have not quite finished our task if we want to use REST.

This is because REST parameters are passed in the URL, and it would seem that eBay has (rightly) decided that passing the devID, appID and certID parameters would be ugly and unnecessary. To use REST, it is necessary to create a REST token, which creates a new, encoded string based on the three production keys. To generate the REST key, go to the REST token site, at developer.ebay.com/tokentool. Indicate that you want to use the production environment, that you want a REST token, and then enter your three production keys.

Then, if you're like me, you'll get an error message. Try as I might, I couldn't get past an eBay login screen that was displayed each time I tried to generate the REST token. Needless to say, I was quite frustrated by this point, and I began to wonder how (and why) a multibillion-dollar company could make it so difficult for developers to use its API. (In contrast, I was up and running within about 30 minutes after deciding to use the Google, Bloglines and Amazon APIs. The difference couldn't be starker.)

I never really figured out what happened. Perhaps I wasn't logged in to eBay, although I thought I was logged in to all three sites (the main site, sandbox and developer site). It also could be that I was using Firefox, which is known to have problems with the registration. In the end, I used a different browser just so I could get the REST token. There were some messages on the eBay developer forums indicating that other Firefox users were having similar problems. It might have had to do with one of eBay's SSL certificates expiring several months earlier, although I doubt it. It seems to me that the login portion of eBay's site is in need of better quality control.

Once you have gotten through the registration nightmare, you can make queries. The REST API is well documented and is quite straightforward to use. First, let's look at a simple program that sees how many matches we can find to a particular text string. The program, shown in Listing 1, is written in Ruby and is similar to some of the Amazon- and Google-searching programs presented during the past few months.

#!/usr/bin/ruby

require 'net/http'
require 'rexml/document'

if ARGV.length == 0
puts "#{$0}: You must enter at least one argument."
  exit
end

output = ""

# Iterate through each of our arguments
ARGV.each do |query_string|

output << "Searching for: #{query_string}\n"

  # Put together an eBay parameter string
  ebay_params = {'CallName' => 'GetSearchResults',
    'RequestToken' => 'XXX',
    'RequestUserId' => 'YYY',
    'Schema' => 1,
    'ItemTypeFilter' => 3,
    'SearchInDescription' => 1,
    'StoreSearch' => 3,
    'DetailLevel' => 3,
    'Query' => query_string}.map {|key,value|
"#{key}=#{value}"}.join("&")

  # Ask eBay what it knows about our query_string
  ebay_response = Net::HTTP.get_response('rest.api.ebay.com',
                                           '/restapi?' << ebay_params)

  xml = REXML::Document.new(ebay_response.body)

  # Get basic information
  how_many_matches =
xml.root.elements["PaginationResult/TotalNumberOfEntries"].text

output << "Number of matches: #{how_many_matches}\n"

end

# Show everyone what we've learned
puts output

The program begins by retrieving our search parameters, automatically placed in the ARGV variable. We iterate over each element of ARGV, calling each individual argument query_string. We then use a hash to create an easily understood set of name-value pairs, in which the hash keys are the parameter names and the hash values are the parameter values. We then use a bit of Ruby magic to combine them, first turning them into pairs with map, and then using join to connect the pairs together with &. In the end, we have a string we can pass to eBay's server.

In this particular example, we're using the Query method in the REST API. Query allows us to enter a text string, for which eBay will then search. The way that eBay has grown somewhat organically over the years becomes apparent when you use its Web services. You must explicitly indicate if you want to search in stores as well as auctions. We also must indicate whether we want auction items, fixed-price items or both. Thus, our example searches through all stores (because StoreSearch = 3), auctions and fixed-price items (ItemTypeFilter = 3), in descriptions as well as item titles (SearchInDescription = 1), and with a fair amount of detail returned (DetailLevel = 3).

We also indicate we want Schema = 1. This tells eBay we want to receive a response using eBay's new XML schema, rather than the older one that is now being deprecated.

We then take ebay_params, a string created from our name-value pairs, and pass it to Net::HTTP.get_response. This sends an HTTP request to eBay's server (rest.api.ebay.com), using the appropriate path (/restapi), followed by our name-value pairs.

When we get a response—and our sample code here assumes that we do receive a response—we expect that it is formatted in XML and parse it using Ruby's built-in REXML library. We grab the total number of entries in eBay's database containing this search string and use the text method to extract the text from between the <TotalNumberOfEntries> tags. Finally, the program displays its output, showing us how many items on eBay contain this text string.

The API is relatively fast, allowing us to perform lookups for a particular string in relatively short time. That said, popular search strings can take far longer than rare words. A search for an ISBN took 1–2 seconds on my computer and indicated how many sellers were offering that ISBN for sale. A search for the term auction, not surprisingly, took more than 30 seconds to return a result and indicated that 29,458,603 sellers mentioned that term in the title or description. Obviously, the choice of search term, as well as the number of sellers and the quantity of text searched for that term, will have a significant effect on the performance of your application.

eBay's API makes it possible to perform Boolean searches of various types. Putting two words together within quotation marks (URL-encoded, of course) allows you to search for a phrase. You can search for two words in the same auction by linking them with commas.

You also can include and exclude particular sellers. If you are a seller on eBay, you might want to look at all of your items—or all of your competitors' items, ignoring yours. These functions make it easier to navigate through the complex world of eBay, which sells a staggering variety of goods from all over the world.

eBay's API, particularly for SOAP and XML, is rich and extensive. This is in addition to the simple, but limited, REST API that we used for the example in Listing 1. However, eBay's tagging of metadata, or information about each listing, is rather limited, especially when compared with Amazon. Perhaps this is because of the difference between the two sites. Amazon, as a vendor with inventory, knows and can pull up information about each item's dimensions, weight and ISBN. By contrast, eBay's only real information about each sold item is its categorization, asking price (and bidding information) and the text used to describe it.

There is a provision in the SOAP and XML APIs to look for items by ExternalProductID, which can be an ISBN or UPC. But when it comes to metadata describing each object, Amazon has beaten eBay hands down.

Amazon also is friendlier when it comes to registration and usage. Amazon makes it easy to register and easy to get started. Its forums are full of friendly people with useful advice. And, it sets relatively straightforward rules of use for its data.

eBay also differs from Amazon in how many queries it allows an application to make. Amazon doesn't restrict the number of queries, except by saying there should be no more than one per second, per IP address. eBay, by contrast, has a 10,000-query limit for each application. However, this limit can be raised substantially if you go through a more thorough certification process, giving eBay more information about your application, how it works and how you intend to use it.

The companies also differ in how many results they return. Each page from eBay contains up to 400 items, as opposed to Amazon's ten. In both cases, you can request subsequent “pages” of data, until you get information about all of the listings that matched your query. In this case, eBay's larger format is a significant improvement for people looking for popular items that might be available from many sellers.

Finally, eBay offers a dashboard showing which calls you have made and which were not compliant with its compatibility rules. This is an excellent feature—especially the part where it tracks how many queries succeeded and failed. I don't expect many of my REST queries to fail after I have debugged them, but it's possible that this could happen.

The bottom line is that I'm far more impressed with everything having to do with Amazon's Web services. eBay clearly is trying to improve things, with extensive documentation, developer forums and a help desk offering paid support. Nevertheless, it remains far inferior to what Amazon is offering. And, although they are not directly comparable, they also are inferior to Google's offerings in the Web services arena.

That said, eBay is a major player in the e-commerce world, and access to its data might well be worth the pain you encounter in using it. Plus, once you have gotten over the registration hurdles, you likely will be using only a handful of API calls, with minor tweaks and changes over time.

eBay's recent changes to its developer program are a welcome step forward. With three interfaces (SOAP, XML and REST) and an extensive set of methods available for developers to use, it's possible to glean all sorts of data from eBay's stores and auctions. Unfortunately, this all comes with a price; with less metadata and an unnecessarily confusing registration process, eBay's offering is far less impressive than it could be.

Resources for this article: /article/9066.