RESTful URIs, Unicorns, and Pixie Dust

It often pains me to hear people talking about  so-called “RESTful URLs”. If you’re using that term, or your spending the majority of you application planning designing URI structures rather than your media types, then chances are you don’t really get the concepts in REST.

Frequently, I see developers sit down and start doodling a “REST API” by mapping out a bunch of URI templates like so:

/users/{userId}

/users/{userId}/stuff

/users/{userId}/stuff/{stuffId}

I’ll admit that I’ve been guilty of taking this approach myself. For one thing, it’s easy to communicate on paper. Most folks in business roles are used to seeing site maps where the content layout and URL structure are usually one in the same. By laying URI templates, you’ve kind of accomplished the same thing. Folks can visualize a high-level structure of your application, but you end up backing yourself into a corner that is difficult to get out of. Stu Charlton perhaps summed it up best in one of his more insightful posts:

If one is thinking of “how to methods map to my URI structures”, we’ve basically lost the plot as to why we have HTTP, URIs and media types in the first place. The goal is to *eliminate* the API as we normally think of it. (It’s not about a library of calls over a network).

The problem is that in approaching application design with the URI structure first is that you’re doing things bassackwards. Some people do this because they’ve followed some debatable advice and identified all of the “nouns” their application and started to work out a series of URL patterns that map to these nouns. As they create these URLs, they’ve followed some questionable advice as to what constitutes a “RESTful URL.” Subbu has another nice post dispelling some of those claims, so I won’t get into it here. The problem with doing all of the URI structure work up front is that you end up create a set of type resource URLs end up becoming fixed. Clients now end up coding to a specific set of URI patterns and/or conventions that are only discoverable from your documentation. The URI says nothing about what the data looks like or how the client should interpret it.

Imagine for a moment that you’re a DBA and you’re designing a set of database tables. Which of the two activities are you likely to spend more time on:

  1. The structure of your primary keys
  2. The schema of your database tables

A few of us would opt for something like MySQL’s auto-increment function and we’d be spending the majority of our time on describing the schema. In designing a RESTful application, you should be focusing on the design of your media types rather than what your URLs look like. To be more blunt: you must be focusing your efforts on what the hell your data looks like to consumers of your application. In addition to that, you need to think about how you are expressing links to other resources within your application or resources that are external to it.

This isn’t to say that URL design should be arbitrary and delegated to your web framework of choice. Of course not, you should still have URI strategy. The point is that the specific URI structures are not what consumers of your API will have to deal with directly. It is bad form to make a client rely on “out-of-band” information to construct a URI in your own special little way in order to get into your application. Take this blog post for example. You likely followed a link posted somewhere else. It could have been from an Atom feed, a search result in Google, or a Bit.ly link off of Twitter. It is highly unlikely that you had to type in the URL yourself and figured out the WordPress permalink structure that I have enabled on this site. If you did do that, well then, you’re awesome!

The fact is that clients will enter your application from some entry point or bookmark you’ve defined (.well-known is looking promising), or somewhere else on the web. Unless you’re a major player like Amazon, Facebook, or others, these clients won’t know that you have documentation that painfully detail your APIs URI structures. These clients will simply follow a link into your application. They didn’t type it in according to your fancy-schmancy URI template scheme that is only found in your documentation.  The URL is only a means to identify an locate a resource on the web, it does not define how the resource is represented or make suggestions as to what it’s about (remember that URIs are opaque?). At the end of the day, the client is going to have to be able to understand the media type that is retuned by requests made that URI. If you spend all of your time up front mapping methods to URI structures, you’ll end up introducing a coupling that you can’t easily break free from.

RESTful URIs, Unicorns, and Pixie Dust

URL vs. URI: URLs as Queries and URIs as Identities

Continuing my with my ranting about the URL vs. URI bit, I thought I’d continue on given my renewed interest in this topic thanks to Ora. In our LEDP position paper, we made the observation that URLs represent queries while URIs are identifiers. If you’re wondering why you should care about this subtle distinction, please read on.

URLs as Queries

We’ve stated that URLs are queries, but what does that really mean? Those of you familiar with blog software such as WordPress, know that the default URL pattern might go something like this:

http://www.damnhandy.com/?p=399

Here, the URL forms a query for a blog post using its internal identifier. In this case, the URL is asking the WordPress database for a post and related items using the primary key of a row that represents the post. For most, it’s pretty obvious that the query parameter “p” refers to the internal identity of the post.

As we mention in the paper, there are many other ways to construct URLs to the same post. For example, we can embed the ID into a path segment:

http://damnhandy.com/archives/399

In all of these cases, the server application is interpreting the URL and using elements of the URL to internally resolve the information the client requested. While I’ve only singled out WordPress, this pattern is quite common among several web application frameworks.

Internal Identity vs. Global Identity

When folks put information on the web, the content they publish usually has two identities:

  1. An Internal or local identity. This maybe the name of a file (i.e. “me.jpg”) or a the primary key of a row in a database.
  2. An external identity which is the ID of the information you’ve published. On the web, this is the “global identity” exposed by the URL of the content

Often, people don’t tend to think about either much. The global identity of a resource is usually an after thought and is determined by the underlying framework driving the application. With web servers serving up documents, we’re usually exposing the the local file name of the of the document. With database driven applictaions, we’re exposing the primary key, or some alternate key, of a row in a database. Quite often when web applictaion changes frameworks, we see the global identity change too and the URL patterns change (i.e. .NET to Java’s JSP, to Ruby, etc.).

Using the previous WordPress example, we know that the internal blog post ID is “399”, but this internal ID really isn’t suitable as a globally unique, unambiguous identifier. Another blog using WordPress, running the same exact version of the software, could also have a blog post ID of “399”. This does not means that the two sites have the same content, it only means that the two instances happen to have a post with an internal ID of “399”. As you might have noticed, the value “399” isn’t a suitable web-scale identifier. We need something else.

Globally Identifiers

Subbu Allamaraju had an interesting post a while back about Resource Identity and Cool URIs. Subbu asserts that there is a distintion between the identity of a resource and its location and his key point is spot on:

URIs uniquely identify resources but a URI used to fetch something is not always a good candidate to serve as a unique identifier in client applications.

And this is where I feel that the core confusion with URLs and URIs: identity vs. location. If we look at his initial example, he desribes the following:


  
      AZA12093
      
      ...
  
  
      ADK31242
      
      ...
  

In his example, the internal identity of each account is being expressed through a path segment in the href attribute of the link element. This approach is functional and is similar to that of the previous WordPress example.

The problem with this approach is that the ID values are only unique within the domain “bank.org”. There’s no reliable way to assert that two sites are referring to the same account if we have to rely on the value of a path segment or query parameter. As stated earlier, if we take WordPress as an exmaple again, blog post “399” might talk about the Kardashians, or something else. There’s no gurantee that the two URLs refer to the same information if they share the same internal identity. Most likely, they don’t.

We could make the link and the ID one in the same:


  
      http://bank.org/account/AZA12093
      
      ...
  
  
      http://bank.org/account/ADK31242
      
      ...
  

You might wonder what the hell is going on here since it looks pretty much the same as the first example. The difference is that we’re saying that the ID and the link are identical. That is, the identity of the account is the URI. That URI also happens to be a URL that can be dereferenced. This works, and it’s considered basic principle of Linked Data.

However, there are some problems with this approach too. As Subbu rightly points out, URIs are not always cool URIs. We are all aware that URIs do change at some point. If Bank.org is acquired by BiggerBank.com, what happens to the ID since we tied the ID to a host name that is likely to be retired soon?

One solution is to follow good web practices and maintain the bank.org domain and either redirect requests to the older URLs to the new ones. Adobe does this with links to FurtureSplash and Macromedia Flash locations. These URLs all resolve to the Adobe Flash product pages.

This strategy allows us to keep the original identity but the link is changed to accommodate the new domain. We can expand on this strategy and change the value of the link:


  
      http://bank.org/account/AZA12093
      
      ...
  
  
      http://bank.org/account/ADK312423
      
      ...
  

Yes, it looks ugly and weird, but it’s valid and it works. If you look closely, it’s not much different from the initial WordPress URL. The only difference is that we’ve replaced a numeric identifier for a URI. It’s a URI that references another URI, but it is valid. For some reason, people just don’t like URLs that look like this.DBPedia does this since they’re describing data on Wikipedia:

http://dbpedia.org/snorql/?describe=http%3A//en.wikipedia.org/wiki/BMW_7_Series_%2528E23%2529

The big difference with this approach is that it’s clear that the identifier is globally unique. There’s significantly less ambigutity about the ID: http://en.wikipedia.org/wiki/BMW_7_Series_(E23) than the ID: BMW_7_Series_(E23). Because no one else can mint valid URLs within the Wikipedia domain, you can have greater confidence that multiple applications are referring to the same thing. URIs as identifiers are globally unique.

The global identity of information resources shouldn’t change as frequently as it does. It drives my wife apeshit that all of her recipe bookmarks change everytime MarthaStewart.com updates their site. Part of the problem I believe is that most folks doing Information Architecture don’t take identity into consideration and that a fair number of web frameworks do very little to assist in quality URI/URL design. But this post is long enough, so I’ll save that for another post.

URL vs. URI: URLs as Queries and URIs as Identities

URL vs. URI vs. URN: The Confusion Continues

A year has passsed since my last post on URIs and URLs and it would seem that some of the concepts are still lost on some folks. With that said, I figured I’d throw up another post that I could try and address some of the questions raised in the comments of both posts.

URLs and URNs are both URIs

This is one point that can’t be stated enough. A URL is a URI and a URN is a URI, plain and simple. It’s really quite challenging to phrase it any other way. It’s like trying to explain that a Human is a Mammal but a Mammal is not always a Human.

Examples of URLs and URNs:

People have also suggested that these posts could have been more helpful if I had provided some examples that illustrate the difference between a URL and a URI. Based on the previous point, one should able conclude that a URL is a URI and therefore there’s no reasonable way to present an example that distingushes the two. However, we can provide examples that distingush a URL from a URN:

All of these Examples are URIs:
Examples of URLs: mailto:someone@example.com
http://www.damnhandy.com/
https://github.com/afs/TDB-BDB.git
file:///home/someuser/somefile.txt
Examples of URNs: urn:mpeg:mpeg7:schema:2001urn:isbn:0451450523
urn:sha1:YNCKHTQCWBTRNJIV4WNAE52SJUQCZO5C
urn:uuid:6e8bc430-9c3a-11d9-9669-0800200c9a66

Again, all of the examples above are all valid examples of URIs. You’ll note of course that all of the URNs are prefixed with “urn:”.
URIs are Opaque Identifiers

There’s a very informative page by Tim Berners-Lee that provides a lot of good deails on Uniform Resource Identifiers. One very important point is the notion of URI opacity, which states:

“The only thing you can use an identifier for is to refer to an object. When you are not dereferencing, you should not look at the contents of the URI string to gain other information.”

When you followed the link to this page, you didn’t have to do anything other than clicking it. Your browser only had to look at the URI scheme and the domain in order to resolve this particular document. Everything after “damnhandy.com” is defined to be opaque to the client. This point may seem orthogonal to the original post, but it’s a very important aspect of URIs. I bring this point up because the following question was asked in the comments:

Can we say that:

“http://www.domain.tld/somepath/file.php?mykey=somevalue”

is an URI

and that the “http://www.domain.tld/somepath/file.php” part is an URL?

No. No you cannot. Both are URLs, which are also URIs. The idea behind URI Opacity is that you should not look at the string to make any inferences as to what is at the other end. The presence of a query string does not distinguish a URL from a URI. Both strings are URIs and URLs.

Another commenter also asserted the following:

URI is whole address of a resource but resource extension is not mentioned. in URL we also mention the extension.
for example:

URI: www.abc/home
URL: www.abc/home.html

This is also incorrect. Being pedantic, neither is a syntactically correct URI or URL. But that aside, the presence of a file extention also does not distinguish a URI from a URL. Again, they are both valid URLs as well as URIs. Going back to URI opacity, you also cannot conclude that the two URLs reference the same resource or the same representation. The URI is an opaque string that identifies a resource. They’re still both URIs.

HTTP URIs Can Identify Non-Document Resources

This the one point that I think hits at the crux of everyone’s confusion on the matter. Things get all meta-meta when we start using URIs to identify things that are not documents on the web. It is perfectly acceptable to use to URIs to identify concepts, or non-document resources. If a URI happens to express scheme that suggests that it can be dereferenced, it is not a requirement that a document resource is actually dereferenced by that URI.

Most people’s expectation of HTTP URIs is that they it can always be dereferenced. There is a common assumption that an HTTP URI must point to a document. If it document resources is not available at that URI, that is you get a 404 error, then the URI is somehow bad. Just because the server could not locate document or a representation for the URI, does not mean that the HTTP URI is an invalid identifier.

The finer bits of this issue are summed up in httpRange-14 as well as this article by Xiaoshu Wang. The concepts around httpRange-14 are deep , but I think it’s these types of ideas that trips people up a lot. IMHO, it’s one of those concepts that gets muddled in the great internet telephone game that causes more confusion. For example, it seems that people are under the impression that if a URL does not express a file extension, then it represents a concept and therefore is a URI, but it just doesn’t work that way.

A URL is a URI and a URI can be used to represent anything. Furthermore, each URI is unique. This is why you cannot assume that http://www.abc/home is the same as http://www.abc/home.html just by looking at the URI. These are both distinct URIs that may or may not represent the same resource. Because URIs are opaque, you as the client should not be attempting to make any decisions about the URI.

URL vs. URI vs. URN: The Confusion Continues

URL vs. URI vs. URN, in More Concise Terms

Without a doubt, the URL vs. URI post is by the most visited page on this blog. Even so, there’s still a lot of confusion on the topic and so I thought I’d break it down in less words. The original post was slightly misleading in that I attempted to compare URI to URL, when in fact it should have defined the relationship between URI, URL, and URN. In this post, I hope to clear that in more concise terms. But first, here’s a pretty picture:

uri_class_diagram

Both URLs and URNs are specializations, or subclasses of URI. You can refer to both URLs and URNs as a URI. In applictaion terms: if your application only calls for a URI, you should be free to use either or.
Now, here’s where the big difference between URN and URL: a URL is location bound and dereferencable over the web. A URN is just a name and isn’t bound to a network location. However, BOTH are still valid URIs. Now, if the application requires a URI that is bound to a network location, you must use the specialization of URI called URL.

Remember that URI stands for Uniform Resource Identifier, which is used to identify some “thing”, or resource, on the web. Both URLs and URNs are specialization’s (or subclasses if you will), of URI. You’d be correct by referring to both a URL or URN as a URI. In applictaion terms: if your application only calls for an identifier, you should be free to use either a URL or a URN. For brevity, you can state that the application simply requires a URI and the use of a URL or URN would statisfy that requirement.

Now if your application needs a URI that dereferencable over the web, you should be aware of the difference between URN and URL. A URL is location bound and defines the mechanism as to how to retrieve the resource over the web. A URN is just a name and isn’t bound to a network location. For example, you may have a URN for a books ISBN number in the form of urn:isbn:0451450523. The URN is still a valid URI, but you cannot dereference it, it’s just a name used to provide identity. So to put in simpler terms:

  • A URI is used to define the identity of some thing on the on the web
  • Both URL and URN are URIs
  • A URN only defines a name, it provides no details about how to get the resource over a network.
  • A URL defines how to retrieve the resource over the web.
  • You can get a “thing” via a URL, you can’t get anything with a URN
  • Both URL and URN are URIs as the both identify a resource

There some other items that need clarification based on some comments I’ve received from the original post:

  • Elements of a URI such as query string, file extension, etc. have no bearing on whether or not a URL is a URI. If the URI is defines how to get a resource over the web, it’s a URL.
  • A URL is not limited to HTTP. There are many other protocol schemes that can be plugged into a URL.
  • If a URL defines a scheme other than HTTP, it does not magically become a URI. The URI defines how to get the resource, whether it be HTTP, FTP, SMB, etc., it’s still a URL. But because the URL identifies a resource, it’s a URI as well.

Yeah, I’ve probably repeated myself a few times, but I wanted to stress a few points.

There’s also been some confusion about when to use the term URI. As I stated in the original post explained above, it depends on what you’re doing. If everything your application does involves accessing data over the web, you’re most likely using URL exclusively. In that case, it wouldn’t be a bad thing to use the term URL. Now, if the application can use either a network location, or a name, then URI is the proper term. For example, XML namespaces are usually declared using a URI. The namespace may just be a name, or a URL that references a DTD or XML Schema. So if you’re using a URL for identity and retrieval, it’s probably best to use URI.

URL vs. URI vs. URN, in More Concise Terms

URI vs. URL: What’s the Difference?

Updated (1/18/2011) : Because there’s still a lot of confusion, I’ve created a third post that attempts to resolve a lot of the questions from the comments on the last two posts. The new post is here.

Updated (8/27/2009): I’ve created another post that attempts to make the distinctions a bit more clear. The new post is here.

What is the difference between a URL and URI and why does it matter? This topic is confusing to some (myself included) and I thought I’d share my understanding of the two concepts. I’m hoping this post will give you a better understanding about how the two differ and why it matters to some.

Note: The goal of this post is to simplify the distinction between URI and URI. If you feel that in the summarization process something was lost, or it’s simply just correct, please post a comment and the information will be corrected. I only ask for any comments/criticism to be constructive.

Update: Thanks some constructive, and not-so constructive, feedback from some readers I have updated this post to correct many of my own misunderstandings. Of which, there were many.

URI

A URI identifies a resource either by location, or a name, or both. More often than not, most of us use URIs that defines a location to a resource. The fact that a URI can identify a resources by both name and location has lead to a lot of the confusion in my opionion. A URI has two specializations known as URL and URN.

URN

A URI identifies a resource by name in a given namespace but not define how the resource maybe obtained. This type of URI is called a URN. You may see URNs used in XML Schema documents to define a namespace, usually using a syntax such as:

<xsd:schema xmlns="http://www.w3.org/2001/XMLSchema"
            xmlns:xsd="http://www.w3.org/2001/XMLSchema"
            targetNamespace="urn:example"

Here the targetNamespace use a URN. It defines an identifier to the namespace, but it does not define a location.

URL

A URL is a specialization of URI that defines the network location of a specific resource. Unlike a URN, the URL defines how the resource can be obtained. We use URLs every day in the form of http://damnhandy.com, etc. But a URL doesn’t have to be an HTTP URL, it can be ftp://damnhandy.com, smb://damnhandy.com, etc.

The Difference Between Them

So what is the difference between URI and URL? It’s not as clear cut as I would like, but here’s my stab at it:

A URI is an identifier for some resource, but a URL gives you specific information as to obtain that resource. A URI is a URL and as one commenter pointed out, it is now considered incorrect to use URL when describing applications. Generally, if the URL describes both the location and name of a resource, the term to use is URI. Since this is generally the case most of us encounter everyday, URI is the correct term.

URI vs. URL: What’s the Difference?