A few weeks back, I attended my first RESTFest “unconfernece””. This was a really great event put on by some fantastic folks. The “everyone talks” aspect of RESTFest is actually awesome idea for a conference this size. This format is great because if you’re like me, you had some things you wanted to get some feedback on and this provides the forum to do just that. Everyone gets a chance to do a quick 5-in-5 talk about anything. I was looking to get some feedback on some ideas I’ve been considering about push notifications in Web APIs.

I titled my talk “Real-Time Web APIs,” but I’m no so much interested in Web APIs that follow “real-time computing” constrains. Really, I’m looking to see if there’s a good way to describe a service that streams push notifications from a Web API that follows “RESTful” architectural constraints. There’s a lot of good ideas out there, but many of them either assume that the client either a web browser or running an HTTP server. Additionally, the mechanism to describe these services need some work.

The Goal

The core building blocks for push notifications are already available, but not so much the means that aid in discovery and the description of such resources. I’d like to be able to organize these pieces so that the following requirements can be met:

  1. Don’t assume that the subscriber is able to receive a callback using HTTP POST. If the subscriber is a web browser or a thick desktop client like a Swing or JavaFX application (yes, people still make these!), or even a nativ app on iOS, running a web server to receive and HTTP callback isn’t always practical or feasible.
  2. Advertise that a given link is exposing a stream of events via a link relation. This might be similar to the monitor link relation, but not necessarily bound to SIP.
  3. The link relation should be able to indicate the media type that event messages are described in, ideally via the type property.
  4. If there’s a sub protocol involved, it should also advertise what sub-protocol. If event stream is a media type that supports embedded content, it should also express that as well.
  5. Subscribing to a stream or feed should be simple

WebSockets and HTM5 Server-Sent Events (SSE) present some interesting opportunities for Web APIs that demand low-latency push notifications while also removing the need for a consumer to run an HTTP server. Keep in mind that I’m not talking about doing something like “REST over WebSockets (Shay does make some great points though!),” I’m simply looking for push notifications without the need for for the consumer to run a web server as well as describing the stream via link relations. I think it could be done, but there’s some missing bits.

PubSubHubub

PubSubHubub ticks a lot of check boxes for me, such as:

  • While the protocol is built around Atom and Atom concepts, it could support a variety of media types. It’s using the Content-Type header to express what is coming over the wire.
  • It sports a discovery model using the rel="hub" link relation either in a link header or a link within an Atom feed.
  • The subscriber subscribes to the Topic URL from the Topic URL’s declared Hub(s) using the PubSubHubbub subscription protocol.
  • Publishers ping the “Hub” to notify it of updates, aggregates the content, and sends it to the subscriber using an HTTP POST request to the call hub.callback URL.

What I like about it a lot is the use of hypermedia to do discovery and call backs. Additionally, it’s using standards means to describe what’s coming over the wire. The rel="hub" link relation combined with the script ion protocol is super easy an fit works. My challenge with PubSubHubub is the requirement of an HTTP callback on the part of the subscriber. As stated earlier, this isn’t always possible.

From my perspective, PubSubHubub has the right foundational model. It’s simply HTTP callbacks that are the sticking point. So can we do something similar with WebSockets or Server-Sent Events? I think so, but there’s some challenges with existing formats in order to make this work.

Server-Sent Events

I REALLY like HTML5 Server-Sent Events (SSE). A lot. I’m also really annoyed by the fact that Internet Explorer still isn’t supporting SSE in IE11. What I like about SSE is that it’s a media type (text/event-stream) so the subscriber will know that this URL represents an event stream. The event stream model is also dead simple:

event: change
data: 73857293

But it could be also like this:

event: change
data: {"@id" : "15628", "@type" : "ChangeEvent",  "value" : "73857293" }

Or even this:

event: change
data: {
data: "@id" : "15628", 
data: "@type" : "ChangeEvent",  
data: "value" : "73857293" 
data: }

Or like this (If you’re into this sort of thing):

event: change
data: <ChangeEvent><id>15628</id><value>73857293</value></ChangeEvent>

As you can see, the data field could contain nested media type like XML, JSON, or something else. The problem is that that there is no way to indicate that. How does one know that the data field contains structured content such as JSON? The browser gets around the issue by embedding JavaScript in an HTML document that references the stream:

var source = new EventSource(’/updates');
source.addEventListener('add',  changeHandler, false);

Obviously, the changeHandler function will parse and handle the embedded content. This works great where the subscriber is a web browser (or embedded browser), but for other environments it’s not so easy.

We could express this via a link, but it’d be missing some details. Let’s assume we have a link relation called stream that informs a client that this link represents a stream of events:

Link: <http://example.com/events>; rel="stream"; 
      type="text/event-stream";

It works and declares that the link is a stream of event and it exposes the events via SSE. But the subscriber has no hints that the data field contains JSON or XML content. In cases where the browser, or an embedded web browser, are not available, how does can a client get more information as to process a stream? For SSE, one option might be to include a media type parameter, call it data if you will, that would indicate that the nested type is something like JSON-LD:

Link: <http://example.com/events>; rel="stream"; 
      type="text/event-stream;data='application/ld+json'"

It’s just an idea, but it could be workable. I would love feedback on this and would REALLY like to see Microsoft add Server-Sent Events to IE at some point.

WebSockets

WebSockets are neat, but the majority of use cases for streaming notifications only really needs to go one way. The bi-directional nature of the WebSockets protocol is a nice to have but not entirely necessary for most applications. WebSockets by itself really isn’t that useful. A number of WebSocket examples you’ll see are effectively someone’s home-grown, JSON-based, socket protocol. It’s a bit too cowboy for my tastes, but it can get the job done.

Where I do find WebSockets more useful is being able to leverage a well-defined subprotocol over a WebSocket. At the moment, I’m quite of fond of STOMP, particularly STOMP over WebSockets. In a Java shop that is already heavily invested in JMS, STOMP over WebSockets is a reasonable leap given that tools such as ActiveMQ, RabbitMQ, and others are support STOMP over WebSockets now.

Building on the Server-Sent Event examples earlier, we have some similar problems such as:

  • We still don’t have a good way to indicating what might be coming over the wire
  • We have a new problem since we’re dealing with another protocol that supports subprotocols, we don’t have a means to identify the sub-protocol that the WebSocket will be using

At RESTFest, I crapped up a variation of link relation like so:

Link: <wss://example.com/events>;rel=”stream/v12.stomp";
       type=“application/ld+json";

Here, we’d overload the rel field to indicate that it’s a stream but that it’s also using STOMP as the sub-protocol, specifically STOMP v1.2 (note I’m using IANA WebSocket subprotocol IDs here). Because the URI begins with wss://, we know that we’re using WebSockets over SSL. The type property is indicating that the messages will be using application/ld+json. The problem with both approaches is that if I want to offer another alternate message formats (say JSON-LD or XML), then this solution does really work. But maybe that’s not a problem.

Constrained Application Protocol

One of the great things about attending a workshop like RESTFest is that you’re surrounded by people who are smarter or more experienced than you. After my 5-in-5, Mike Amudsen had a few good questions about what I was trying to do. He then asked if I had considered CoAP, or the Constrained Application Protocol. Having never heard of CoAP, I obviously hadn’t taken it into consideration. CoAP more than likely satisfies a number of my needs. Since it’s still in draft form, it’s not an easy sell yet. Without a doubt, CoAP is something to keep an eye one.

Wrapping Up

Right now, I’m going down the STOMP over WebSockets route. I’d REALLY prefer Server-Sent Events, but the fact that Microsoft isn’t supporting SSE in IE10 and IE11 is AND the corporate standard in most shops, it sadly makes SSE a non-starter. In the coming weeks, I’ll be slapping some code up on GitHub to test out some ideas. I’d love to get feedback on these ideas to see if I’m going off the rails or if these ideas have some merit.

After a months of not blogging and kind of working on my URI Template library, I finally managed to get out version 2.0.1. The new API makes a quite a bit of changes and it does break some things in version 1.x. Here’s a run down of what has changed:

  • Better error handling. The API can now be more specific about what the error was and more importantly, where it was.
  • The code was refractored to support and internal model, which will help support reverse matching.
  • A new UriTemplateBuilder API for programatically creating URI templates
  • Support for Jackson serializer/deserializers

Overall, it’s cleaner and easier on the eyes. Now to wrap up reverse matching.

Yesterday Handy URI Templates 1.1.7 was released which fixed a few uritemplate-test issues as well as some URI encoding fixes. Hat tip to tekener for those fixes.

With that out of the way, I’ll be focusing on 1.2 which will change the API a bit, but will finally add reverse mapping so that you can use it to match request URLs to a URI template pattern. It turns out that this is a bit more complicated than I first imagined. A number of folks have pointed to the excellent wo-furi project as this already does reverse mapping. However, it only handles level 1, and maybe level 2 templates. Things get hairy when you start reverse mapping level 3 and level 4 templates.

A few years back, I dabbled a bit with Aspect Oriented Programming and dorked around with JBoss AOP and AspectJ. I created a few aspects an threw them up on Java.net. Over the years, I never really kept up with maintaing the project. Since then, Java.net migrated projects and Handy Aspects was removed. Since then, I have received a few requests for the code and I planned on moving it to GitHub.

But rather than simply throwing the code up there “as-is”, I decided to bring the project up to date a bit. For example, I moved the build from Ant to Maven and brought the dependencies up to the latest versions. I also removed JBoss AOP version and everything is now based on AspectJ. With that said, you can follow and fork the new project Handy-Apsects project here.

After a few weeks of tweaking, I put out a new release of Handy URI Templates. What’s important about version 1.1.2 is that it is now being tested against the uritemplate-test suite started by Mark Nottingham. Most importantly, it is also now passing all tests. Additionally, this release also marks the introduction of expression validation as well. If you’ve been using the 1.0.x versions, I’d highly recommend moving up to 1.1.2.

URI Templates can make interacting with the web APIs like the Facebook Graph API a little easier. Here, I’ll show you how you can use URI Templates to create request URIs into the Facebook Graph API using the Handy URI Templates library. The URI Template examples described should be usable by any RFC6570 URI Template processor, but I’ll be focusing on how to use the Handy URI Templates API in Java. If you’re using PHP, have a look at Michael Dowling’s excellent Guzzle Framework which has great URI template support.

URI Template Basics

We’ll assume you have some familiarity with URI templates. Basically, a URI template is expression where replacement variables are identified by the text between ‘{‘ and ‘}’, like so:

https://graph.facebook.com/{id}

Where {id} is the name of the variable. This is similar to how paths are expressed in JAX-RS, and OpenSearch URL template syntax . The RFC6570 spec provides a lot more details on the URI template standard syntax, so we’ll focus on how to use URI templates in conjunction with the Facebook Graph API.

Facebook Graph API Basics

For the most part, most URIs in the Graph API follow the basic pattern of hostname + id. In a URI template, this pattern can be expresed as:

https://graph.facebook.com/{id}

When the template variable is expanded, it will allow you to create URIs that can be used request resources like so:

  • Users: https://graph.facebook.com/btaylor (Bret Taylor)
  • Pages: https://graph.facebook.com/cocacola (Coca-Cola page)
  • Events: https://graph.facebook.com/251906384206 (Facebook Developer Garage Austin)

And so on. Facebook also requires you to supply the access_token in a query parameter on the request URI, so now we end up with the expression:

https://graph.facebook.com/{id}{?access_token}

Please check the Facebook documenation on how to get the token.

Using the Handy URI Templates API, you can create a URI from an expression like so:

String expression = "https://graph.facebook.com/{id}{?access_token}";
String uri  = 
   UriTemplate.fromExpression(expression)
              .set("id", "bgolub")
              .set("access_token", System.getProperty("fb.access_token"))
              .expand();

This will give you the following URI:

https://graph.facebook.com/bgolub?access_token=your_fb_access_token

Because the {id} variable can contain sub paths, we need a way to express that. If we have want to express a URI template that gets a users information or the users photo albums, we need additional path segements. We could use multiple path segments with more variables, but this can make the template more complicated. One option is to modify the expression so that {id} can accomodate a single path segement or multiple path segments by rewriting the expression as:

https://graph.facebook.com{/id*}{?access_token}

This does a few things:

  • By putting the path ‘/’ operator in the variable expression, we’re stating that the values in this variable path segements. By default, if the variable values is an array, Collection or Map, the values will be ‘,’ delimited.
  • The * modifier means ‘explode’. With an array or Collection plus the explode modifier, the values will be delimited by the ‘/’ operator.

Now if we change the code a little bit:

String expression = "https://graph.facebook.com{/id*}{?access_token}";
String uri =  
    UriTemplate.fromExpression(expression)
               .set("id", new String[] {"bgolub","albums"})
               .set("access_token", System.getProperty("fb.access_token"))
               .expand();

You’ll note that the value we pass to the id variable is an array of strings and the processor will render valuse as two strings separated by a /.The resulting URI is now:

https://graph.facebook.com/bgolub/albums?access_token=your_fb_access_token

Now lets see how we can make more advanced types of requests.

More Advanced Requests

The Graph API has a number of query parameters that modifiy the request. All of these are defined in the Facebook Graph API documenation so I won’t detail them here. With all of the query parameters collected, you end up with the following URI template expression:

https://graph.facebook.com{/id*}{?q,ids,fields,type,center,distance,limit,offset,until,since,access_token}

You can pretty much use this template express for the majority, if not all, of the Facebook Graph API. We’ll use that expression for rest of the examples.

Search Request URIs

With the URI template, we can create URIs that map to Graph API search requests. Using the “coffe” example from the Facebook documenation:

String expression = "https://graph.facebook.com{/id*}{?q,ids,fields,type,center,distance,limit,offset,until,since,access_token}";

String uri = 
    UriTemplate.fromExpression(expression)
                .set("id","search")
                .set("q", "coffee")
                .set("type","place")
                .set("center", new float[] {37.76f,-122.427f})
                .set("distance", 1000)
                .set("limit", 5)
                .set("offset", 10)
                .set("access_token", System.getProperty("fb.access_token"))
                .expand();

This will give us the URI:

https://graph.facebook.com/search?q=coffee&type=place&center=37.76,-122.427&distance=1000&limit=5&offset=10&access_token=your_access_token

Note how the variables until and since are not included in the expanded URI.

FQL Request URIs

Even things URIs with FQL queries can be expressed in a template. Give our code:

String expression = "https://graph.facebook.com{/id*}{?q,ids,fields,type,center,distance,limit,offset,until,since,access_token}";

  String uri = 
    UriTemplate.fromExpression(expression)
               .set("id","fql")
               .set("q", "SELECT uid2 FROM friend WHERE uid1=me()")
               .set("access_token", System.getProperty("fb.access_token"))
               .expand();

Wrap Up

Hopefully this gives you a good idea on both how to use URI Templates in general, and a good insight into how you can use teh Handy URI Templates API. If you want more exmaples, have a look at the code on GitHub here. There are examples for Facebook, Twitter, and GitHub.

Having recieved a Kindle Fire this year, I was really surprised at the unpacking process and minimal security. While the device is very nice, the whole unpacking process made me realize that the Kindle Fire is a package thief’s dream. Here’s a few reason why:

Packaging
One thing that makes stealing a Kinlde Fire easy is the packaging, Amazon makes it quite clear what is in the box. When it arrives to your door, or your bulding’s mail room, the package is clearly marked with the words “Kindle Fire” all over it. In my case, the package sat in our building lobby clearly advertising what’s inside:

This makes it really easy to identify which package has the 7-inch tablet inside.

Automatic Sign-In
When I first opened the package and turned on the Kindle for the 1st time, it skipped past the sign-in screen since my wife’s credentials where already entered and she was automatically logged into Amazon. At first, this seemed like a nice touch for folks like my grandmother who don’t quite get sign-in procedures. However, this approach has far more cons than advantages.

1-Click Enabled by Default
The real kicker is that 1-Click is enabled by default. When you purchase content, you are never prompted for a password. This gives potential gift takers an added bonus: the ability to purchase Amazon content on your dime. And if you have kids, they can snag games as often as they want.

So if you’re considering getting an Amazon Kindle for someone this year, make sure you check the box that says “this is a gift” as this will make sure the device doesn’t sign you in automatically when you get it. Even if you’re going to get it for yourself, this is probably a good thing to do. Second, if you live in an apartment building or condo, you might want to consider shipping the box to your work or someplace where the box isn’t in plain sight.

 

I have to admit that the new $99 AppleTV caught my attention when it was announced a few days ago. Small, cheap, works with iTunes, Netflix built in, and all for $99. Sounds pretty cool, right? I don’t think so anymore.

See, in my home everything is wireless and there are only laptops and soon a NAS. I want to have access to all of our files and media assets in one spot and be able to access those files on multiple devices. Most decent NAS solutions provide the storage and retrieval functionalities I need. What’s missing is the devices to play it back on. Clearly, I can play back stuff on my Mac and PC laptops and other devices. Getting media on my TV is another story.

The new AppleTV  has no internal storage and only streams media files from another source. In order to do this, a Mac or PC has to be running with iTunes open in order to stream the media files. This is rather inefficient since one of my laptops would have to be running while I’m watching TV. Now one could use a NAS to stream media files. Most NAS devices can stream audio via some type of iTunes friendly media server. Video likely won’t work on such NAS devices since some content would be DRM’d with FairPlay.  Since Apple does not support DLNA and has gone the proprietary route, there isn’t really a good way to stream video from a centralized source other than a desktop running iTunes.

Apple appears to be hanging on to the “digital hub” mentality whereby the Mac is still the center of “your digital life.” Actually, my  so called “digital life” resides in the data itself and not so much the mechanisms that I use to access it. At $99, the Apple TV isn’t a bargain once you start to realize the extras you’ll need in order to make it participate in a complete solution. For me, this complete solution doesn’t appear to exist yet. The closest thing that comes to it doesn’t bear an an Apple logo but rather a Windows one.

Without a doubt, the URL vs. URI post is by the most visited page on this blog. Even so, there’s still a lot of confusion on the topic and so I thought I’d break it down in less words. The original post was slightly misleading in that I attempted to compare URI to URL, when in fact it should have defined the relationship between URI, URL, and URN. In this post, I hope to clear that in more concise terms. But first, here’s a pretty picture:

uri_class_diagram

Both URLs and URNs are specializations, or subclasses of URI. You can refer to both URLs and URNs as a URI. In applictaion terms: if your application only calls for a URI, you should be free to use either or.
Now, here’s where the big difference between URN and URL: a URL is location bound and dereferencable over the web. A URN is just a name and isn’t bound to a network location. However, BOTH are still valid URIs. Now, if the application requires a URI that is bound to a network location, you must use the specialization of URI called URL.

Remember that URI stands for Uniform Resource Identifier, which is used to identify some “thing”, or resource, on the web. Both URLs and URNs are specialization’s (or subclasses if you will), of URI. You’d be correct by referring to both a URL or URN as a URI. In applictaion terms: if your application only calls for an identifier, you should be free to use either a URL or a URN. For brevity, you can state that the application simply requires a URI and the use of a URL or URN would statisfy that requirement.

Now if your application needs a URI that dereferencable over the web, you should be aware of the difference between URN and URL. A URL is location bound and defines the mechanism as to how to retrieve the resource over the web. A URN is just a name and isn’t bound to a network location. For example, you may have a URN for a books ISBN number in the form of urn:isbn:0451450523. The URN is still a valid URI, but you cannot dereference it, it’s just a name used to provide identity. So to put in simpler terms:

  • A URI is used to define the identity of some thing on the on the web
  • Both URL and URN are URIs
  • A URN only defines a name, it provides no details about how to get the resource over a network.
  • A URL defines how to retrieve the resource over the web.
  • You can get a “thing” via a URL, you can’t get anything with a URN
  • Both URL and URN are URIs as the both identify a resource

There some other items that need clarification based on some comments I’ve received from the original post:

  • Elements of a URI such as query string, file extension, etc. have no bearing on whether or not a URL is a URI. If the URI is defines how to get a resource over the web, it’s a URL.
  • A URL is not limited to HTTP. There are many other protocol schemes that can be plugged into a URL.
  • If a URL defines a scheme other than HTTP, it does not magically become a URI. The URI defines how to get the resource, whether it be HTTP, FTP, SMB, etc., it’s still a URL. But because the URL identifies a resource, it’s a URI as well.

Yeah, I’ve probably repeated myself a few times, but I wanted to stress a few points.

There’s also been some confusion about when to use the term URI. As I stated in the original post explained above, it depends on what you’re doing. If everything your application does involves accessing data over the web, you’re most likely using URL exclusively. In that case, it wouldn’t be a bad thing to use the term URL. Now, if the application can use either a network location, or a name, then URI is the proper term. For example, XML namespaces are usually declared using a URI. The namespace may just be a name, or a URL that references a DTD or XML Schema. So if you’re using a URL for identity and retrieval, it’s probably best to use URI.

When it comes to manipulating photographs, I live in Photoshop. One feature of all Adobe products that I like is the ability to annotate images and other documents using their eXtensible Metadata Platform, or XMP. XMP is a collection of RDF statements that get embedded into a document that describe many facets of the document. I’ve always wanted to be able to somehow get that data out of these files and doing something with it for application purposes.

There are projects like Jempbox, which work on manipulating the XMP data but offers no facilities to extract the XMP packet from image files. The Apache XML Graphics Commons is more the ticket I was looking for. The library includes and XMP parser that performs by scanning a files for the XMP header. The approach works quite well and supports pretty much every format supported by the XMP specification. The downside of XML Graphics Commons is that it doesn’t property read all of the RDF statements. Some of the data is skipped or missed completely. To top it off, neither framework allows you to get at the raw RDF data.

What I really wanted to do was to get the XMP packet in its entirety and load it into a triples store like Sesame or Virtuoso. This of course means that you want to have the data available as RDF. Rather than inventing my own framework to do all of this, I found the Aperture Framework. Aperture is simply amazing framework that can extract RDF statements from just about anything. Of course, the one thing that is missing is XMP support. So, I set out on implementing my own Extractor that can suck out the entire XMP packet as RDF. It’s based on the work started in the XML Graphics Commons project, but modified significantly so that it pulls out the RDF data. Once extracted, it’s very easy to store the statements into a triple store and execute SPARQL queries on it.

Right now the, this  XMPExtractor can read XMP from the following formats:

  • JPEG Images (image/jpeg)
  • TIFF Images (image/tiff)
  • Adobe DNG (image/x-adobe-dng)
  • Portable Network Graphic (image/png)
  • PDF (application/pdf)
  • EPS, Postscipt, and Adobe Illustrator files (application/postscript)
  • Quicktime (video/quicktime)
  • AVI (video/x-msvideo)
  • MPEG-4 (video/mp4)
  • MPEG-2 (video/mpeg)
  • MP3 (audio/mpeg)
  • WAV Audio (audio/x-wav)

On the downside, I’ve found that if you use the XMPExtractor with a Crawler, you’ll run into some problems with Adobe Illustrator files. The problem is that the PDFExtractor mistakes these files for PDFs and then fails. But as long as you’re not using Illustrator files, you should be ok. There’s also a few nitpicks with JPEG files and the JpgExtractor in that the sample files included in the XMP SDK are flagged as invalid JPEG files. However, every JPEG file I created from Photoshop and iPhoto seem to work fine. But after a little more testing, I’ll look at offering it up as a contribution to the project.