{"id":2278,"date":"2010-06-29T20:06:00","date_gmt":"2010-06-29T20:06:00","guid":{"rendered":"https:\/\/ikaisays.com\/?p=2278"},"modified":"2021-04-20T20:11:22","modified_gmt":"2021-04-20T20:11:22","slug":"using-asynchronous-urlfetch-on-java-app-engine","status":"publish","type":"post","link":"https:\/\/ikaisays.com\/2010\/06\/29\/using-asynchronous-urlfetch-on-java-app-engine\/","title":{"rendered":"Using Asynchronous URLFetch on Java App Engine"},"content":{"rendered":"\n

Developers building applications on top of Java App Engine can use the familiar java.net interface<\/a> for making off-network calls. For simple requests, this should be more than sufficient. The low-level API, however, provides one feature not available in java.net: asynchronous URLFetch.<\/p>\n\n\n\n

The low-level URLFetch API<\/h2>\n\n\n\n

So what does the low-level API look like? Something like this:<\/p>\n\n\n\n

import java.io.IOException;\nimport java.net.URL;\nimport java.util.List;\n \nimport com.google.appengine.api.urlfetch.HTTPHeader;\nimport com.google.appengine.api.urlfetch.HTTPResponse;\nimport com.google.appengine.api.urlfetch.URLFetchService;\nimport com.google.appengine.api.urlfetch.URLFetchServiceFactory;\n \n        URLFetchService fetcher = URLFetchServiceFactory.getURLFetchService();\n        try {\n            URL url = new URL(\"http:\/\/someurl.com\");\n            HTTPResponse response = fetcher.fetch(url);\n \n            byte[] content = response.getContent();\n                     \/\/ if redirects are followed, this returns the final URL we are redirected to\n            URL finalUrl = response.getFinalUrl();\n \n            \/\/ 200, 404, 500, etc\n            int responseCode = response.getResponseCode();\n            List headers = response.getHeaders();\n \n            for(HTTPHeader header : headers) {\n                String headerName = header.getName();\n                String headerValue = header.getValue();\n            }\n \n        } catch (IOException e) {\n            \/\/ new URL throws MalformedUrlException, which is impossible for us here\n        }<\/code><\/pre>\n\n\n\n
The full Javadocs are here<\/a>.<\/p>\n\n\n\n
So it\u2019s a bit different than the standard java.net interface<\/a>, where we\u2019d get back a reader and iterate line by line on the response. We\u2019re protected from a heap overflow because URLFetch is limited to 1mb responses.<\/p>\n\n\n\n
Asynchronous vs. Synchronous requests<\/h2>\n\n\n\n
Using java.net has the advantage of portability \u2013 you could build a standard fetcher that will work in any JVM environment, even those outside of App Engine. The tradeoff, of course, is that App Engine specific features won\u2019t be present. The one killer feature of App Engine\u2019s low-level API that isn\u2019t present in java.net is asynchronous URLFetch. What is asynchronous fetch? Let\u2019s make an analogy:<\/p>\n\n\n\n
Let\u2019s pretend you, like me at home, are on DSL and have a pretty pathetic downstream speed, and decide to check out a link sharing site like Digg<\/a>. You browse to the front page and decide to open up every single link. You can do this synchronously or asynchronously.<\/p>\n\n\n\n
Synchronously<\/h3>\n\n\n\n
Synchronously, you click link #1. Now you look at this page. When you are done looking at this page, you hit the back button and click link #2. Repeat until you have seen all the pages. Now, again, you are on DSL, so not only do you spend time reading each link, before you read each destination page, you have to wait for the page to load. This can take a significant portion of your time. The total amount of time you sit in front of your computer is thus:<\/p>\n\n\n\n
N = number of links\nL = loading time per page\nR = reading time per page\n\nN * (L + R)<\/pre>\n\n\n\n
(Yes, before I wrote this equation, I thought that by including some mathematical formulas in my blog post would make me look smarter, but as it turns out the equation is something comprehensible by 8 year olds internationally\/American public high school students)<\/p>\n\n\n\n
Asynchronously<\/h3>\n\n\n\n
Using a tabbed browser<\/a>, you control-click every single link on the page to open them in new tabs. Now before you go look at any of the pages, you decide to go to the kitchen and make a grilled cheese sandwich<\/a>. When you are done with the sandwich, you come back to your computer sit down, enjoying your nice, toasty sandwich while you read articles about Ron Paul<\/a> and look at funny pictures of cats<\/a>. How much time are you spending?<\/p>\n\n\n\n
N = number of links\nS = loading time for the slowest loading page\nR = reading time per page\nG = time to make a grilled cheese sandwich<\/pre>\n\n\n\n
MAX((N * R + G), (N * R + S))<\/pre>\n\n\n\n
Which takes longer, your DSL, or the time it takes you to make a grilled cheese sandwich? The point that I\u2019m making here is that you can save time by parallelizing things. No, I know it\u2019s not a perfect analogy, as downloading N pages in parallel consumes the same crappy DSL connection, but you get what I am trying to say. Hopefully. And maybe you are also in the mood for some cheese.<\/p>\n\n\n\n
Asynchrous URLFetch in App Engine<\/h2>\n\n\n\n
So what would the URLFetch above look like asynchronously? Probably something like this:<\/p>\n\n\n\n
import java.io.IOException;\nimport java.net.URL;\nimport java.util.List;\nimport java.util.concurrent.ExecutionException;\nimport java.util.concurrent.Future;\n \nimport com.google.appengine.api.urlfetch.HTTPHeader;\nimport com.google.appengine.api.urlfetch.HTTPResponse;\nimport com.google.appengine.api.urlfetch.URLFetchService;\nimport com.google.appengine.api.urlfetch.URLFetchServiceFactory;\n \n    protected void makeAsyncRequest() {\n        URLFetchService fetcher = URLFetchServiceFactory.getURLFetchService();\n        try {\n            URL url = new URL(\"http:\/\/someurl.com\");\n            Future future = fetcher.fetchAsync(url);\n \n            \/\/ Other stuff can happen here!\n \n            HTTPResponse response = future.get();\n            byte[] content = response.getContent();\n            URL finalUrl = response.getFinalUrl();\n            int responseCode = response.getResponseCode();\n            List headers = response.getHeaders();\n \n            for(HTTPHeader header : headers) {\n                String headerName = header.getName();\n                String headerValue = header.getValue();\n            }\n \n        } catch (IOException e) {\n            \/\/ new URL throws MalformedUrlException, which is impossible for us here\n        } catch (InterruptedException e) {\n            \/\/ Exception from using java.concurrent.Future\n        } catch (ExecutionException e) {\n            \/\/ Exception from using java.concurrent.Future\n            e.printStackTrace();\n        }\n \n    }<\/code><\/pre>\n\n\n\n
This looks pretty similar \u2013 EXCEPT: fetchAsync doesn\u2019t return an HTTPResponse. It returns a Future. What is a future?<\/p>\n\n\n\n
java.concurrent.Future<\/h2>\n\n\n\n
From the Javadocs<\/a>:<\/p>\n\n\n\n
\u201cA Future represents the result of an asynchronous computation. Methods are provided to check if the computation is complete, to wait for its completion, and to retrieve the result of the computation. The result can only be retrieved using method get when the computation has completed, blocking if necessary until it is ready. Cancellation is performed by thecancel method. Additional methods are provided to determine if the task completed normally or was cancelled.\u201d<\/p><\/blockquote>\n\n\n\n
Future<\/h2>\n\n\n\n
What does this mean in English? It means that the Future object is NOT the response of the HTTP call. We can\u2019t get the response until we call the get() method. Between when we call fetchAsync() and get, we can do other stuff: datastore operations, insert things into the Task Queue, heck, we can even do more URLFetch operations. When we finally DO call get(), one of two things happens:<\/p>\n\n\n\n
  1. We\u2019ve already retrieved the URL. Return an HTTPResponse object<\/li>
  2. We\u2019re still retrieving the URL. Block until we are done, then return an HTTPResponse object.<\/li><\/ol>\n\n\n\n
    In the best case scenario, the amount of time it takes for us to do other things is equal to the amount of time it takes to do the URLFetch, and we save time. In the worst case scenario, the maximum amount of time is the time it takes to do the URLFetch or the other operations, whichever takes longer. The end result is that we lower the amount of time to return a response to the end-user.<\/p>\n\n\n\n
    Twitter Example<\/h2>\n\n\n\n
    So let\u2019s build a servlet that retrieves my tweets<\/a>. Just for giggles, let\u2019s do it 20 times and see what the performance difference is. We\u2019ll make it so that if we pass a URL parameter, async=true (or anything, for simplicity), we do the same operation using fetchAsync. The code is below:<\/p>\n\n\n\n
    package com.ikai.urlfetchdemo;\n \nimport java.io.IOException;\nimport java.io.PrintWriter;\nimport java.net.URL;\nimport java.util.ArrayList;\nimport java.util.concurrent.ExecutionException;\nimport java.util.concurrent.Future;\n \nimport javax.servlet.http.HttpServlet;\nimport javax.servlet.http.HttpServletRequest;\nimport javax.servlet.http.HttpServletResponse;\n \nimport com.google.appengine.api.urlfetch.HTTPResponse;\nimport com.google.appengine.api.urlfetch.URLFetchService;\nimport com.google.appengine.api.urlfetch.URLFetchServiceFactory;\n \n@SuppressWarnings(\"serial\")\npublic class GetTwitterFeedServlet extends HttpServlet {\n \n    protected static String IKAI_TWITTER_RSS = \"http:\/\/twitter.com\/statuses\/user_timeline\/14437022.rss\";\n \n    public void doGet(HttpServletRequest req, HttpServletResponse resp)\n            throws IOException {\n \n        boolean isSyncRequest = true;\n \n        if(req.getParameter(\"async\") != null) {\n            isSyncRequest = false;\n        }\n \n        resp.setContentType(\"text\/html\");\n        PrintWriter out = resp.getWriter();\n        out.println(\"<h1>Twitter feed fetch demo<\/h1>\");\n \n        long startTime = System.currentTimeMillis();\n        URLFetchService fetcher = URLFetchServiceFactory.getURLFetchService();\n        URL url = new URL(IKAI_TWITTER_RSS);\n \n        if(isSyncRequest) {\n            out.println(\"<h2>Synchronous fetch<\/h2>\");\n            for(int i = 0; i < 20; i++) {\n                HTTPResponse response = fetcher.fetch(url);\n                printResponse(out, response);\n            }\n        } else {\n            out.println(\"<h2>Asynchronous fetch<\/h2>\");\n            ArrayList<Future<HTTPResponse>> asyncResponses = new ArrayList<Future<HTTPResponse>>();\n            for(int i = 0; i < 20; i++) {\n                Future<HTTPResponse> responseFuture = fetcher.fetchAsync(url);\n                asyncResponses.add(responseFuture);\n            }\n \n            for(Future<HTTPResponse> future : asyncResponses){\n                HTTPResponse response;\n                try {\n                    response = future.get();\n                    printResponse(out, response);\n                } catch (InterruptedException e) {\n                    \/\/ Guess you would do something here\n                } catch (ExecutionException e) {\n                    \/\/ Guess you would do something here\n                }\n            }\n \n        }\n \n        long totalProcessingTime = System.currentTimeMillis() - startTime;\n        out.println(\"<p>Total processing time: \" + totalProcessingTime + \"ms<\/p>\");\n    }\n \n    private void printResponse(PrintWriter out, HTTPResponse response) {\n        out.println(\"<p>\");\n        out.println(\"Response: \" + new String(response.getContent()));\n        out.println(\"<\/p>\");\n    }\n \n}<\/code><\/pre>\n\n\n\n
    As you can see, it\u2019s a bit more involved to store all the Futures in a list, then to iterate through them. We\u2019re also not being too intelligent about iterating through the futures: we\u2019re assuming first-in-first-out (FIFO) with URLFetch, which may or may not be the case in production. A more optimized case may try to fetch the response from a call we know is faster before fetching from one we know is slower first. However \u2013 empirical testing will show that more often than not, doing things asynchronously is significantly faster for the user than synchronously.<\/p>\n\n\n\n
    Using Asynchronous URLFetch and HTTP POST<\/h2>\n\n\n\n
    So far, our examples have been focused on read operations. What if we don\u2019t care about the response? For instance, what if we decide to make an API call that essentially is a \u201cwrite\u201d operation, and can, for the most part, safely assume it will succeed?<\/p>\n\n\n\n
    \/\/ JavaAsyncUrlFetchDemoServlet.java\npackage com.ikai.urlfetchdemo;\n \nimport java.io.IOException;\n \nimport javax.servlet.http.HttpServlet;\nimport javax.servlet.http.HttpServletRequest;\nimport javax.servlet.http.HttpServletResponse;\n \nimport com.google.appengine.api.urlfetch.URLFetchService;\nimport com.google.appengine.api.urlfetch.URLFetchServiceFactory;\n \n@SuppressWarnings(\"serial\")\npublic class JavaAsyncUrlFetchDemoServlet extends HttpServlet {\n \n    public void doGet(HttpServletRequest req, HttpServletResponse resp)\n            throws IOException {\n \n        long startTime = System.currentTimeMillis();\n        URLFetchService fetcher = URLFetchServiceFactory.getURLFetchService();\n        fetcher.fetchAsync(FetchHelper.makeGuestbookPostRequest(\"Async\", \"At\" + startTime));\n        long totalProcessingTime = System.currentTimeMillis() - startTime;\n \n        resp.setContentType(\"text\/html\");\n        resp.getWriter().println(\"<h1>Asynchronous fetch demo<\/h1>\");\n        resp.getWriter().println(\"<p>Total processing time: \" + totalProcessingTime + \"ms<\/p>\");\n    }\n \n}<\/code><\/pre>\n\n\n\n
    \/\/ FetchHelper.java\npackage com.ikai.urlfetchdemo;\n \nimport java.net.MalformedURLException;\nimport java.net.URL;\n \nimport com.google.appengine.api.urlfetch.HTTPMethod;\nimport com.google.appengine.api.urlfetch.HTTPRequest;\n \npublic class FetchHelper {\n \n    protected static final String signGuestBookUrl = \/\/\"http:\/\/bootcamp-demo.appspot.com\/sign\";\n \n    public static HTTPRequest makeGuestbookPostRequest(String name, String content){\n        HTTPRequest request = null;\n        URL url;\n        try {\n            url = new URL(signGuestBookUrl);\n            request = new HTTPRequest(url, HTTPMethod.POST);\n            String body = \"name=\" + name + \"&amp;content=\" + content;\n            request.setPayload(body.getBytes());\n \n        } catch (MalformedURLException e) {\n            \/\/ Do nothing\n        }\n        return request;\n    }\n}<\/code><\/pre>\n\n\n\n
    I\u2019ve decided to spam my own guestbook here, for better or for worse.<\/p>\n\n\n\n
    Download the code<\/h2>\n\n\n\n
    You can download the code from this post here using git: http:\/\/github.com\/ikai\/Java-App-Engine-Async-URLFetch-Demo<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
    Developers building applications on top of Java App Engine can use the familiar java.net interface for making off-network calls. For simple requests, this should be more than … Read More<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_generate-full-width-content":"","footnotes":""},"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/ikaisays.com\/wp-json\/wp\/v2\/posts\/2278"}],"collection":[{"href":"https:\/\/ikaisays.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ikaisays.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ikaisays.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ikaisays.com\/wp-json\/wp\/v2\/comments?post=2278"}],"version-history":[{"count":2,"href":"https:\/\/ikaisays.com\/wp-json\/wp\/v2\/posts\/2278\/revisions"}],"predecessor-version":[{"id":2281,"href":"https:\/\/ikaisays.com\/wp-json\/wp\/v2\/posts\/2278\/revisions\/2281"}],"wp:attachment":[{"href":"https:\/\/ikaisays.com\/wp-json\/wp\/v2\/media?parent=2278"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ikaisays.com\/wp-json\/wp\/v2\/categories?post=2278"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ikaisays.com\/wp-json\/wp\/v2\/tags?post=2278"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}