{"id":2266,"date":"2010-07-09T10:11:00","date_gmt":"2010-07-09T10:11:00","guid":{"rendered":"https:\/\/ikaisays.com\/?p=2266"},"modified":"2021-04-20T10:11:52","modified_gmt":"2021-04-20T10:11:52","slug":"using-the-java-mapper-framework-for-app-engine","status":"publish","type":"post","link":"https:\/\/ikaisays.com\/2010\/07\/09\/using-the-java-mapper-framework-for-app-engine\/","title":{"rendered":"Using the Java Mapper Framework for App Engine"},"content":{"rendered":"\n

The recently released Mapper framework<\/a> is the first part of App Engine\u2019s mapreduce offering. In this post, we\u2019ll be discussing some of the types of operations we can perform using this framework and how easily they can be done.<\/p>\n\n\n\n

Introduction to Map Reduce<\/h2>\n\n\n\n

If you aren\u2019t familiar with Map Reduce, read more about it from a high level from Wikipedia here<\/a>. The official paper can be downloaded from this site<\/a> if you\u2019re interested in a more technical discussion.<\/p>\n\n\n\n

The simplest breakdown of MapReduce is as follows:<\/p>\n\n\n\n

  1. Take a large dataset and break it into pieces, mapping individual pieces of data<\/li>
  2. Work on those mapped datasets and reduce them into the form you need<\/li><\/ol>\n\n\n\n

    A simple example here is full text indexing. Suppose we wanted create indexes from existing text documents. We would use the Map step to iterate over every document and \u201cmap\u201d each phrase or term to a document, then we would \u201creduce\u201d the mappings by writing them to an index. Map\/reduce problems have the advantage of not only being easy to conceptualize as problems that can be distributed and parallelized, but also because there are frameworks that support many of the administrative functions of map-reduce: failure recovery, distribution of work, tracking status of jobs, reporting and so forth. The appengine-mapreduce project<\/a> seeks to provide as many of these features as possible while making it as easy as possible for developers to write large batch processing jobs without having to think about the plumbing details.<\/p>\n\n\n\n

    But I only have Map available!<\/h3>\n\n\n\n

    Yes, this is true  \u2013 as of the writing of this post, only the \u201cmap\u201d step exists, hence why it\u2019s currently referred to as the \u201cMapper API\u201d. That doesn\u2019t mean it\u2019s not useful. For starters, it is a very easy way to perform some operation on every single Entity of a given Kind in your datastore in parallel. What would you have to build for yourself if Mapper weren\u2019t available?<\/p>\n\n\n\n

    1. Begin querying over every Entity in chained Task Queues<\/li>
    2. Store beginning and end cursors (introduced in 1.3.5)<\/li>
    3. Create tasks to work with chunks of your datastore<\/li>
    4. Write the code to manipulate your data<\/li>
    5. Build an interface to control your batch jobs<\/li>
    6. Build a callback system for your multitudes of parallelized workers to call when the entire task has completed<\/li><\/ol>\n\n\n\n

      It\u2019s certainly not a trivial amount of work. Some things you can do very easily with the Mapper library include:<\/p>\n\n\n\n