App Engine 1.5.0 Release

The App Engine team has been working furiously in preparation for Google I/O time and today, we are excited to announce the release of App Engine 1.5.0, complete with a bunch of new features. This release brings a whole new dimension to App Engine Applications with the introduction of Backends, some big improvements to Task Queues, a completely new, experimental runtime for the Go language, High Replication Datastore as the new default configuration (and a lower price!), and even more tweaks and bug fixes.

Serving Changes

• Backends: Until now all App Engine applications have been running on short-lived dynamic instances that we spin up and down in response to application requests. This is great for building scalable web applications, but has a number of limitations if you are looking to build larger, long-lived, and/or memory intensive infrastructure. With 1.5.0, we are introducing Backends which will allow developers to do precisely this! Backends are developer-controlled, long-running, addressable sets of instances which are as large as the developer specifies. There are no request deadlines, they can be started and stopped (or dynamically start when called), they can use between 128M and 1G of memory and proportional CPU. If you’d like to find out more, have a read through our Backend docs for Java and Python.


• Pull Queues: Most of our users are heavily using Task Queues in their applications today, but there is lots of room for more flexibility. With 1.5.0 we are introducing Pull Queues to allow developers to “pull” tasks from a queue as applications are ready to process them, rather than relying on Task Queues to push tasks at a pre-configured rate. This means you can write a Backend to do some background processing and pull 1, 10, or 100s of tasks off the Pull Queue when the Backend is ready for more. In addition, we’ve introduced a REST API which will allow external services to do the same thing. For example, if you have an external server running to do image conversion or OCR, you can now use the REST API to pull tasks off, run them, and return the results. In conjunction with these 2 improvements, we’ve also increased the payload limits and processing rate. We are excited both about expanding the use of Task Queues as well as improving the ease of integration between App Engine and the rest of the cloud.

Datastore

• High Replication Datastore as default: After months of usage and feedback on the High Replication datastore (as well as a record of 99.999% uptime so far) we are now confident that it is the right path forward for the majority of our users. So, today we are doing two things: setting HRD as the default for all new apps created, lowering the price of HRD storage from $0.45 down to $0.24, and encouraging everybody to begin plans to migrate. We really appreciate all the time that early users of HRD put into trying it out and finding issues and have fixed a number of those issues with this release.

Changed APIs

• In response to popular demand, the HTTP request and response sizes have been increased to 32 MB.


• Mail API: We have added a few restrictions to the Mail API to improve the reliability and reputation of the service for all applications. First, emails must be sent from email accounts managed by Google (either Gmail, or a domain signed up for Google Apps). Second, we’ve reduced the number of free recipients per day from 2000 to 100 for newly created applications. Both of these will help ensure mail from your application arrives at the destination reliably.

Administration

• Code downloads: As of 1.5.0, we have expanded the ability to download an Application’s source code to include both the user who uploaded the code to download it as well as the Owner(s) of the project as listed in the Admin Console. Owners were introduced in 1.4.2 as an admin role.

Go

• New runtime: With 1.5.0 we are launching an experimental runtime for the Go Programming Language. Go is an open source, statically typed, compiled language with a dynamic and lightweight feel. It’s also an interesting new option for App Engine because Go apps will be compiled to native code, making Go a good choice for more CPU-intensive tasks. As of today, the App Engine SDK for Go is available for download, and we will soon enable deployment of Go apps into the App Engine infrastructure. If you’re interested in starting early, sign up to be first through the door when we open it up to early testers. See the announcement on the Go Blog for more details.

There are plenty of additional changes and bug fixes in this release so please check out the full release notes, including all issues fixed for Java and Python. Finally if you are interested in where App Engine is heading later this year, check out our other announcement at I/O 2011!

The App Engine Team

Who's at Google I/O: Evite

This post is part of Who's at Google I/O, a series of guest blog posts written by developers who are appearing in the Developer Sandbox at Google I/O. It's also cross-posted to the Google Code blog which has similar posts for all sorts of Google developer products.

Evite is one of web's oldest social planning services. Since the launch in 1998, Evite has delivered over a billion party invitations. Although it has served us well, after ten years of operation it became necessary to replace the aging platform, and position Evite for another decade of successful party planning.

The reengineering effort took well over a year. Our development team replaced Java with Python and saw a significant increase in developer productivity. However, dealing with a ten year-old Oracle database remained a challenge. To overcome scalability limitations and inflexibility of a large relational DBMS, we designed and implemented a proprietary data store. We conducted a series of A/B tests and gradually started migrating production traffic to the new system.

Our initial motivation to use Google App Engine was fast provisioning. We introduced App Engine as a temporary solution while we increased scale and optimized our proprietary data store. However, once we imported user profile data and put App Engine backend services in production, we never looked back.

Importing a large data set of user profiles from Oracle RAC into App Engine was challenging at first because we had to perform a bulk import and then keep the data synchronized between the two datastores. As we developed our data synchronization tools we gained better understanding of the API and performance characteristics of the App Engine datastore.

Once we synchronized profile data and enabled production traffic, App Engine really started to impress. We would watch our daily traffic grow and observe App Engine’s automatic scaling in action. Additional server instances would come online to meet increased demand and disappear as traffic lowered, without any sysadmin intervention. Despite our proficiency in the use of cloud computing resources and system automation, it was never this easy to provision new servers. As Grig, Evite’s devops guru, likes to say "It's not in production until it's monitored and graphed." App Engine’s dashboard automatically manages instances and graphs system usage data.

Following our positive experience with Evite profiles, we have continued to use App Engine for other services. Occasional frustrations with elevated error rates on the Master/Slave Datastore disappeared as we switched to the High Replication Datastore. At this point we see no technical obstacles to using App Engine more extensively. Architectural decisions and best practices implemented by the App Engine team are very well aligned with the choices we made in designing Evite’s new platform. This makes it easy to deploy additional services and data sets to App Engine.

The real obstacle to using App Engine exclusively for most Evite services is risk management. As reliable and cost-effective as App Engine has been for us, it is difficult to depend completely on a single vendor/service provider. To ensure availability of our application we use multiple cloud providers. Unfortunately, this strategy prevents us from using certain App Engine features and API's because we do not have adequate replacements for them in other deployment environments.

Evite’s new platform built on Python, NoSQL and App Engine has been a success. Our new web application has been well received by users, and our first iPhone app is receiving positive reviews. (Android version is in the works). We look forward to continued use of App Engine for data warehousing, rapid launches of new services and other projects.

Come see Evite in the Developer Sandbox at Google I/O on May 10-11.

Dan Mesh is a Vice President of Engineering at Evite. His extended responsibilities include espresso deliveries and release day food orders.

Posted by Scott Knaster, Editor

Google I/O: countdown to the keynote kickoff

(Cross-posted from the Code Blog)

In less than 24 hours, we’ll be kicking off Google I/O 2011, our annual developer conference here in San Francisco. This year’s keynote presentations will highlight the biggest opportunities for developers and feature two of our most popular and important developer platforms: Android and Chrome. Google engineers from Andy Rubin and Sundar Pichai’s teams will unveil new features, preview upcoming updates, and provide new insights into the growing momentum behind these platforms.

Plus, for the first time in Google I/O history, you’ll be able to join us throughout the two days at I/O Live. We’ll live stream the two keynote presentations, two full days of Android and Chrome technical sessions, and the After Hours party. Recorded videos from all sessions across eight product tracks will be available within 24 hours after the conference. Whether you’ll be joining us in San Francisco or from the farthest corner of the world, bookmark www.google.com/io and check back on May 10 at 9:00 a.m. PDT for a fun treat as we count down to 00:00:00:00.

Posted by Vic Gundotra, Senior Vice President of Engineering

Share and personalize your Google Goggles experience with Goggles 1.4

Since its launch, we have worked to make Google Goggles faster and more accurate when returning search results for a wide variety of images. Today, we are taking several steps to make Goggles a better and more personal experience. Goggles 1.4 for Android devices introduces an enhanced search history experience, the ability to suggest better results to Google if we are not able to accurately match your image, and improved business card recognition.

Enhanced search history
The new search history experience lets you search your Goggles results, make personal notes on specific results and share your results with friends. When you add a personal note to a Goggles result, the note will appear in your search history. I’m trying to learn more about wine, so when I taste something new, it’s easy for me to add a note to help me remember what I liked about the wine. Later, I can search my search history for words in my note to help me find that bottle that went so well with steak. Read more about how to enable search history for Goggles here.

To make a personal note, tap the pencil in the corner when viewing a search result.

Notes are intended to help you better organize your search history, so if you choose to share a result with a friend, your notes will not be shared. However, you can always add a personalized message to your friend when you share your results with them.

Share a result by sending a link to your friends.

Suggest a better result
We are constantly working to improve the accuracy of Goggles at recognizing certain categories of items, but visual recognition is still a complicated task. With Goggles 1.4, you are now able to suggest a better result when Goggles cannot find an image match, or the quality of the result is poor.

To send your suggestion to Google, tap “Can you suggest a better result?” on the results page. You can then select the relevant part of the image and submit a tag. Tags will be used to improve recognition in object categories where Goggles already provides some results, like artwork or wine bottles.

When suggesting a better result, you can crop the image and add a description.

Improved business card recognition
Business card recognition is one of the most popular uses of Google Goggles, so we're rolling out some new updates to make the experience even quicker and easier. Additionally, instead of simply recognizing the content as text, Goggles now recognizes the information as a contact, making it easier to add to your phone's contact list.

Call or add the person directly as a contact

Google Goggles is available for Android 1.6+ devices. Download it by visiting Android Market or by scanning the QR code below:

Posted by Jacob Smullyan, Software Engineer

Accessing Gmail accounts from App Engine with Context.IO

This is part of our on going series of blog posts from guest authors highlighting success stories from applications and services built on or targeting App Engine developers. Today, we have a post from Bruno Morency of Context.IO. Bruno has been involved in startups since graduating from McGill Engineering in 2001. And since being introduced to Pine on UNIX terminals, he’s had a love-hate relationship with email.

Email mailboxes contain years of important conversations and business information yet there are no easy ways for App Engine developers to find and use that information. This is what Context.IO does. It’s the missing email API that turns mailboxes into a data source developers can leverage. This is particularly interesting for Google App Engine developers since it makes content of Gmail accounts (and any other IMAP accessible email account) available through a set of HTTP calls.

In this post, we’re going to walk you through using Context.IO to build an app that lets users search for contacts from their inbox then easily get the history of recent emails and attachments exchanged with these contacts. A working demo is available at http://contextio-demo.appspot.com. The code for this application is available on Context.IO’s GitHub account.

Querying the mailbox

Our demo is quite simple in its functionality: there’s a search box used to find contacts, and once a contact has been found, we list recent emails and attachments associated with the contact. To do this, the application offers 3 urls that are called by the JavaScript running in the browser to obtain the data: search.json, messages.json and files.json.

The actual UI formatting is all implemented in the JavaScript running in the browser, but our focus here is how we get that data out of the mailbox and return it to the browser.

Let’s see how we respond to the request to get message history for a given contact. This is done by calling /messages.json which accepts an email address as a GET parameter. Note, this functionality requires an authentication step not shown here. The code behind that call is as follows:

class MessagesHandler(webapp.RequestHandler):
    def get(self):
        current_user = users.get_current_user()
        current_email = current_user.email()

        emailAddr = self.request.get('email')
        contextIO = ContextIO(api_key=settings.CONTEXTIO_OAUTH_KEY,
                              api_secret=settings.CONTEXTIO_OAUTH_SECRET,
                              api_url=settings.CONTEXTIO_API_URL)
        response = contextIO.contactmessages(emailAddr,account=current_email)
        self.response.out.write(simplejson.dumps(response.get_data()))

The code simply uses the contactmessages.json API call of and returns all the messages including the subject, other recipients, thread ID, and even attachments in JSON format. Check the documentation for a complete breakdown of the response.

You can see how we handle the other cases similarly in handlers.py. To get a complete overview of other calls offered by Context.IO, please refer to the documentation available here http://context.io/docs/latest.

Building your own email based application

The complete code for this demo application has been made available by the Context.IO team on our GitHub account. Feel free to use it as the base for your own application. To get started, you’ll need to setup your application with your own Google API authorization and Context.IO API keys.

If you have questions about using Context.IO to embed your user’s email in your app, feel free to contact us through http://support.context.io.

Posted by Bruno Morency, Co-founder of Context.IO