App Engine 1.5.2 SDK Released

As the summer heat descends on the Northern Hemisphere we thought we’d release our newest App Engine version with some changes that are sure to keep you playing around in the cool, air-conditioned indoors (hey, you don’t want your computer to overheat, right?).

Production Changes

• Adjustable Scheduler Parameters - As we previously discussed, we are introducing two scheduler knobs (okay, they actually look like sliders) that will allow you to control some of the parameters that influence how many Instances run your application. Today you will be able to set the minimum pending latency and maximum number of idle instances for your application.

Datastore Changes

• Advanced Query Planning - We are removing the need for exploding indexes and reducing the custom index requirements for many queries. The SDK will suggest better indexes in several cases and an upcoming article will describe what further optimizations are possible.


• Namespaced Datastore Stats - Now, in addition to getting overall datastore stats, we are providing a new option to query datastore stats per namespace.

Task Queue Changes

• New Task Queue details page - We’ve revamped the Task Queue details page in the Administration Console to provide more information about the tasks being run. You can now see the headers included in the enqueued task, the payload, and information from previous task runs.


• 1MB Pull Task Size - It’s our belief that there is only one way for size limits to go - and that’s up! So with this release we’ve increased the size for pull tasks to 1MB.


• Pull queue lease modification - We’ve introduced a new method for Pull Queues that allows you to extend the lease on existing tasks if the initial lease on the task was insufficient.

Lastly, we have some exciting news related to the experimental Go runtime. While it still remains experimental, starting with 1.5.2, all HRD apps will have access to the Go runtime in production.

As always, there are also some small features and bug fixes, the full list of which can be found in our release notes (Python, Java). We look forward to your feedback and questions in our forum.

Posted by The App Engine Team

Nexus S Comes to AT&T

Since launching Nexus S with Samsung, T-Mobile and Sprint customers have enjoyed the pure Google experience, running the latest Android releases and Google mobile apps. Today we’re excited to announce a Nexus S device optimized for AT&T, available in Best Buy stores this weekend.

Nexus S runs Android 2.3 and features a 4” Super AMOLED screen, a 1 GHz processor for showing 3D graphics, front and rear facing cameras and support for NFC. It also features the latest Google mobile apps, including Google Earth, Google Maps with Navigation, Google Talk with video chat, Google Voice and Voice Actions, as well as access to more than 250,000 apps in Android Market.

Nexus S for AT&T will be available in Best Buy stores starting July 24. For those of you who just can’t wait, it’s on pre-sale today in Best Buy stores.

You can find more information at google.com/nexus or follow @googlenexus on Twitter for the latest Nexus S updates. And if you need something to pass the time until you can get your hands on one, try our Nexus Contraptions game, where you can bubble, bounce, and bump apps into the phone.


Updated 1:09pm 7/21/2011: Pre-sales are currently only available in Best Buy stores, not on www.bestbuy.com. 

Posted by Hiroshi Lockheimer, Director of Engineering

Postmortem: Java App Engine outage, July 14, 2011

Summary

Last week, we posted about a limited outage on July 14, 2011. Now that our internal postmortem is complete, we thought you would also like to get more detail about what went wrong and what we are going to do to ensure this doesn't happen again.

Root Cause and Analysis

The main lesson learned is to improve our live traffic testing as a relatively minor bug triggered a corner case for some of our customers. The bug was in a new release of the infrastructure in the App Engine Java execution environment. During development, testing, and qualification, this bug was essentially hidden from view because it only manifested itself under specific load patterns. During the outage, requests to affected applications would fail with errors when traffic was routed to affected instances. Application logs would have shown affected instances experienced high latency, error rates, or were not reachable from the Internet. This could have been caught by letting the live traffic testing run longer.

In order for live traffic testing to work properly, we need to improve our monitoring as well. In this case, having more points from which to do black box monitoring would have helped immensely. We are currently working on much broader monitoring for App Engine and will be integrating more extensive black box testing in upcoming quarters.

Once again, we’d like to point out that we could have done a much better job of communicating issues to all of you. While we strive to strike a balance between letting you know about major issues and not bothering you about the day-to-day operations; we clearly should have communicated this incident to you sooner. Rest assured you’ll be better informed of issues in the future.

Timeline

July 14, 2011 - 11:30 AM US/Pacific - The new Java execution environment is released to production.

July 14, 2011 - 5:00-6:00 PM US/Pacific - The previously scheduled Master/Slave read-only maintenance period occurred.

July 14, 2011 - 8:00-9:30 PM US/Pacific - Monitoring shows error rates and latency for Java applications using the Master/Slave datastore are slowly increasing across the entire system. Investigation reveals that the new Java execution environment is malfunctioning.

July 14, 2011 - 9:30 PM US/Pacific - Rollback of the Java execution environment to the previous version begins. Latency and error rates begin to fall.

July 14, 2011 - 11:30 PM US/Pacific - Rollback of the Java execution environment to the previous version completes. Java Master/Slave applications are functioning normally.

Remediation

• Faster notification on our status site and downtime-notify mailing list


• More live traffic stress tests for new releases


• Better black box monitoring to detect small impacts more quickly

[Edit] Clarification: no HR datastore apps were affected. Overall, the outage resulted in a 1.9% error rate, affecting approximately 0.005% of all App Engine traffic at peak.

Posted by Wesley Chun, Google App Engine Team