EsheleD Marketing & Technology

Google Cloud SQL: Your database in the cloud

Cross-posted from the Google Code Blog

One of App Engine’s most requested features has been a simple way to develop traditional database-driven applications. In response to your feedback, we’re happy to announce the limited preview of Google Cloud SQL.

You can now choose to power your App Engine applications with a familiar relational database in a fully-managed cloud environment. This allows you to focus on developing your applications and services, free from the chores of managing, maintaining and administering relational databases.

Google Cloud SQL brings many benefits to the App Engine community:

• No maintenance or administration - we manage the database for you.
• High reliability and availability - your data is replicated synchronously to multiple data centers. Machine, rack and data center failures are handled automatically to minimize end-user impact.
• Familiar MySQL database environment with JDBC support (for Java-based App Engine applications) and DB-API support (for Python-based App Engine applications).
• Comprehensive user interface for administering databases.
• Simple and powerful integration with Google App Engine.

The service includes database import and export functionality, so you can move your existing MySQL databases to the cloud and use them with App Engine.

Cloud SQL is available free of charge for now, and we will publish pricing at least 30 days before charging for it. The service will continue to evolve as we work out the kinks during the preview, but let us know if you’d like to take it for a spin.

Posted by Navneet Joneja, Product Manager for Google Cloud SQL

Project WOW

WOW - Showing Live Temperature Data Across Europe

Fall 2011 App Engine events

Pictarine: pictures in the cloud

Pictarine is a photo management web application, launched in 2010, that allows people to easily manage and share all of their photos from Flickr, Picasa, Facebook, Twitter and other sites. Pictarine developers Guillaume Martin and Maxime Rafalimanana have contributed the following post discussing their experiences using Google App Engine and Google Web Toolkit.

From the start, we used Google technologies in developing Pictarine and we wanted to share our experience with them so far. In this post, we will shed some light on the weaknesses and strengths we found in Google Web Toolkit (GWT) and Google App Engine. We will also discuss how we leveraged GWT to build a new technology that allows Pictarine to seamlessly display photos from the computer directly into the browser. The following diagram is an overview of how our application works.

Building a mashup in the cloud with Google App Engine

The Pictarine team is made of a web designer and two developers who previously worked mainly with Java based enterprise technologies and had a little experience with web technologies. When we started the project in early 2009, we were quite open on learning new languages like Python or Ruby, but when App Engine announced that Java would be supported, we were really excited to give Google App Engine a try.

The first few months, learning about the App Engine environment was quite easy and dare I say fun. Testing our code on Google’s servers from Eclipse IDE was only one click away. So we built our first prototype fast and we quickly decided to adopt App Engine. Then we started to build the core of our application: the engine that uses the API from Flickr, Picasa, Facebook to fetch the users’ photos. This is where we hit the first limitations of App Engine. Most users have a lot of photos on these services and retrieving them can take some time. But App Engine has strict limits on how long a request should last: an outgoing HTTP request cannot last more than 10 seconds and cannot process a request for more than 30 seconds. So while building our architecture we found ourselves writing approximately one third of our code dealing with these limitations: paginating our requests, creating background tasks to store data in small batches, etc.

In early 2010, when we launched our alpha version, everything went smoothly. We had some good press coverage and App Engine met our expectations in handling our first users. During 2010, we worked on implementing new features requested by our users, and during this period of time we were really impressed by the way App Engine evolved. Many of the limitations were lifted and great new features were added. We are now able to use Task Queues for requests that last up to 10 minutes, which we fully use to sync our users’ photos and albums. One of the features we like the most is the Channel API, a push notification system that allows us to instantly show a photo in every connected browser as soon as it is uploaded.
App Engine is still not perfect but has greatly improved and when we see its roadmap, we are quite confident it will continue to improve.

Building a fresh photo experience with Google Web Toolkit 

When we started Pictarine, we wanted a fast, distraction free interface that would allow our users to focus on their photos. We wanted the interface to adapt to the screen resolution, displaying a lot of photos on large screens and fewer on small ones. We wanted it to be automatically updated when new comments or new photos are added. We wanted a web application. As we eliminated Flash quite quickly (based on our user experience...) we started to look at tools to build HTML/CSS/Javascript applications. We settled quickly on GWT: while coding in Java, with all the tools we are used to (even the debugger), we could produce optimized Javacript that would run in every browser! When we started with GWT, it was already 3 years old, so we had few complaints about it. The main issue was that we had to always keep in mind that the Java code we produced was ultimately translated to Javascript. So some Java methods, such as the Reflection API, are not allowed. Another thing that was not obvious to us when we started with GWT was that a java developer needs an intimate knowledge of HTML/CSS if he/she wants to go beyond the basic user interface provided by the GWT widgets.

What we really like about GWT in our architecture is the ability to share code between client and server: we can use the same Photo or Album class on the client and the server and the GWT RPC system allows us to automatically share the same Java object on both side. We can also have the same data validation code on both sides: we can alert the user immediately on errors and still validate the data on the server just in case.

Another great feature we like about GWT is its handling of internationalisation. From the beginning we wanted to build a website available for all Internet users, so supporting English as well as our native language (French) was almost obligatory. Fortunately, GWT makes it really easy to generate centralized localization files so that we just have to translate.

Finally, to illustrate how great Javascript generation is, when IE9 came out, we waited a few weeks for GWT to support it and our application was compatible after a recompile! Of course, the IE9 team also did a good job with their HTML5/CSS3 engine.

Building an universal uploader 

After the launch of our alpha in 2010, our users were able to see and share their photos from Flickr, Picasa, Facebook. But they still had to put their photos on these websites first before coming to Pictarine. This limitation quickly became the first request on our feedback system. We needed to let our users do everything from Pictarine, including uploading photos. Uploading many photos from a website is still not a trivial process. Most websites choose Flash to allow users to upload multiple files at once, but our experience with it was that it often crashed after a while. Some use Java applets, but they are never well integrated and always look odd. At Pictarine we decided to tackle this problem by using Java Applet for their stability across all platforms but without using it to render photos or folders.

We have built a technology that uses the GWT RPC mechanism to talk to a Java Applet: photos, upload progression are rendered in HTML/CSS and the applet takes care of photos resizing and uploading. Sharing a photo from a camera is now a one-step process. This technology also allows users to browse their local files directly in their browser and it is fully integrated in our design.

We believe that this new use of Java applets can help blur the line between the Desktop and the Cloud by seamlessly integrating desktop files in any web application.

In conclusion, we can say that we are really happy with the choices we made with App Engine and GWT. App Engine is a great service that perfectly handled the spike in traffic we saw right after articles on Mashable and Lifehacker were published. So we recommend it to every lean startup out there who loves developing in Java, Python or Go.

Posted by The App Engine Team

App Engine 1.5.4 SDK Release

It’s been a busy four weeks for us (you may have heard), but we are still on track with our new monthly release schedule. Today we have new SDKs for you, with some new features as well as some bug fixes.

Overall Changes

• Blobstore API - We’ve introduced an option to specify a limit on your blob upload size. This feature will allow you to expose blob uploads to your users, while still being able to ensure that their uploads won't exceed your preferred limits.
• Datastore Query Improvements - We’re also continuing our theme from the past few releases to tune our Datastore query planner to give users more flexibility. Starting with 1.5.4, queries with equality filters on multiple properties will now continue scanning up to the 30 second Datastore query deadline. Many of these queries that used to generate an error due to inefficient indexes will now succeed.
• Datastore Write Ops in the SDK - In response to the feedback from our recent Side By Side Bills release, we are now displaying the number of write ops needed to store an entity in the SDK dataviewer. Write ops include both the entity write as well as the index writes that are executed when an entity is added. You can always reduce the number of write ops by turning your indexed properties into unindexed properties, just make sure you don’t reference those properties in any of your queries! (Java, Python).

Java

• Prospective Search API - We’ve released the experimental Java version of our Prospective Search API. Prospective Search lets you detect and take action on datastore entities that match certain criteria when they are written.

Python

• Memcache - You can now make calls asynchronously with the Memcache API. With Asynchronous Memcache, your application does not need to block on calls to the Memcache API and can continue processing a request instead of waiting for a response from Memcache. Fear not Java friends, we’ll have matching functionality for you in an upcoming release.

Full release notes can be found in the usual location (Java, Python), and we’re always listening for your feedback in the groups. Happy coding!

Posted by The App Engine Team

A few adjustments to App Engine’s upcoming pricing changes

Last week we rolled out side-by-side billing to give you a more detailed preview of how you’ll be affected by the price changes that we announced in May. We received a variety of feedback and have made a few important changes based on it. Our intent is to be as open and transparent about the changes as possible and to give you enough time to prepare. In that spirit, our Engineering Director has also shared some of his personal thoughts.

We understand that the new rates surprised some of you. We’ve been listening closely to your feedback, and we wanted to share an update on the changes we’re making to help ensure you have an accurate picture of how the new pricing will affect your app. Although prices will increase, we’re confident that you’ll find App Engine still provides great value.

Based on your feedback we’re taking the following steps:

• Extended review period: We’re now giving you almost eight weeks before introducing the new pricing. You now have until November 1 to configure and tune your application to manage your costs. 
• Increased free Instance Hours: We are increasing the number of free Instance Hours from 24 to 28. This means that people who are using a free app to try out App Engine can run a single instance all day with a few spikes and still remain below our free quota.  This will be reflected in the comparison bills soon.
• Extended discount: We’ll continue to offer the 50% discount on instance prices until December 1st, at which time Python 2.7 should be available. Python 2.7 will include support for concurrent requests, which could further lower your costs. 
• Faster Usage Reports: We appreciate the importance of quickly being able to see the effect your tuning has on your bill and starting today we’ll provide your Usage Report (and the included comparison bills) within one day instead of the previous three. 
• Better analysis tools: We are working on better ways for you to model the cost of your apps. We will add the “billing” line into the instances graph on the Admin Console. We’re adding datastore billing information into the dev console to making it easier for you to track how the changes you make affect your bill, which should also help lower the cost. 
• Premier accounts: we know a lot of our customers are eagerly awaiting Premier accounts to get operational support, offline billing, unlimited accounts, and the SLA. So we will not wait until November 1st for this, but rather launch Premier accounts as soon as possible.  If you are interested in a Premier account, please contact us at appengine_premier_requests@google.com.

We also wanted to share some of the main ways for you to lower your bill and get a better sense of the true cost of App Engine:

• Set Max Idle instances: Setting Max Idle Instances to a lower level will help lower your costs as we will only charge for idle instances up to the maximum you set. This could impact your performance so it’s worth reading up on the ramifications. 
• Always-On reflected in bills: Currently the side-by-side bills still include the cost of always-on even though it will be retired when the new pricing launches (to be replaced by min idle instances). We’re working on a fix for this. Until then you can comfortably subtract 48 instance hours per day from the estimate. 
• Reserved instance hours: The simplest way to lower the charge for instance hours is to consider using reserved instance hours. They are 37.5% cheaper than on-demand, but you do need to commit to a certain number of them over the course of a week. 
• Managing resources: Check out this article, which provides more helpful advice on how to efficiently manage your resources and lower costs. 

We launched App Engine in preview three years ago to make it easier for you to build, maintain and scale web applications. Since then, we’ve added many features like Java and Go language support, created a High Replication Datastore, and added many other APIs. And there’s a lot more cool stuff to come. It’s heartening to us that so many developers- 150,000 and counting- have chosen to use App Engine to run their apps. While we can’t continue to offer App Engine at our original prices, we can commit to listening to your feedback, acting on it, and working hard to give you a great platform for your apps at the most competitive price possible. If you find that’s not the case, or have any questions about this at all, please feel free to contact us at appengine_updated_pricing@google.com.

The App Engine Team

$50 Credit for new billing signups and budget changes

App Engine 1.5.3 SDK Released

We’re pleased to announce another App Engine release today. You might have noticed that the rate of releases has gone up slightly in the past few months. We’ve made some changes internally so we are looking to push out a new release every month. This month includes a few Datastore updates, some changes to Blobstore API and Memcache API, and finally a new feature for Java developers.

Python and Java Changes

• Blobstore API - We’ve removed the limits on the size of blob uploads. You can now upload files of any size, allowing your app to serve images, video, or anything your internet connection can handle.

Datastore Changes

• Index retrieval - We’ve added the ability for you to programmatically retrieve the list of indexes you’ve currently defined in the datastore, as well as their statuses.
• Datastore Admin - You can now enable the Datastore Admin function from the Admin Console. This will allow Java users to make use of this functionality, like deleting all entities of a certain kind, without having to upload a Python version of their application. And for Python developers, you no longer need to enable this in your app.yaml file.
• HRD Migration Trusted Testers - We are seeking early adopters to try out an improved HRD migration tool that requires a read-only period relative to your datastore write rate (as opposed to your datastore size, which is how the current version behaves). Please see the release notes for more information.

Python Updates

• Memcache API - We now support the CAS (compare-and-swap) operation in our Python Memcache API (Java already had it). This can be used to update a value in Memcache only if no other requests have updated it between when the value was retrieved and when you go to update it.

Java Updates

• Download app - Using the AppCfg download_app command, you can download any files that were uploaded from your war directory when you last updated the app version.

This release also contains small updates and bugfixes for both Python and Java so be sure to check out the full release notes. Feedback, discussion, and questions can be posted in our Google Group.
Posted by The App Engine Team

Google App Engine receives SAS70 and SSAE-16 certification

As we get ready for App Engine to leave preview release later this year, we want to make clear that in addition to improving our reliability, adding new features, and introducing developer support, we are also committed to demonstrating we have sufficient controls in place to safeguard the data you store with us.

Today we are happy to announce that over the past few weeks Google App Engine successfully completed the audit process for the SAS70 Type II, SSAE 16 Type II, and ISAE 3402 Type II standards.

You can read more about the Google’s certifications and commitment to protect user data on the Enterprise blog.

Posted by The App Engine Team

App Engine 1.5.2 SDK Released

As the summer heat descends on the Northern Hemisphere we thought we’d release our newest App Engine version with some changes that are sure to keep you playing around in the cool, air-conditioned indoors (hey, you don’t want your computer to overheat, right?).

Production Changes

• Adjustable Scheduler Parameters - As we previously discussed, we are introducing two scheduler knobs (okay, they actually look like sliders) that will allow you to control some of the parameters that influence how many Instances run your application. Today you will be able to set the minimum pending latency and maximum number of idle instances for your application.

Datastore Changes

• Advanced Query Planning - We are removing the need for exploding indexes and reducing the custom index requirements for many queries. The SDK will suggest better indexes in several cases and an upcoming article will describe what further optimizations are possible.
• Namespaced Datastore Stats - Now, in addition to getting overall datastore stats, we are providing a new option to query datastore stats per namespace.

Task Queue Changes

• New Task Queue details page - We’ve revamped the Task Queue details page in the Administration Console to provide more information about the tasks being run. You can now see the headers included in the enqueued task, the payload, and information from previous task runs.
• 1MB Pull Task Size - It’s our belief that there is only one way for size limits to go - and that’s up! So with this release we’ve increased the size for pull tasks to 1MB.
• Pull queue lease modification - We’ve introduced a new method for Pull Queues that allows you to extend the lease on existing tasks if the initial lease on the task was insufficient.

Lastly, we have some exciting news related to the experimental Go runtime. While it still remains experimental, starting with 1.5.2, all HRD apps will have access to the Go runtime in production.

As always, there are also some small features and bug fixes, the full list of which can be found in our release notes (Python, Java). We look forward to your feedback and questions in our forum.

Posted by The App Engine Team