Cloud Computing: AWS Adds Simple Workflow Service to Its Wares

It says it’s used internally for such things as MapReduce analytics for operational decision-making and managing user-facing content such as web pages, videos and Kindle books.

According to AWS CTO Werner Vogels:

"A growing number of applications are relying on asynchronous and distributed processing, with scalability of the application as the primary motivation. By designing autonomous distributed components, developers get the flexibility to deploy and scale out parts of the application independently as load increases. The asynchronous and distributed model has the benefits of loose coupling and selective scalability, but it also creates new challenges. Application developers must coordinate multiple distributed components to get the desired results. They must deal with the increased latency and unreliability inherent in remote communication. Components may take extended periods of time to complete tasks, requests may fail and errors originating from remote systems must be handled. Today, to accomplish this, developers are forced to write complicated infrastructure that typically involves message queues and databases along with complex logic to synchronize them. All this ‘plumbing’ is extraneous to business logic and makes the application code unnecessarily complicated and hard to maintain.

"Amazon SWF enables applications to be built by orchestrating tasks coordinated by a decider process. Tasks represent logical units of work and are performed by application components that can take any form, including executable code, scripts, web service calls, and human actions. Developers have full control over implementing and orchestrating tasks, without worrying about underlying complexities such as tracking their progress and keeping their state.

"Developers implement workers to perform tasks. They run their workers either on cloud infrastructure, such as Amazon EC2, or on-premise. Tasks can be long-running, may fail, may timeout and may complete with varying throughputs and latencies. Amazon SWF stores tasks for workers, assigns them when workers are ready, tracks their progress, and keeps their latest state, including details on their completion. To orchestrate tasks, developers write programs that get the latest state of tasks from Amazon SWF and use it to initiate subsequent tasks in an ongoing manner. Amazon SWF maintains an application’s execution state durably so that the application can be resilient to failures in individual application components."

Here’s a picture of how it works. Note that the execution of each step is visible through a Management Console:

SWF retains a history of executions for up to 90 days.

Although still a beta, Amazon says NASA Jet Propulsion Lab (JPL) uses it as part of several space and earth science missions including the Mars Exploration Rover. The lab has integrated SWF into its Cloud Oriented Architecture and reference implementation, Polyphony.

RightScale CTO Thorsten von Eicken said, "Using Amazon SWF, we are able to reduce the time-to-market for our higher-level infrastructure automation features. We are able to focus on our value-add without having to worry about the challenges that are associated with implementing a distributed workflow engine. In the end we are able to ship new features faster and don’t have to concern ourselves with maintaining that engine."

At present SWF is only available in Amazon’s US Eastern Region (three availability zones).

A thousand workflow executions, 10,000 tasks, timers, signals and markers and 30,000 workflow days can be had for free. Otherwise it’ll cost $1 for starting 10,000 executions, fractional costs for as long as an execution remains open and for as long as Amazon retains a completed execution up to 90 days, and 25 cents per 10,000 for scheduling a task, starting a timer, sending a signal or recording a marker.

There are SDKs for Java, Ruby, .NET and PHP. The Java SDK automatically turns Java code into tasks.

See www.aws.amazon.com/swf.