AWS StepFunctions: Fine Tuning Serverless Workflows Using the Result Selector
AWS StepFunctions 🚀
StepFunctions is yet another service offered by Amazon Web Services that can be used to orchestrate serverless workflows with AWS Lambda. StepFunctions are particularly useful in highly distributed systems and micro-service architectures.
One feature that I personally love about stepFunctions is the visual flow of an activity type diagram that can be understood by all areas of the business, see below;
Introduction
In this example, I will be talking about how you can improve your workflows when using StepFunctions by customising the results from AWS Lambda at each step.
AWS recently released, in August 2020, a way to do this using the ResultSelector; read more about this release on the AWS blog here. 🎉
There is another alternative to using the result selector and getting around large data payloads; Writing the data to a file in S3. This option would give you the opportunity to store in S3 the specific data needed for the workload which could be read directly in a Lambda function and therefore mitigate the StepFunctions payload limit.
Firstly, I must admit that this article is not just for serverless applications and I hope that you can utilise this example in any AWS environment you may have. However, I personally believe it leans itself nicely towards a serverless solution for a number of reasons, I’ll explain.
Suppose you have a serverless application that consists of a set of AWS Lambda functions that are invoked by hitting various requests on AWS API Gateway or AWS AppSync endpoints. Wouldn’t it be nice to reuse the lambda functions you have already created and configured inside your StepFunction workloads.
In this example, we have a ‘Get Post’ Lambda function that retrieves the data from AWS DynamoDB and we can reuse this same Lambda function inside our ‘Report Post Metrics’ workload. Perhaps we want to fetch the meta data on the post, evaluate the data and release it to another area in the service.
Why Reuse the same Lambda? 🤓
There are many reasons why you would want to use the same lambda function inside the workload. For example, within a serverless application you have account limits that AWS enforce to guide you on best practices and design. One of these default limits currently sets the number of concurrent Lambdas running per account to 1000. This may seem a lot however, consider a large application with high throughput and lambdas scaling out.
Not only have you got such limits within your account, if you have multiple lambdas to retrieve the same data from the datasource you’ll end up with higher inefficiency and extra functions and code to maintain.
So, Why do we need the Result Selector? 💡
As we are reusing our ‘Get Post’ Lambda throughout the application where possible, let’s imagine that the object that the Lambda returns contains all the post details, meta data, authorization access and much more data. For the StepFunction we only need the postId and all or some of the meta data on the object. All of the other information is redundant for this use case.
Up until recently AWS StepFunctions had a payload limit of 32,768 characters, so if you were getting multiple posts, each returning the entire data object, you could quite easily hit this character limit and receive the following error:
ERROR: The state/task returned a result with a size exceeding the maximum number of characters service limit.
Fortunately, AWS release another recent update, in September 2020, to increase this limit to 256kb. Read more here. 🎉
Okay so, AWS have increased the limit… do we still need to worry about the data returned from the lambda?
Maybe not for a small number of posts, but if you’re running a workload for all your posts or another large data set you may still see the error. Plus, it is good practice not to be fetching or logging any data you don’t need.
Result Selector in action 💥
Our StepFunciton definition for the getPosts task could look like the following:
GetPost: { Type: 'Task', Resource: '${getPostLambdaArn}', Next: 'EvaluateMetricsStep', ResultPath: '$.postDetails', ...}
We can add the result selector to fetch the postViews metric and the postId and therefore removing any additional data that is not going to be used and would be redundant in our workflow.
GetPosts: { Type: 'Task', Resource: '${getPostsLambdaArn}', Next: 'EvaluateMetricsStep', ResultSelector: { 'postViews.$': '$.metrics.viewCount', 'postId.$': '$.id', }, ResultPath: '$.posts', ...}
Then if we inspect our $.posts result object we’ll now have only postViews and postId that can then be passed as parameters to our next step.
EvaluateMetricsStep: { Type: 'Task', Resource: '${EvaluateMetricsLambdaArn}', Next: 'GenerateReport', ResultPath: '$.metrics', Parameters: { 'viewCount.$': '$.posts.postViews', 'id.$': '$.posts.postId', }, ...}
And… we’re done! ✅
As mentioned above, we can also mitigate the payload size limit by using additional services such as S3 and reading the data from there within the Lambdas rather than passing information in as parameters. This is not always ideal, it is yet another service to manage, clear out data after it has been used, and would require additional Lambda function code to manage and provision.
Hopefully, the Result Selector can be an in-built StepFunction feature that you can use to tidy up and make your workloads more efficient and secure. I hope you’ve enjoyed reading this article, please let me know your opinions and thoughts. If the post and Result Selector has come in useful for your workload it would be great to hear from you!
Find more information on the links used in this post below:
Happy building!… 🚀
Thank you for reading, Robert.
Reach me on linkedIn here: https://www.linkedin.com/in/robertbulmer/
