Byte Sized Rust

Chicago Rust Meetup
January 2020
Steve Hoffman - @bacoboy

Motivation...

https://xkcd.com/149/

Me: Run my code

Cloud: Okay

Less Stuff I Gotta Do...

https://blogs.oracle.com/developers/functions-as-a-service:-evolution,-use-cases,-and-getting-started

Idle Servers Cost $$$

Type vCPU Mem GB Hourly on demand Monthly on demand Reserved 3yr all upfront
t3.nano 2 0.5 $0.0052 $3.75 $1.41
t3.small 2 2 $0.0208 $14.98 $5.72
m5.large 2 8 $0.096 $69.12 $26.89
c5.large 2 4 $0.085 $61.20 $23.39
r5.large 2 16 $0.126 $90.72 $36.31
c5n.2xlarge 8 21 $0.432 $311.04 $119.67
c5n.18xlarge 72 192 $3.888 $2,799.36 $896.97 
https://aws.amazon.com/ec2/pricing/on-demand/

Deploying Faster...

What is a λ?

AWS Says...

https://aws.amazon.com/lambda/

A λ is a "handler" function YOU provide

 

that runs in an AWS container "somewhere"

so you focus on your code not plumbing

 

Python Handler

def my_handler(event, context):
    message = 'Hello {} {}!'.format(event['first_name'], 
                                    event['last_name'])  
    return { 
        'message' : message
    }

Reacts to Events...

(sync like HTTP or async like message on queue)

Function

Trigger

Does things to..

All The Clouds Have Them

Cheap!

In Theory...

Then $0.0000166667 GB/Second after that...

First million invocations free...

Then $0.20/million after that...

400,000 GB-Seconds free...

Plus

Plus

Other AWS Costs (Databases, Data Transfer...)

Mo RAM Mo $

https://aws.amazon.com/lambda/pricing/

Pricing is "complicated"...

https://medium.com/@zackbloom/serverless-pricing-and-costs-aws-lambda-and-lambda-edge-169bfb58db75

Just Lambda

Lambda + API GW

263 Billiable Line Items Per Region Just for Lambda before you add the "other stuff"

Node Handler

exports.myHandler = function(event, context, callback) {   
   callback(null, "some success message");
   // or 
   // callback("some error type"); 
}

Old Hacks

https://aws.amazon.com/lambda/faqs/
exports.myHandler = function(event, context, callback) {
  const execFile = require('child_process').execFile;
  execFile('./myprogram', (error, stdout, stderr) => {
    if (error) {
      callback(error);
    }
    callback(null, stdout);
  });
}

First Go apps would use this to run any
Amazon Linux compatible binary/shell script:

Go Handler

type MyEvent struct {
        Name string `json:"name"`
}

func HandleRequest(ctx context.Context, name MyEvent) (string, error) {
        return fmt.Sprintf("Hello %s!", name.Name ), nil
}

Eventually Go got official support:

Supported Languages

https://aws.amazon.com/lambda/faqs/

Cold Start

REPORT RequestId: 6f127cc4-c2d7-4422-9490-774092cf5042	Duration: 1.36 ms	Billed Duration: 100 ms	Memory Size: 128 MB	Max Memory Used: 35 MB	Init Duration: 28.56 ms	
REPORT RequestId: 6ad595b5-d679-42e2-b790-ab48811cf9cb	Duration: 0.87 ms	Billed Duration: 100 ms	Memory Size: 128 MB	Max Memory Used: 35 MB

First invocation add Startup Time

Additional runs don't incur overhead

Always logs to Cloudwatch

Each instance gets its own Log Stream in Cloudwatch Logs

https://aws.amazon.com/cloudwatch/pricing/

Don't be noisy, CWL are $$$$

First Try...

https://github.com/srijs/rust-aws-lambda

First Rust Lambdas were
pretending to be Go binaries

Became AWS Open Source Project

https://aws.amazon.com/blogs/opensource/rust-runtime-for-aws-lambda/

Events Survived...

https://github.com/LegNeato/aws-lambda-events

Building off the Golang SDK

type S3Bucket struct {
	Name          string         `json:"name"`
	OwnerIdentity S3UserIdentity `json:"ownerIdentity"`
	Arn           string         `json:"arn"`
}

#[derive(Debug, Clone, PartialEq, Deserialize, Serialize)]
pub struct S3Bucket {
    #[serde(deserialize_with = "deserialize_lambda_string")]
    #[serde(default)]
    pub name: Option<String>,
    #[serde(rename = "ownerIdentity")]
    pub owner_identity: S3UserIdentity,
    #[serde(deserialize_with = "deserialize_lambda_string")]
    #[serde(default)]
    pub arn: Option<String>,
}

https://github.com/aws/aws-lambda-go/tree/master/events
https://github.com/LegNeato/aws-lambda-events/tree/master/aws_lambda_events_codegen

Well Documented AWS APIs and Events

https://docs.aws.amazon.com/lambda/latest/dg/lambda-services.html
{
  "Records": [
    {
      "eventVersion": "2.1",
      "eventSource": "aws:s3",
      "awsRegion": "us-east-2",
      "eventTime": "2019-09-03T19:37:27.192Z",
      "eventName": "ObjectCreated:Put",
      "userIdentity": {
        "principalId": "AWS:AIDAINPONIXQXHT3IKHL2"
      },
      "requestParameters": {
        "sourceIPAddress": "205.255.255.255"
      },
      "responseElements": {
        "x-amz-request-id": "D82B88E5F771F645",
        "x-amz-id-2": "vlR7PnpV2Ce81l0PRw6jlUpck7Jo5ZsQjryTjKlc5aLWGVHPZLj5NeC6qMa0emYBDXOo6QBU0Wo="
      },
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "828aa6fc-f7b5-4305-8584-487c791949c1",
        "bucket": {
          "name": "lambda-artifacts-deafc19498e3f2df",
          "ownerIdentity": {
            "principalId": "A3I5XTEXAMAI3E"
          },
          "arn": "arn:aws:s3:::lambda-artifacts-deafc19498e3f2df"
        },
        "object": {
          "key": "b21b84d653bb07b05b1e6b33684dc11b",
          "size": 1305107,
          "eTag": "b21b84d653bb07b05b1e6b33684dc11b",
          "sequencer": "0C0F6F405D6ED209E1"
        }
      }
    }
  ]
}

https://www.rusoto.org/usage-and-example.html

AWS SDK in Rust

extern crate rusoto_core;
extern crate rusoto_dynamodb;

use std::default::Default;

use rusoto_core::Region;
use rusoto_dynamodb::{DynamoDb, DynamoDbClient, ListTablesInput};

fn main() {
  let client = DynamoDbClient::new(Region::UsEast1);
  let list_tables_input: ListTablesInput = Default::default();

  match client.list_tables(list_tables_input).sync() {
    Ok(output) => {
      match output.table_names {
        Some(table_name_list) => {
          println!("Tables in database:");

          for table_name in table_name_list {
            println!("{}", table_name);
          }
        }
        None => println!("No tables in database!"),
      }
    }
    Err(error) => {
      println!("Error: {:?}", error);
    }
  }
}

Client for service specific API

Build the Request

Call the API

Process the response

https://docs.rs/aws_lambda_events/0.2.5/aws_lambda_events/event/sqs/struct.SqsEvent.html

Sample SQS Event

{
  "Records": [
    {
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "receiptHandle": "MessageReceiptHandle",
      "body": "Hello from SQS!",
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000",
        "SenderId": "123456789012",
        "ApproximateFirstReceiveTimestamp": "1523232000001"
      },
      "messageAttributes": {},
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSource": "aws:sqs",
      "eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:MyQueue",
      "awsRegion": "us-east-1"
    }
  ]
}
#[derive(Debug, Clone, PartialEq, Deserialize, Serialize)]
pub struct SqsEvent {
    #[serde(rename = "Records")]
    pub records: Vec<SqsMessage>,
}

#[derive(Debug, Clone, PartialEq, Deserialize, Serialize)]
pub struct SqsMessage {
    #[serde(deserialize_with = "deserialize_lambda_string")]
    #[serde(default)]
    #[serde(rename = "messageId")]
    pub message_id: Option<String>,
    #[serde(deserialize_with = "deserialize_lambda_string")]
    #[serde(default)]
    #[serde(rename = "receiptHandle")]
    pub receipt_handle: Option<String>,
    #[serde(deserialize_with = "deserialize_lambda_string")]
    #[serde(default)]
    pub body: Option<String>,
    #[serde(deserialize_with = "deserialize_lambda_string")]
    #[serde(default)]
    #[serde(rename = "md5OfBody")]
    pub md5_of_body: Option<String>,
    #[serde(deserialize_with = "deserialize_lambda_string")]
    #[serde(default)]
    #[serde(rename = "md5OfMessageAttributes")]
    pub md5_of_message_attributes: Option<String>,
    #[serde(deserialize_with = "deserialize_lambda_map")]
    #[serde(default)]
    pub attributes: HashMap<String, String>,
    #[serde(deserialize_with = "deserialize_lambda_map")]
    #[serde(default)]
    #[serde(rename = "messageAttributes")]
    pub message_attributes: HashMap<String, SqsMessageAttribute>,
    #[serde(deserialize_with = "deserialize_lambda_string")]
    #[serde(default)]
    #[serde(rename = "eventSourceARN")]
    pub event_source_arn: Option<String>,
    #[serde(deserialize_with = "deserialize_lambda_string")]
    #[serde(default)]
    #[serde(rename = "eventSource")]
    pub event_source: Option<String>,
    #[serde(deserialize_with = "deserialize_lambda_string")]
    #[serde(default)]
    #[serde(rename = "awsRegion")]
    pub aws_region: Option<String>,
}

#[derive(Debug, Clone, PartialEq, Deserialize, Serialize)]
pub struct SqsMessageAttribute {
    #[serde(rename = "stringValue")]
    pub string_value: Option<String>,
    #[serde(rename = "binaryValue")]
    pub binary_value: Option<Base64Data>,
    #[serde(rename = "stringListValues")]
    pub string_list_values: Vec<String>,
    #[serde(rename = "binaryListValues")]
    pub binary_list_values: Vec<Base64Data>,
    #[serde(deserialize_with = "deserialize_lambda_string")]
    #[serde(default)]
    #[serde(rename = "dataType")]
    pub data_type: Option<String>,
}

https://docs.rs/aws_lambda_events/0.2.5/aws_lambda_events/event/apigw/struct.ApiGatewayProxyRequest.html

Even HTTP is an Event

{
  "body": "eyJ0ZXN0IjoiYm9keSJ9",
  "resource": "/{proxy+}",
  "path": "/path/to/resource",
  "httpMethod": "POST",
  "isBase64Encoded": true,
  "queryStringParameters": {
    "foo": "bar"
  },
  "multiValueQueryStringParameters": {
    "foo": [
      "bar"
    ]
  },
  "pathParameters": {
    "proxy": "/path/to/resource"
  },
  "stageVariables": {
    "baz": "qux"
  },
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "Accept-Encoding": "gzip, deflate, sdch",
    "Accept-Language": "en-US,en;q=0.8",
    "Cache-Control": "max-age=0",
    "CloudFront-Forwarded-Proto": "https",
    "CloudFront-Is-Desktop-Viewer": "true",
    "CloudFront-Is-Mobile-Viewer": "false",
    "CloudFront-Is-SmartTV-Viewer": "false",
    "CloudFront-Is-Tablet-Viewer": "false",
    "CloudFront-Viewer-Country": "US",
    "Host": "1234567890.execute-api.us-east-1.amazonaws.com",
    "Upgrade-Insecure-Requests": "1",
    "User-Agent": "Custom User Agent String",
    "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)",
    "X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==",
    "X-Forwarded-For": "127.0.0.1, 127.0.0.2",
    "X-Forwarded-Port": "443",
    "X-Forwarded-Proto": "https"
  },
  "multiValueHeaders": {
    "Accept": [
      "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
    ],
    "Accept-Encoding": [
      "gzip, deflate, sdch"
    ],
    "Accept-Language": [
      "en-US,en;q=0.8"
    ],
    "Cache-Control": [
      "max-age=0"
    ],
    "CloudFront-Forwarded-Proto": [
      "https"
    ],
    "CloudFront-Is-Desktop-Viewer": [
      "true"
    ],
    "CloudFront-Is-Mobile-Viewer": [
      "false"
    ],
    "CloudFront-Is-SmartTV-Viewer": [
      "false"
    ],
    "CloudFront-Is-Tablet-Viewer": [
      "false"
    ],
    "CloudFront-Viewer-Country": [
      "US"
    ],
    "Host": [
      "0123456789.execute-api.us-east-1.amazonaws.com"
    ],
    "Upgrade-Insecure-Requests": [
      "1"
    ],
    "User-Agent": [
      "Custom User Agent String"
    ],
    "Via": [
      "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)"
    ],
    "X-Amz-Cf-Id": [
      "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA=="
    ],
    "X-Forwarded-For": [
      "127.0.0.1, 127.0.0.2"
    ],
    "X-Forwarded-Port": [
      "443"
    ],
    "X-Forwarded-Proto": [
      "https"
    ]
  },
  "requestContext": {
    "accountId": "123456789012",
    "resourceId": "123456",
    "stage": "prod",
    "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef",
    "requestTime": "09/Apr/2015:12:34:56 +0000",
    "requestTimeEpoch": 1428582896000,
    "identity": {
      "cognitoIdentityPoolId": null,
      "accountId": null,
      "cognitoIdentityId": null,
      "caller": null,
      "accessKey": null,
      "sourceIp": "127.0.0.1",
      "cognitoAuthenticationType": null,
      "cognitoAuthenticationProvider": null,
      "userArn": null,
      "userAgent": "Custom User Agent String",
      "user": null
    },
    "path": "/prod/path/to/resource",
    "resourcePath": "/{proxy+}",
    "httpMethod": "POST",
    "apiId": "1234567890",
    "protocol": "HTTP/1.1"
  }
}
pub struct ApiGatewayProxyRequest {
    /// The resource path defined in API Gateway
    #[serde(deserialize_with = "deserialize_lambda_string")]
    #[serde(default)]
    pub resource: Option<String>,
    /// The url path for the caller
    #[serde(deserialize_with = "deserialize_lambda_string")]
    #[serde(default)]
    pub path: Option<String>,
    #[serde(deserialize_with = "deserialize_lambda_string")]
    #[serde(default)]
    #[serde(rename = "httpMethod")]
    pub http_method: Option<String>,
    #[serde(deserialize_with = "deserialize_lambda_map")]
    #[serde(default)]
    pub headers: HashMap<String, String>,
    #[serde(deserialize_with = "deserialize_lambda_map")]
    #[serde(default)]
    #[serde(rename = "multiValueHeaders")]
    pub multi_value_headers: HashMap<String, Vec<String>>,
    #[serde(deserialize_with = "deserialize_lambda_map")]
    #[serde(default)]
    #[serde(rename = "queryStringParameters")]
    pub query_string_parameters: HashMap<String, String>,
    #[serde(deserialize_with = "deserialize_lambda_map")]
    #[serde(default)]
    #[serde(rename = "multiValueQueryStringParameters")]
    pub multi_value_query_string_parameters: HashMap<String, Vec<String>>,
    #[serde(deserialize_with = "deserialize_lambda_map")]
    #[serde(default)]
    #[serde(rename = "pathParameters")]
    pub path_parameters: HashMap<String, String>,
    #[serde(deserialize_with = "deserialize_lambda_map")]
    #[serde(default)]
    #[serde(rename = "stageVariables")]
    pub stage_variables: HashMap<String, String>,
    #[serde(rename = "requestContext")]
    pub request_context: ApiGatewayProxyRequestContext,
    #[serde(deserialize_with = "deserialize_lambda_string")]
    #[serde(default)]
    pub body: Option<String>,
    #[serde(rename = "isBase64Encoded")]
    pub is_base64_encoded: Option<bool>,
}

Rust Handler

pub trait Handler<E, O> {
  fn run(&mut self, event: E, ctx: Context) -> 
  	Result<O, HandlerError>;
}

#[derive(Debug, Clone)]
pub struct HandlerError {
  msg: String,
  backtrace: Option<backtrace::Backtrace>,
}

Digging Deeper - SerDe!

pub type Handler<E, O> = fn(E, Context) -> Result<O, HandlerError>

You implement:

E  JSON that you want to provide to your Lambda function as input.
O It is the JSON representation of the object returned by the Lambda function. This is present only if the invocation type is RequestResponse.

Our Hander Function

#[derive(Deserialize, Clone)]
struct CustomEvent {
    #[serde(rename = "firstName")]
    first_name: String,
}

#[derive(Serialize, Clone)]
struct CustomOutput {
    message: String,
}

fn my_handler(e: CustomEvent, c: lambda::Context) -> Result<CustomOutput, HandlerError> {
    if e.first_name == "" {
        error!("Empty first name in request {}", c.aws_request_id);
        return Err(c.new_error("Empty first name"));
    }

    Ok(CustomOutput {
        message: format!("Hello, {}!", e.first_name),
    })
}

https://aws.amazon.com/blogs/opensource/rust-runtime-for-aws-lambda/

main starts runtime w/handler

#[macro_use]
extern crate lambda_runtime as lambda;
#[macro_use]
extern crate serde_derive;
#[macro_use]
extern crate log;
extern crate simple_logger;

use lambda::error::HandlerError;

use std::error::Error;

fn main() -> Result<(), Box<dyn Error>> {
    simple_logger::init_with_level(log::Level::Info)?;
    lambda!(my_handler);
    Ok(())
}

Stuff outside handler persists between runs on same lambda instance
(Think reusing connections or other "costly" things)

Cargo Changes

[dependencies]
lambda_runtime = "^0.1"
serde = "^1"
serde_json = "^1"
serde_derive = "^1"
log = "^0.4"
simple_logger = "^1"

[[bin]]
name = "bootstrap"
path = "src/main.rs"

Lambdas using "provided" runttimes must be named:

bootstrap

So what can you
build with this?

https://apievangelist.com/2017/11/06/a-simple-api-using-aws-rds-lambda-and-api-gateway/

Synchronous Application

Like a public API or marketing tracking pixel

https://github.com/aws-samples/lambda-refarch-webapp

More Complicated Version

https://github.com/aws-samples/lambda-refarch-fileprocessing

Asynchronous File Processing

https://github.com/aws-samples/lambda-refarch-streamprocessing

Asynchronous Stream Processing

Manual/SAM/Serverless

On a Mac cross
compile with Docker

$ docker pull clux/muslrust
$ docker run -v $PWD:/volume --rm -t clux/muslrust cargo build --release

$ zip -j hello-world.zip ./target/x86_64-unknown-linux-musl/release/bootstrap

Can be kinda slow...

On a Mac cross
compile directly

$ rustup target add x86_64-unknown-linux-musl
$ brew install filosottile/musl-cross/musl-cross
$ ln -s /usr/local/bin/x86_64-linux-musl-gcc /usr/local/bin/musl-gcc

$ cat .cargo/config
[target.x86_64-unknown-linux-musl]
linker = "x86_64-linux-musl-gcc"

$ cargo build --target=x86_64-unknown-linux-musl --release

$ file target/x86_64-unknown-linux-musl/release/bootstrap
.../bootstrap: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped

$ zip -j hello-world.zip ./target/x86_64-unknown-linux-musl/release/bootstrap

^--  Correct!

<--  Takes 30 min - get coffee!

https://aws.amazon.com/blogs/opensource/rust-runtime-for-aws-lambda/

Upload function
with AWS CLI

$ aws lambda create-function \
    --function-name hello-world \
    --runtime provided \
    --zip-file fileb://hello-world.zip \
    --handler NOTUSED \
    --role arn:aws:iam::123456789012:role/service-role/hello-world

Or deploy with fav tools

resource "aws_lambda_function" "hello-world" {
  function_name    = "hello-world"
  filename         = "hello-world.zip"
  source_code_hash = "${base64sha256(file("hello-world.zip"))}"
  runtime          = "provided"
  handler          = "NOTUSED"
  role             = aws_iam_role.hello-world.arn
}

Manually invoke

$ aws lambda invoke \
    --function-name hello-world \
    --payload '{"firstName":"Steve"}' response.json
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

$ cat response.json
{"message":"Hello, Steve!"}

Invoke in AWS Console

Must Grant Permissions
for Services to invoke

$ aws lambda add-permission \
    --function-name hello-world \
    --action lambda:InvokeFunction \
    --statement-id sqs \
    --principal sqs.amazonaws.com

$ aws lambda get-policy --function-name hello-world | jq -r '.Policy' | prettier --stdin --parser json
{
  "Version": "2012-10-17",
  "Id": "default",
  "Statement": [
    {
      "Sid": "sqs",
      "Effect": "Allow",
      "Principal": { "Service": "sqs.amazonaws.com" },
      "Action": "lambda:InvokeFunction",
      "Resource": "arn:aws:lambda:us-east-2:123456789012:function:hello-world"
    }
  ]
}

Default rules are nobody has permissions
to do anything - even AWS Services

Or deploy with fav tools

resource "aws_lambda_permission" "sqs-invokes-hello-world" {
  function_name = aws_lambda_function.hello-world.function_name
  action        = "lambda:InvokeFunction"
  statement_id  = "sqs"
  principal     = "sqs.amazonaws.com"
}

Serverless Application Model

https://aws.amazon.com/serverless/sam/
  • Command Line Tools
  • Processes YAML-ish files into CloudFormation Templates
  • Builds, packages, uploads to s3 for deployment, deploys via CloudFormation
  • AWS Only
  • Local Execution Environment via Docker
brew tap aws/tap
brew install aws-sam-cli

SAM Deploy Templates

https://aws.amazon.com/serverless/sam/
AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Hello World Function

Resources:
  HelloRustFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: sam-hello-world
      Handler: NOTUSED
      Runtime: provided
      CodeUri: hello-world.zip
      Role: arn:aws:iam::123456789012:role/service-role/hello-world

deploy-hello-world.yaml

SAM Packaging/S3

https://aws.amazon.com/serverless/sam/
sam package 
  --template-file deploy-hello-world.yaml \
  --s3-bucket chicago-rust-s3 \
  --output-template-file cf-deploy-hello-world.yaml

cf-deploy-hello-world.yaml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Hello World Function
Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: sam-hello-world
      Handler: NOTUSED
      Runtime: provided
      MemorySize: 128
      CodeUri: s3://chicago-rust-s3/cebcadfd6fbdd2b1bc570f18ec1b562c
      Role: arn:aws:iam::123456789012:role/service-role/hello-world

SAM Deploy Cloudformation

https://aws.amazon.com/serverless/sam/
sam deploy \
  --template-file cf-deploy-hello-world.yaml \
  --stack-name sam-hello-world \
  --capabilities CAPABILITY_NAMED_IAM

SAM Local Testing

https://aws.amazon.com/serverless/sam/
$ sam local generate-event sqs receive-message
{
  "Records": [
    {
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "receiptHandle": "MessageReceiptHandle",
      "body": "Hello from SQS!",
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000",
        "SenderId": "123456789012",
        "ApproximateFirstReceiveTimestamp": "1523232000001"
      },
      "messageAttributes": {},
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSource": "aws:sqs",
      "eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:MyQueue",
      "awsRegion": "us-east-1"
    }
  ]
}

SAM Local Testing

https://aws.amazon.com/serverless/sam/
Uses Docker lambci/lambda:provided container as runtime environment

Using Docker Directly

https://hub.docker.com/r/lambci/lambda/
$ docker run --rm \
  -v "$PWD/target/x86_64-unknown-linux-musl/release/":/var/task:ro,delegated \
  lambci/lambda:provided \
  handler \
  '{"firstName": "Steve"}'
START RequestId: a7ac181c-ded5-1b69-5f5e-a7f1f3d30c64 Version: $LATEST
2020-01-27 17:29:55,923 INFO  [lambda_runtime::runtime] Received new event with AWS request id: a7ac181c-ded5-1b69-5f5e-a7f1f3d30c64
2020-01-27 17:29:55,924 INFO  [lambda_runtime::runtime] Response for a7ac181c-ded5-1b69-5f5e-a7f1f3d30c64 accepted by Runtime API
END RequestId: a7ac181c-ded5-1b69-5f5e-a7f1f3d30c64
REPORT RequestId: a7ac181c-ded5-1b69-5f5e-a7f1f3d30c64	Init Duration: 59.08 ms	Duration: 4.01 ms	Billed Duration: 100 ms	Memory Size: 1536 MB	Max Memory Used: 9 MB

{"message":"Hello, Steve!"}

Can still use Docker environments to test if using other means of deployment (i.e. terraform, etc)

LocalStack

https://localstack.cloud/

Mock AWS Services for local development

(override API endpoints to point at local Docker containers)

Free and Pro Tiers

(Pro gets more services, tools, and supports a great project)

Can also be used to run your Lambdas

Serverless

https://serverless.com/
  • Command Line Tools
  • Processes (more) YAML-ish files into CloudFormation Templates - mixed syntax since cf functions can be used
  • Builds, packages, uploads to s3 for deployment
  • Multi-Cloud Support
  • Local Execution Environment via Docker
  • Paid PRO version includes:
    • dashboard
    • monitoring
    • alerts
    • ci/cd
    • rainbows
    • unicorns

Serverless vs SAM

https://sanderknape.com/2018/02/comparing-aws-sam-with-serverless-framework/

Check out Sander's blog post for great comparison

In the end its all just json/yaml representations

Other Resources To Read/Watch

Step Functions

https://aws.amazon.com/getting-started/tutorials/create-a-serverless-workflow-step-functions-lambda/

NEW: Lambda Destinations for simple flows

https://aws.amazon.com/blogs/compute/introducing-aws-lambda-destinations/

HTTP API Gateway

https://aws.amazon.com/blogs/compute/announcing-http-apis-for-amazon-api-gateway/

Cheaper Option

Fewer Dials to Set

https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-vs-rest.html

v1 vs v2:

AWS XRay Tracing

https://docs.aws.amazon.com/xray/latest/devguide/aws-xray.html

Lots of moving parts!

Know what's going on!

 

Can configure lambda to have xray - just use the SDK to send data

 

Or use some non-AWS alternative
(i.e. Espagon)

Map Reduce in λ

http://pywren.io/
  • S3 instead of HDFS
  • Lambdas instead of Servers

Big Data with Serverless

https://www.slideshare.net/AmazonWebServices/building-big-data-applications-with-serverless-architectures-june-2017-aws-online-tech-talks

Big Data on a Budget...

https://github.com/awslabs/lambda-refarch-mapreduce
|-----------------------|---------|---------|--------------|
| Technology            | Scan 1a | Scan 1b | Aggregate 2a | 
|-----------------------|---------|---------|--------------|
| Amazon Redshift (HDD) | 2.49    | 2.61    | 25.46        |
|-----------------------|---------|---------|--------------|
| Impala - Disk - 1.2.3 | 12.015  | 12.015  | 113.72       |
|-----------------------|---------|---------|--------------|
| Impala - Mem - 1.2.3  | 2.17    | 3.01    | 84.35        |
|-----------------------|---------|---------|--------------|
| Shark - Disk - 0.8.1  | 6.6     | 7       | 151.4        |
|-----------------------|---------|---------|--------------|
| Shark - Mem - 0.8.1   | 1.7     | 1.8     | 83.7         |
|-----------------------|---------|---------|--------------|
| Hive - 0.12 YARN      | 50.49   | 59.93   | 730.62       |
|-----------------------|---------|---------|--------------|
| Tez - 0.2.0           | 28.22   | 36.35   | 377.48       |
|-----------------------|---------|---------|--------------|
| Serverless MapReduce  | 39      | 47      | 200          |   
|-----------------------|---------|---------|--------------|

Serverless MapReduce Cost:

|---------|---------|--------------|
| Scan 1a | Scan 1b | Aggregate 2a | 
|---------|---------|--------------|
| 0.00477 | 0.0055  | 0.1129       |   
|---------|---------|--------------|

Honeycomb primary storage

https://www.honeycomb.io/blog/secondary-storage-to-just-storage/

re:Invent 2018 - SRV409

Lambda Deep Dive

More Isn't Always Better

Figure out the best bang for the buck -- don't guess, use data!

https://github.com/alexcasalboni/aws-lambda-power-tuning

And figure out your POST-free-tier costs before you go too far down this path
Google: lambda hidden costs

Thanks!

Questions?