Introduction

In today’s post, we’re going to be doing the fun part of putting everything together to get our project into actions. We’ll be uploading our code to S3, creating our Lambda function, and then creating an event subscription that will trigger the function when we’ve uploaded an mp4 file to the bucket being used. As well already are aware, this should then begin processing of the file and create a transcription using Transcribe.

Uploading our Code to S3

When creating a Lambda function that runs Go code, we need to provide a zipped file of the Go code. This can either be carried out via an upload of the file on your development system, or alternatively by uploading the zip file to S3 and providing the location information.

Of the two options, the latter is the most flexible for us since we can update our code and upload a new zip file to this location without needing to change any configuration.

Note that I’m doing these steps on OSX and using Bash. For Windows systems you may need to use slightly different context.

Ensure you’re using the current Github release

Change the current directory to ./src/transcribe within the project folder.

cd ./src/transcribe

The Lambda code runs on Linux, so we need to ensure when compiling the code that the compiler knows this (via GOOS=linux). We also specify the output file to be main

GOOS=linux go build -o main

zip main.zip main

Uploading the File

../upload/upload -bucket "aws-lambda-go-transcribe2srt" -filename main.zip

Now, if you login to the AWS console, and take a look at the S3 bucket, main.zip will be there. Click on the file to get its properties and copy the URL under Link into the clipboard. We’re going to be using this in the next step.

Creating the Lambda Function

From the AWS console:

Click Services on the black bar, and then Lambda

Click Create Function

Now, in the Author from scratch section, enter the following values.

• Name : transcribe
• Runtime: Go 1.x
• Role: Create new role from template(s)
• Role name: transcribe_role

Click Create function

The Designer window appears, featuring the transcribe function, and with a role already defined to allow access to Cloudwatch.

Next, we tell Lambda where to get the code.

In the Function code section, select:

• Code entry type : Upload a file from Amazon S3
• Runtime: Go 1.x
• S3 link URL : <paste the link that you copied into the clipboard  in the previous steps>
• Hander : main

Create an S3 Trigger

Now that the basic function is in place, we need to configure it for our specific needs. its As previously mentioned, we want it to run when a new .mp4 file is created in our S3 bucket.

On the left hand side, click S3

This will add it onto the console, and a Configure triggers dialog will appear.

Change Suffix to .mp4 and select Add

Click Save

Give the Lambda function access to Transcribe

We need to give transcribe_role permissions to access amazon transcribe in addition to S3 and Cloudwatch

• Click Services, IAM
• Click Roles
• Click transcribe_role

• Click Attach policies
• Find and put a check next to AmazonTranscribeFullAccess
• Attach Policy

With this complete, the policy is now attached to the role.

A return to the Lambda function will also now show Amazon Transcribe on the right hand side, indicating it has permission to access this service.

Upload our movie

We’re ready to test the functionality out! Let’s find an mp4 video file and upload it. In my case, I’ve a file, movie.mp4, which is going to be used.

As before, you can choose either to manually upload the file via the AWS console, via the AWS CLI, or using the Upload program we created earlier.

../upload/upload -bucket "aws-lambda-go-transcribe2srt" -filename /movie.mp4

The Results

The function should kick in pretty much as soon as the file has finished copying to S3. Let’s have a look from the console by going to Machine Learning, Amazon Transcribe.

And it’s there. We’ve now got an end-to-end mp4 to transcript file.

Conclusion

In this post, we’ve created our Go package, uploaded it to S3, setup the Lambda function, made an event subscription for when an MP4 file is uploaded to our bucket, configured the role associated with the function to allow it to use Transcribe, and verified its operation.

At this point there are further steps we could think of.

• During the time of putting this series of blogs together, AWS added CloudWatch events for Transcribe. We could write another Lambda function, which ran once either a Transcribe completed or failed event occurred. Using this, we could do things like notify us when a job has completed, or even do something like converting the output to .srt format.
• Add an endpoint and some code to allow us to query the status of one of the jobs.
• We could even look into using a completely different way of getting a file into S3, such as passing a link to an MP4 file on a website and getting the Lambda function to download the file and store directly in S3 prior to creating the job.
• Via the above, we could also look at adding in additional event sources, such as via API Gateway.

There’s lots and lots of possibilities, and a forthcoming blog series will cover one or more of these.

thanks for reading! Feedback always welcome. 🙂

cheersy,

Tim

Introduction

In today’s post, we’re going to be looking at the code within the handler’s function. As part of this, we’ll be covering using structs and JSON together, logging to CloudWatch, marshaling, processing of S3 event data, and how to start a Transcribe job. It’s a bit of a longer post today as we’ll go through the entire code in the function.

The Story So Far…

At this point, our handler has been triggered by a file being placed in our S3 bucket to which there is an event subscription for CreateObject (more about this in the next blog). We’ve received the event information, which is placed in our variable S3Event, a struct. We have the information we need for further processing of S3Event and can proceed immediately with it. However, it’s worthwhile spending a couple of minutes looking at how Go processes the information received to place it into the variable.

A Bit About JSON & Go

Go does not have a native parsing mechanism for JSON data that allows dynamic generation of a struct based on the content (think PowerShell’s ConvertFrom-JSON cmdlet for example). Instead, (unless you want to go into the murky world of reflection and creating maps) you are expected to have some degree of awareness of the schema of data being received. Go still handles the conversion process, but it looks to you for information on how to map content. This is done in struct definitions simply by indicating the location of the data to map to, of the format ““`json:”location”““`.

In our situation, the definition of an S3Event for example (an array of type S3EventRecord), maps to the “Records” section of the event data (see below). When information is nested (i.e. a subsection of another), we just make sure our struct matches this. An example of this is the EventVersion string, contained in the S3EventRecord struct, which is mapped to “eventVersion” in the JSON file.

An extract of the struct configuration and JSON data is below. You can examine the complete definition of an S3Event from the aws-lambda-go sdk, here:

Here’s the first two levels of our struct…

type S3Event struct {
    Records []S3EventRecord `json:"Records"`
}

type S3EventRecord struct {
    EventVersion      string              `json:"eventVersion"`
    EventSource       string              `json:"eventSource"`
    AWSRegion         string              `json:"awsRegion"`
    EventTime         time.Time           `json:"eventTime"`
    EventName         string              `json:"eventName"`
    PrincipalID       S3UserIdentity      `json:"userIdentity"`
    RequestParameters S3RequestParameters `json:"requestParameters"`
    ResponseElements  map[string]string   `json:"responseElements"`
    S3                S3Entity            `json:"s3"`
}

…and an extract from the S3Data JSON

    "Records": [
        {
            "eventVersion": "2.0",
            "eventSource": "aws:s3",
            "awsRegion": "eu-west-1",
            "eventTime": "2018-07-10T05:26:14.8Z",
            "eventName": "ObjectCreated:Put",
            "userIdentity": {
                "principalId": "AWS:xxxxxx:xxxx@xxxx.nl"
            },
        }
    ]

Notice how they match up. The parser uses this information to populate the struct properties.

One aspect that is quite nice about the mapping process is that providing the top level outline structure matches the data being received, the entire substructure does not need to be present. You have complete control over which properties should be mandatory or not.

Naturally, it makes sense to have your struct defined to represent the “full” content schema of the JSON data, but content being received into this struct does not need to be as complete. This can help when JSON content may contain less fields, yet still come from the same event source.

If you’d like to spend some more time looking how a Go struct can be designed from a JSON source, I’d recommend taking a look at:

https://mholt.github.io/json-to-go/

Code Description

Let’s move onto the code itself now. Here’s our entire function as a reminder, before we burrow down into what it’s doing.

package main

func Handler(ctx context.Context, s3Event events.S3Event) {
	data, _ := json.Marshal(s3Event)
	streventinfo := string(data)
	fmt.Printf("S3 Event : %s\n", streventinfo)
	for _, record := range s3Event.Records {
		s3 := record.S3
		fmt.Printf("Object : %s\n", s3.Object.Key)

		sess, _ := session.NewSessionWithOptions(session.Options{
			Config:  aws.Config{Region: aws.String("eu-west-1")},
			Profile: "development",
		})

		log.Printf("Opening Transcribe session\n")
		transcriber := transcribeservice.New(sess)

		if transcriber == nil {
			log.Printf("Unable to create Transcribe session\n")
			return
		} else {
			log.Printf("Transcribe session successfully created\n")
		}

		jobname := GUID()
		mediafileuri := fmt.Sprintf("https://s3-eu-west-1.amazonaws.com/%s/%s", s3.Bucket.Name, s3.Object.Key)
		log.Printf("Job name :  %s\nMediaFileUri : %s\n", jobname, mediafileuri)
		mediaformat := "mp4"
		languagecode := "en-US"

		log.Printf("Creating transcription job\n")
		var StrucMedia transcribeservice.Media
		StrucMedia.MediaFileUri = &mediafileuri

		transcriber.StartTranscriptionJob(&transcribeservice.StartTranscriptionJobInput{
			TranscriptionJobName: &jobname,
			Media:                &StrucMedia,
			MediaFormat:          &mediaformat,
			LanguageCode:         &languagecode,
		})
	}
	log.Printf("Complete")
}

As mentioned previously, we’re going to use Cloudwatch for logging, using the aptly named log package. The first logging we’ll do is that of the S3Event data.

Initially, we need to marshal the data. Marshaling takes an interface (our struct in this case) and returns a JSON encoding of it.

data, _ = json.Marshal(s3Event)

With this done, we use the string function, which in turn converts our byte data to a string.

streventinfo = string(data)

We then log this information to CloudWatch. A nice touch of CloudWatch is that it picks up that the string data is JSON and formats it nicely for us. You’ll see this firsthand in the final part of this series.

log.Printf("S3 Event : %s\n", streventinfo)

We then need to iterate through each record entry in the event data. We use for to do this, assigning the record variable on each interaction from s3Event.Records. We do not need to set an initial value in the declaration, hence the “_”.

for _, record := range s3Event.Records {...}

Inside the loop, we set s3 to the value of the s3 branch, and from this we log the key referred to in the event. We will use this later as a parameter for our Transcribe job.

s3 := record.S3
log.Printf("Object : %s\n", s3.Object.Key)

Any time we want to perform operations with another AWS service, a session needs to be created. We define the parameters that the session will use (a region of eu-west-1, and using my own development profile), then create it using transcribeservice.New, a function from the github.com/aws/aws-sdk-go/service/transcribeservice package.

sess, _ := session.NewSessionWithOptions(session.Options{
Config:  aws.Config{Region: aws.String("eu-west-1")},
Profile: "development",
})
log.Printf("Opening Transcribe session\n")
transcriber := transcribeservice.New(sess)

Next, a check is made to ensure that a successful session was established. This is easily verified by ensuring that a non nil value was returned to our variable, transcriber. If a nil value was returned, we exit the function. We log the result irrespective to CloudWatch.

if transcriber == nil {
log.Printf("Unable to create Transcribe session\n")
return
} else {
log.Printf("Transcribe session successfully created\n")
}

Now we want to get our parameters set before starting a transcription job.

• A random job name needs be created, so the GUID function we created earlier is used to populate the variable jobname.
• We set mediafileuri, using string expansion with the the bucket name and key name that we got from the S3EventData
• Mediaformat is set to mp4
• Lastly, we set a language code of en-US for the languagecode variable.

jobname := GUID()
mediafileuri := fmt.Sprintf("https://s3-eu-west-1.amazonaws.com/%s/%s", s3.Bucket.Name, s3.Object.Key)
log.Printf("Job name :  %s\nMediaFileUri : %s\n", jobname, mediafileuri)
mediaformat := "mp4"
languagecode := "en-US"
log.Printf("Creating transcription job\n")

We define StrucMedia, which is of type transcribeservice.Media. One thing to be mentioned here is that we pass in a pointer to mediafileuri, not a string. This is because the MediaFileUri definition in the transcribeservice.Media struct specifies via *string that it expects to receive a pointer.

MediaFileUri *string `min:"1" type:"string"`

As such, our definition is as below.

var StrucMedia transcribeservice.Media
StrucMedia.MediaFileUri = &mediafileuri

Then, we invoke the StartTranscriptionJob function. This takes as its parameter a pointer to a StartTranscriptionJobInput struct, whose properties we set within it.  Lastly, a completion message is logged to CloudWatch.

transcriber.StartTranscriptionJob(&transcribeservice.StartTranscriptionJobInput{

TranscriptionJobName: &jobname,
Media:                &StrucMedia,
MediaFormat:          &mediaformat,
LanguageCode:         &languagecode,
})

Conclusion

In this post, we’ve covered the code within our lambda function and in doing so have covered how structs and json interoperate, logging to CloudWatch, marshaling, processing of S3 event data and finally how to create a Transcribe job.

We’re nearly there. In the next blog, we’ll run through the entire process of getting our code in S3, creating the lambda function, creating the event subscription, and triggering our function.

thanks for reading! Feedback always welcome. 🙂

cheersy,

Tim

Introduction

In todays post we’ll cover an event handler that our Lambda function is going to use when it receives notification of an MP4 file being dropped in our S3 bucket from a subscribed S3 event. This will in turn cover the Context and Event objects. Lastly, we’ll look at the one specific to our function, S3Event.

Our Code

Because we’re only covering the handler and background info on the same and events, the code within the function is removed for this post.

.
.
.
// Handler is the Lambda function handler
// It uses an S3 event source, with the Lambda function being trigged
// when a CreateObject event occurs on an S3 bucket that has Events configured
func Handler(ctx context.Context, s3Event events.S3Event) {
.
.
.

}

func main() {
	lambda.Start(Handler)
}

Lambda Function Handlers for Go

For building a Lambda function handler in Go, you have a degree of scope with regards to the input and output parameters you use, provided they conform to, per the latest documentation, the following rules.

Although not strictly required for our function, Handler, we are using two parameters. The first, per the requirements of above, will be the implementation of context.Context. The second is the actual event data.

Context Object

The service which calls your Lambda function carries metadata, which the developer may find useful to view, or use. This is where the Context object comes into play. When your function signature contains a parameter for this, information is passed into this. There’s a plethora of information that can be available, some of which are service specific and others standard. An example of one of the latter is the AwsRequestID, a unique identifier that can be used as a reference later should AWS support be required. The complete documentation for the Context object is available here:

Event Data

This is the core information passed from the service to the function. It’s format is completely based on said service. In order to manage this, the Go SDK features interfaces for most event sources. In our case, this is the events.s3Event one.

If you wish to look at it’s construction in more detail, you can find it in the s3.go file, located within the events directory of the aws-lambda-go package.

We’ll be setting up an event subscription so that once an MP4 file is dropped into our S3 bucket, it invokes the Lambda function. What does the typical S3 event data our function would be passed look like? Look below.

Here’s the type of information we could expect to see once we have our Lambda function fully in place and an event subscription created to our S3 bucket. More on the latter later.

{
    "Records": [
        {
            "eventVersion": "2.0",
            "eventSource": "aws:s3",
            "awsRegion": "eu-west-1",
            "eventTime": "2018-07-10T05:26:14.8Z",
            "eventName": "ObjectCreated:Put",
            "userIdentity": {
                "principalId": "AWS:xxxxxx:xxxx@xxxx.nl"
            },
            "requestParameters": {
                "sourceIPAddress": "xxxxxxx"
            },
            "responseElements": {
                "x-amz-id-2": "d8RfIrVGqhNSjmtSMkE6BmKHyzuAJBE+eDizEKiO9Jv0NUCwfesfrqYsRptsAdQ+4nZXtR4vRvc=",
                "x-amz-request-id": "706F905A81A24629"
            },
            "s3": {
                "s3SchemaVersion": "1.0",
                "configurationId": "2717c2d4-8189-485d-b7e0-cafe707be54a",
                "bucket": {
                    "name": "xxxxxxxxx",
                    "ownerIdentity": {
                        "principalId": "xxxxxxxx"
                    },
                    "arn": "arn:aws:s3:::xxxxxxxxxxxx"
                },
                "object": {
                    "key": "sample.mp4",
                    "size": 5381540,
                    "urlDecodedKey": "",
                    "versionId": "",
                    "eTag": "ca7caf9e588d04ca0fbeee760284ba04",
                    "sequencer": "005B444375F09FE3D9"
                }
            }
        }
    ]
}

There is a lot of information there, but the key part of information passed that we’ll be using is contained within the object section.

Conclusion
In this post, we’ve covered the basics of a Lambda event handler for Go, the valid signatures that can be used with it and their purpose. We’ve also looked at the typical information that we can expect to be passed into our S3 event.

In the next blog, we’ll dig deeper into the function and the code within.

thanks for reading! Feedback always welcome. 🙂

cheersy,

Tim