AWS: Retrieve DynamoDB Data With Lambda
Intro​
In the previous post DynamoDB Intro we covered how to add items to a DynamoDB table using Lambda functions. Now we will cover a couple of methods for retrieving items from the database. We are building a sample application that stores and requests user data attributes such as height, weight and income.
Fetch Data Methods​
We can start by referencing the 📘 SDK v2 API Guide: DynamoDB Methods where we can see several methods for retrieving data from DynamoDB.
getItem​
getItem(params = {}, callback) ⇒ AWS.Request
The getItem operation returns a set of attributes for the item with the given primary key.
scan​
scan(params = {}, callback) ⇒ AWS.Request
The scan operation returns one or more items and item attributes by accessing every item in a table or a secondary index.
query​
query(params = {}, callback) ⇒ AWS.Request
The Query operation finds items based on primary key values.
Example: Fetch All User Data​
Let us review our blank Lambda function here. This Lambda function will be handling requests to our applications API, and those requests are going to include a type. The type will either be all or single to indicate whether we are requesting the data for all users or a single user. And we can see that we have a simple if statement here in our function to send a different request based on the type.
// Import the AWS SDK
var AWS = require("aws-sdk")
var dynamodb = new AWS.DynamoDB({
region: "us-east-2",
apiVersion: "2012-08-10",
})
exports.handler = (event, context, callback) => {
const type = event.type
// app will request all user data or single user data
if (type === "all") {
callback(null, "Requested All User Data")
} else if (type === "single") {
callback(null, "Requested Single User Data")
} else {
callback(null, "Invalid Type Request in URL Parameter")
}
}
For now we will be working on the request if (type === 'all'). Therefore let us start by using the scan method.
Let us first define our parameters. The name of the table we are querying is compare-yourself.
var params = {
TableName: "compare-yourself",
}
and then we can invoke the scan method
dynamodb.scan(params, function (err, data) {
if (err) {
console.log(err, err.stack) // an error occurred
} else {
console.log(data) // successful response
callback(null, data) // highlight-line
}
Note that we included our callback in the highlight above.
So our complete function looks like this
// Import the AWS SDK
var AWS = require("aws-sdk")
var dynamodb = new AWS.DynamoDB({
region: "us-east-2",
apiVersion: "2012-08-10",
})
exports.handler = (event, context, callback) => {
const type = event.type
// app will request all user data or single user data
if (type === "all") {
var params = {
TableName: "compare-yourself",
}
dynamodb.scan(params, function (err, data) {
if (err) {
console.log(err, err.stack) // an error occurred
} else {
console.log(data) // successful response
callback(null, data)
}
})
} else if (type === "single") {
callback(null, "Requested Single User Data")
} else {
callback(null, "Invalid Type Request in URL Parameter")
}
}
and we can make a test GET request to the API /compare-yourself/all and then look in the CloudWatch console logs and check for our data dump.
AccessDeniedException: User: arn:aws:sts::XXXXXXXXXXXXXXX:assumed-role/cyGetData-role-XXXXXXXX/cyGetData is
not authorized to perform: dynamodb:Scan on resource: arn:aws:dynamodb:us-east-2:XXXXXXXXXXXX:table/compare-yourself
So we can see that the IAM role for our Lambda function is not authorized to access the database. So let us go in and fix that just like we did in our last post.
And now we can see that the console log has come through for us.
And a Postman test returns the following JSON formatted data
{
"Items": [
{
"UserId": {
"S": "user002"
},
"income": {
"N": "100000"
},
"height": {
"N": "150"
},
"age": {
"N": "33"
}
},
{
"UserId": {
"S": "user001"
},
"income": {
"N": "100000"
},
"height": {
"N": "150"
},
"age": {
"N": "33"
}
},
{
"UserId": {
"S": "user003"
},
"income": {
"N": "5670000"
},
"height": {
"N": "150"
},
"age": {
"N": "55"
}
}
],
"Count": 3,
"ScannedCount": 3
}
So this is working perfectly. The next steps in this application would be to make these GET and POST requests from our application instead of manually using Postman. We will get to that later, for now let us cover some more security best practices.
Currently with IAM permissions we have given these Lambda functions blanket access to DynamoDB. However in a production environment we would want to fine-tune these permissions to be read/write only and only for specific tables. We will get into that in the next post.
Example: Fetch Single User Data​
Before the next example let us quickly do some cleanup here. We learned in another post about the AWS SDK Document Client
Which reformats our data for us. So the following example will be using what we learned in our post about reformatting our DynamoDB requests and responses
📘 Ncoughlin: Reformatting DynamoDB Data
And I also have reformatted our code to be cleaner by creating objects for each of our request types, so that our if statement at the bottom is less crowded.
// Import the AWS SDK
var AWS = require("aws-sdk")
var docClient = new AWS.DynamoDB.DocumentClient({ region: "us-east-2" })
// Handler
exports.handler = (event, context, callback) => {
const type = event.type
// Request type helpers
let all = {
params: {
TableName: "compare-yourself",
},
request:
(err, data) => {
if (err) {
// an error occurred
console.log(err, err.stack)
}
else {
// successful response
console.log(data);
callback(null, data);
}
}
};
let single = {
};
// app will request all user data or single user data
if (type === "all") {
docClient.scan(all.params, all.request) // highlight-line
}
else if (type === "single") {
callback(null, "Requested Single User Data")
}
else {
callback(null, "Invalid Type Request in URL Parameter")
}
}
A couple other things to note here, is that we will now need to include a key in our request so that DynamoDB knows what data we are requesting, and in this case that is the UserId. Also note that the method to retrieve a single item using the document client is get and not getItem. Otherwise we will get an error that docClient.getItem() is not a function, because it is not a valid method. And lastly we need to expand the permissions of the cyGetData role that we have attached to this Lambda function, otherwise we will not have permission to run this function on the database.
And if we take all that into account we end up with this.
// Import the AWS SDK
var AWS = require("aws-sdk")
var docClient = new AWS.DynamoDB.DocumentClient({ region: "us-east-2" })
// Handler
exports.handler = (event, context, callback) => {
// Request type (all/single)
const type = event.type;
// extract User ID
const userid = event.userid;
// Request type helpers
let all = {
params: {
TableName: "compare-yourself",
},
request:
(err, data) => {
if (err) {
// an error occurred
console.log(err, err.stack);
}
else {
// successful response
console.log(data);
callback(null, data);
}
}
};
let single = {
params: {
TableName: "compare-yourself",
Key: {
"UserId": `${userid}`
},
},
request:
(err, data) => {
if (err) {
// an error occurred
console.log(err, err.stack);
}
else {
// successful response
console.log(data);
callback(null, data);
}
}
};
// app will request all user data or single user data
if (type === "all") {
docClient.scan(all.params, all.request);
}
else if (type === "single") {
docClient.get(single.params, single.request);
}
else {
callback(null, "Invalid Type Request in URL Parameter");
}
};
which if we give the following data in a request
{
"type": "single",
"userid": "user001"
}
gives us this result
{
"Item": {
"UserId": "user001",
"income": 100000,
"height": 150,
"age": 33
}
}
So this is working perfectly. Note that our data is already returned to us in the correct data type/format thanks to the Document Client.
Automated Amazon Reports
Automatically download Amazon Seller and Advertising reports to a private database. View beautiful, on demand, exportable performance reports.
