After creating our DynamoDB table we need to finish uploading data into it.
To do that we’ll need to install the aws-sdk-dynamodb crate.
cargo add -p upload-pokemon-data aws-sdk-dynamodb
To write data to DynamoDB we’ll have to construct a WriteRequest. We could send each request individually, but DynamoDB supports batch writing up to 25 of these requests at a time so we’ll batch them instead.
Instead of debugging the first Pokemon in our Vec, we’re going to iterate over it using chunks.
.chunks returns the Chunks struct, which implements the Iterator trait, so we can use a for loop to iterate over chunks of 25 Pokemon at a time.
poke_chunk is a slice (that is, a continuous, read-only section) of 25 Pokemon from the original Vec.
for poke_chunk in pokemon.chunks(25) {
}
For each slice of Pokemon, we’ll iterate and construct a DynamoDB WriteRequest for each Pokemon in the batch.
If we .map over the items, the type of each item is going to be a shared referenced to a PokemonCsv. That is, &PokemonCsv. This is because we’re using slices, which are themselves shared references to the data in the original Vec.
We’ll need ownership over the Pokemon data because constructing the AttributeValue variants requires owned types such as String, Vec, and HashMap. Using .cloned will clone each of the PokemonCsv values, giving us a new copy that we own.
When using .cloned(), .map's pokemon is now a PokemonCsv that we own instead of a reference to the old PokemonCsv in the original Vec.
We’ll construct the WriteRequest for each Pokemon inside of map, then collect it into a Vec<WriteRequest> which makes batch a Vec<WriteRequest>.
for poke_chunk in pokemon.chunks(25) {
let batch = poke_chunk
.iter()
.cloned()
.map(|pokemon| {
...
})
.collect::<Vec<WriteRequest>>();
}
Inside of .map we need to build up a HashMap, so make sure you bring HashMap into scope at the top of the file.
use std::collections::HashMap;
Then construct a new HashMap with HashMap::new(). This map will need to be mutable, as we’ll be inserting values into it.
When we build our WriteRequest later, we’ll need to have a HashMap<String, AttributeValue> that defines all of the values that we’re inserting into Dynamo, so that will be the type of the HashMap.
Rust is fully capable of inferring the type of this HashMap for us due to the later usage, but If it makes you more comfortable you can also specify the type when we construct it.
let mut map: HashMap<String, AttributeValue> = HashMap::new();
A HashMap is basically the same concept as a JSON object. There are keys and values. In Rust the big difference is that all the values have to be the same type, which is why the type of the keys is String and the type of the values is the enum AttributeValue.
for poke_chunk in pokemon.chunks(25) {
let batch = poke_chunk
.iter()
.cloned()
.map(|pokemon| {
let mut map = HashMap::new();
map.insert(
"name".to_string(),
AttributeValue::S(pokemon.name.clone()),
);
map.insert(
"pokedex_id".to_string(),
AttributeValue::N(
pokemon.pokedex_id.to_string(),
),
);
map.insert(
"abilities".to_string(),
AttributeValue::L(
pokemon
.abilities
.into_iter()
.map(AttributeValue::S)
.collect(),
),
);
map.insert(
"typing".to_string(),
AttributeValue::L(
pokemon
.typing
.into_iter()
.map(AttributeValue::S)
.collect(),
),
);
map.insert(
"health_points".to_string(),
AttributeValue::N(
pokemon.hp.to_string(),
),
);
map.insert(
"pk".to_string(),
AttributeValue::S(
pokemon.name.to_kebab_case(),
),
);
WriteRequest::builder()
.put_request(
PutRequest::builder()
.set_item(Some(map))
.build(),
)
.build()
})
.collect::<Vec<WriteRequest>>();
}
Now we have a choice of which keys we want to store in Dynamo. Each key will be inserted into the HashMap as a String and each value will have to be wrapped in an AttributeValue that is appropriate for its type.
In this case we’ve chosen to include the Pokemon name. and used the AttributeValue::S variant, which is Dynamo’s string type.
Yes, unfortunately all of the DynamoDB AttributeValue variants are single-letter names so you may want to refer to the documentation to see the full list and which single letters are associated with what database types.
map.insert(
"name".to_string(),
AttributeValue::S(pokemon.name.clone()),
);
Moving on to numbers, we can see something a bit unintuitive. The AttributeValue::N, or number, variant accepts a String, not any kind of number. This is a DynamoDB API decision to maximize the compatibility across languages.
Since we’re working with a fairly low level DynamoDB API client, we have to see some of the odd-feeling API decisions like this. It doesn’t affect the data in Dynamo, as any mathematical operations in queries will still treat this field as a number. The String is only for sending it across the network to Dynamo in the first place.
map.insert(
"pokedex_id".to_string(),
AttributeValue::N(
pokemon.pokedex_id.to_string(),
),
);
The final type we’ll show off is the AttributeValue::L type, also known as a “List”.
We’ll take the pokemon.abilities value, which is a Vec, iterate over it using into_iter, which will give us String instead of &String, and apply the AttributeValue::S variant to the values inside the Vec.
This gives us a List of Strings for Dynamo.
map.insert(
"abilities".to_string(),
AttributeValue::L(
pokemon
.abilities
.into_iter()
.map(AttributeValue::S)
.collect(),
),
);
One field we can not forget is the field used in our index, a lowercased version of the pokemon name. It’s important to note that the Pokemon names can have spaces in them, so we actually need to not just lowercase, but also replace spaces and such. To do this we can use the Inflector crate (yes the capital I is important).
cargo add Inflector
The inflector crate offers us a trait that extends Strings with additional functions such as to_kebab_case that change the representation of the words in them.
We’ll use this to set the pk attribute name to the “kebab cased” Pokemon name.
map.insert(
"pk".to_string(),
AttributeValue::S(
pokemon.name.to_kebab_case(),
),
);
We can continue going down the list of all of the fields and include any we want from the PokemonCsv into the HashMap. I’ve gone ahead and included all of them here although I encourage you to try to do it yourself before checking your work.
for poke_chunk in pokemon.chunks(25) {
let batch = poke_chunk
.iter()
.cloned()
.map(|pokemon| {
let mut map = HashMap::new();
map.insert(
"name".to_string(),
AttributeValue::S(pokemon.name.clone()),
);
map.insert(
"pokedex_id".to_string(),
AttributeValue::N(
pokemon.pokedex_id.to_string(),
),
);
map.insert(
"abilities".to_string(),
AttributeValue::L(
pokemon
.abilities
.into_iter()
.map(AttributeValue::S)
.collect(),
),
);
map.insert(
"typing".to_string(),
AttributeValue::L(
pokemon
.typing
.into_iter()
.map(AttributeValue::S)
.collect(),
),
);
map.insert(
"health_points".to_string(),
AttributeValue::N(
pokemon.hp.to_string(),
),
);
map.insert(
"attack".to_string(),
AttributeValue::N(
pokemon.attack.to_string(),
),
);
map.insert(
"defense".to_string(),
AttributeValue::N(
pokemon.defense.to_string(),
),
);
map.insert(
"special_attack".to_string(),
AttributeValue::N(
pokemon.special_attack.to_string(),
),
);
map.insert(
"special_defense".to_string(),
AttributeValue::N(
pokemon.special_defense.to_string(),
),
);
map.insert(
"speed".to_string(),
AttributeValue::N(
pokemon.speed.to_string(),
),
);
map.insert(
"height".to_string(),
AttributeValue::N(
pokemon.height.to_string(),
),
);
map.insert(
"weight".to_string(),
AttributeValue::N(
pokemon.weight.to_string(),
),
);
map.insert(
"generation".to_string(),
AttributeValue::N(
pokemon.generation.to_string(),
),
);
if let Some(rate) = pokemon.female_rate {
map.insert(
"female_rate".to_string(),
AttributeValue::N(rate.to_string()),
);
}
map.insert(
"genderless".to_string(),
AttributeValue::Bool(
pokemon.genderless,
),
);
map.insert(
"is_legendary_or_mythical".to_string(),
AttributeValue::Bool(
pokemon.is_legendary_or_mythical,
),
);
map.insert(
"is_default".to_string(),
AttributeValue::Bool(
pokemon.is_default,
),
);
map.insert(
"forms_switchable".to_string(),
AttributeValue::Bool(
pokemon.forms_switchable,
),
);
map.insert(
"base_experience".to_string(),
AttributeValue::N(
pokemon.base_experience.to_string(),
),
);
map.insert(
"capture_rate".to_string(),
AttributeValue::N(
pokemon.capture_rate.to_string(),
),
);
map.insert(
"egg_groups".to_string(),
AttributeValue::L(
pokemon
.egg_groups
.into_iter()
.map(AttributeValue::S)
.collect(),
),
);
map.insert(
"base_happiness".to_string(),
AttributeValue::N(
pokemon.base_happiness.to_string(),
),
);
if let Some(evolves_from) =
pokemon.evolves_from
{
map.insert(
"evolves_from".to_string(),
AttributeValue::S(evolves_from),
);
}
map.insert(
"primary_color".to_string(),
AttributeValue::S(
pokemon.primary_color,
),
);
map.insert(
"number_pokemon_with_typing"
.to_string(),
AttributeValue::N(
pokemon
.number_pokemon_with_typing
.to_string(),
),
);
map.insert(
"normal_attack_effectiveness"
.to_string(),
AttributeValue::N(
pokemon
.normal_attack_effectiveness
.to_string(),
),
);
map.insert(
"fire_attack_effectiveness".to_string(),
AttributeValue::N(
pokemon
.fire_attack_effectiveness
.to_string(),
),
);
map.insert(
"water_attack_effectiveness"
.to_string(),
AttributeValue::N(
pokemon
.water_attack_effectiveness
.to_string(),
),
);
map.insert(
"electric_attack_effectiveness"
.to_string(),
AttributeValue::N(
pokemon
.electric_attack_effectiveness
.to_string(),
),
);
map.insert(
"grass_attack_effectiveness"
.to_string(),
AttributeValue::N(
pokemon
.grass_attack_effectiveness
.to_string(),
),
);
map.insert(
"ice_attack_effectiveness".to_string(),
AttributeValue::N(
pokemon
.ice_attack_effectiveness
.to_string(),
),
);
map.insert(
"fighting_attack_effectiveness"
.to_string(),
AttributeValue::N(
pokemon
.fighting_attack_effectiveness
.to_string(),
),
);
map.insert(
"poison_attack_effectiveness"
.to_string(),
AttributeValue::N(
pokemon
.poison_attack_effectiveness
.to_string(),
),
);
map.insert(
"ground_attack_effectiveness"
.to_string(),
AttributeValue::N(
pokemon
.ground_attack_effectiveness
.to_string(),
),
);
map.insert(
"fly_attack_effectiveness".to_string(),
AttributeValue::N(
pokemon
.fly_attack_effectiveness
.to_string(),
),
);
map.insert(
"psychic_attack_effectiveness"
.to_string(),
AttributeValue::N(
pokemon
.psychic_attack_effectiveness
.to_string(),
),
);
map.insert(
"bug_attack_effectiveness".to_string(),
AttributeValue::N(
pokemon
.bug_attack_effectiveness
.to_string(),
),
);
map.insert(
"rock_attack_effectiveness".to_string(),
AttributeValue::N(
pokemon
.rock_attack_effectiveness
.to_string(),
),
);
map.insert(
"ghost_attack_effectiveness"
.to_string(),
AttributeValue::N(
pokemon
.ghost_attack_effectiveness
.to_string(),
),
);
map.insert(
"dragon_attack_effectiveness"
.to_string(),
AttributeValue::N(
pokemon
.dragon_attack_effectiveness
.to_string(),
),
);
map.insert(
"dark_attack_effectiveness".to_string(),
AttributeValue::N(
pokemon
.dark_attack_effectiveness
.to_string(),
),
);
map.insert(
"steel_attack_effectiveness"
.to_string(),
AttributeValue::N(
pokemon
.steel_attack_effectiveness
.to_string(),
),
);
map.insert(
"fairy_attack_effectiveness"
.to_string(),
AttributeValue::N(
pokemon
.fairy_attack_effectiveness
.to_string(),
),
);
map.insert(
"pk".to_string(),
AttributeValue::S(
pokemon.name.to_kebab_case(),
),
);
WriteRequest::builder()
.put_request(
PutRequest::builder()
.set_item(Some(map))
.build(),
)
.build()
})
.collect::<Vec<WriteRequest>>();
}
In the final section of our .map we need to actually construct the WriteRequest. A WriteRequest can be either a PutRequest or a DeleteRequest but for some reason the SDK didn’t make the WriteRequest an enum.
The API in the AWS Rust SDK for building these values definitely feels like an API that was code-generated, which it was, and not an API that was built to be “Rusty”, so it’s ok if you feel like this is pretty awkward.
We’ll use the WriteRequest::builder function to build a new WriteRequest. We only have one function to call, which is to set the put_request.
We also build the PutRequest via its own builder() function.
The PutRequest also only needs one function, which is set_item. set_item accepts an Option<HashMap<String, AttributeValue>> which feels a little odd because it’s the only function we can use to construct a PutRequest anyway, but it all works so we live with the API AWS is offering us for now.
Finally we need to call .build on each of the builders to finish them off and return the relevant struct.
WriteRequest::builder()
.put_request(
PutRequest::builder()
.set_item(Some(map))
.build(),
)
.build()
Now batch is a Vec<WriteRequest> and we need to send it to Dyanmo. We do this via a client , using the batch_write_item function.
.request_items takes the table name we’re sending the request to and the batch we’re sending.
We can then call .send and await the send to send the data to Dynamo.
let sent = client
.batch_write_item()
.request_items(&table_name, batch)
.send()
.await?;
We can also deal with unprocessed items. sent is a BatchWriteItemOutput which has an unprocessed_items field if any of the items we sent weren’t processed.
In this case they all should be, so if there are any items we panic.
if let Some(items) = sent.unprocessed_items {
if !items.is_empty() {
panic!("items didn't make it");
}
}
After writing the code to upload the Pokemon CSV to DynamoDB, we can use cargo run to execute our application. We’ll want to set AWS_PROFILE and TABLE_NAME as environment variables.
AWS_PROFILE=rust-adventure-playground TABLE_NAME=InfraStack-PokemonTable7DFA0E9C-1II2IAD7OZ2EJ cargo run
After a few seconds, the program will stop running and we can head over to the AWS console website and see all of the data in our DynamoDB table.