STA 250 :: Advanced Statistical Computing (UCD, Fall 2013)

Code + goodies used in Prof. Baines' STA 250 Course (UC Davis, Fall 2013)

Project maintained by STA250 Hosted on GitHub Pages — Theme by mattgraham

Getting Started With Amazon Web Services (AWS)

Once you receive an email from Prof. Baines with your AWS login and password, proceed to the AWS login page
First, you need to create your key pair by going to:
https://console.aws.amazon.com/ec2/
Click on "Key Pairs" under "Network & Security" in the left panel of the console.
Click "Create Key Pair" and give your key a distinctive name (e.g., "JimsKeyPair"). The key will be created and you will be prompted to save the file to your computer.

Keep this file safe, it will be your private key for logging into your EMR job.

Note: Only follow the next steps when you have written your mapper and reducer scripts and are ready to deploy your job.
Next, go to the Elastic MapReduce tab, and click to launch new work flow.
Under the "Create a Job Flow" option, check "Run Your Own Application" and select "Hive Program".
Click "Next", and then select "Start an Interactive Hive Session".
You should now be in the "Configure EC2 Instances" tab. For the Master Instance Group select "Large", the Core Instance Group specify 4 "Large" instances. The Task Instance Group can be left unused. Click "Next"
Under "Advanced Options" and "Amazon EC2 Key Pair" select the name of the key pair you created earlier and click "Next".
For "Bootstrap Actions" select "Configure your Bootstrap Actions" and then "Memory Intensive Configuration" before clicking "Continue".
When everything is ready, click "Create Job Flow".
Go back to the EMR console and click refresh until your job has starts (it may take a few minutes). Click on the job name, and wait until the "Master Public DNS Name" appears. Copy the DNS, as this will be how you ssh into your instance. Using the DNS, ssh into your session as follows:
```
    ssh -i /path/to/yourkeypair.pem hadoop@ec2-xxxx.us-west-2.compute.amazonaws.com
  
```
Where you need to replace the DNS with the one corresponding to your job.
You are now logged into an interactive session on your Hadoop (" Hive) configured cluster.
To transfer files into your cluster run the following from your local machine:
```
    scp -i yourkey.pem file_to_copy hadoop@(amazon_dns):~/
  
```
The above command assumes your private key file is stored in the directory from which you are ssh-ing (if not, just specify the full path), and you must replace (amazon_dns) with the public DNS shown in the AWS console. Similarly, to copy results back to your local machine upon completion:
```
    scp -i yourkey.pem hadoop@(amazon_dns):~/file_to_copy dest_to_copy_to
  
```
Do not forget to terminate the session when you are done!!!

Happy coding! :)