> **Public Version Notice:** This is the public version of the Gormley Lab setup guide. Account-specific details (AWS Account ID and S3 bucket name) have been replaced with placeholders. To complete the setup, you will need the private version of this guide with those details filled in, as well as your personal AWS credentials — both of which are provided by Adam. Contact him at adam.gormley@rutgers.edu to get started.

---

# Getting Started with AWS Tools in the Gormley Lab

Welcome to the lab! This guide will walk you through setting up AWS tools for your research work. You'll learn to configure:

1. **AWS CLI** - Access to lab AWS resources
2. **Claude Code** - AI-powered coding assistant (billed to lab AWS account)
3. **AWS SageMaker** - Machine learning model training platform

All usage will be billed directly to the lab's AWS account, which means costs can be allocated to research grants.

**What you'll need:**
- The AWS credentials CSV file sent to you (contains your Access Key ID and Secret Access Key)
- About 30-40 minutes to complete the setup
- A computer with internet access and Python installed

---

## Table of Contents

### Part 1: AWS Setup
1. [Install AWS CLI](#step-1-install-aws-cli)
2. [Configure AWS Credentials](#step-2-configure-aws-credentials)

### Part 2: Claude Code Setup
3. [Install Claude Code](#step-3-install-claude-code)
4. [Configure Claude Code for AWS Billing](#step-4-configure-claude-code-for-aws-billing)
5. [Verify Claude Code Setup](#step-5-verify-claude-code-setup)

### Part 3: SageMaker Setup
6. [Install Python Dependencies](#step-6-install-python-dependencies)
7. [Configure SageMaker Access](#step-7-configure-sagemaker-access)
8. [Your First SageMaker Training Job](#step-8-your-first-sagemaker-training-job)

### Additional Resources
- [Troubleshooting](#troubleshooting)
- [Best Practices](#best-practices)
- [Quick Reference](#quick-reference-card)

---

## Part 1: AWS Setup

## Step 1: Install AWS CLI

The AWS CLI (Command Line Interface) allows you to interact with AWS services, including SageMaker and S3.

### For macOS

**Option A: Using Homebrew (Recommended)**

If you have Homebrew installed:

```bash
brew install awscli
```

**Option B: Using the AWS Installer**

1. Download the installer:
   ```bash
   curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
   ```

2. Run the installer:
   ```bash
   sudo installer -pkg AWSCLIV2.pkg -target /
   ```

3. Verify installation:
   ```bash
   aws --version
   ```
   
   You should see something like: `aws-cli/2.x.x Python/3.x.x Darwin/xx.x.x`

### For Windows

**Option A: Using Windows Installer (Recommended)**

1. Download the AWS CLI installer from: https://awscli.amazonaws.com/AWSCLIV2.msi
2. Run the downloaded MSI installer
3. Follow the on-screen instructions
4. Open a **new** Command Prompt or PowerShell window (important: existing windows won't have the updated PATH)
5. Verify installation:
   ```powershell
   aws --version
   ```
   
   You should see something like: `aws-cli/2.x.x Python/3.x.x Windows/xx`

**Option B: Using winget (Windows 10/11)**

Open PowerShell and run:
```powershell
winget install Amazon.AWSCLI
```

### For Linux

```bash
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
```

---

## Step 2: Configure AWS Credentials

Now you'll set up your AWS credentials using the CSV file you received.

### Open Your Credentials File

1. Locate the CSV file sent to you (filename looks like: `username_accessKeys.csv`)
2. Open it with a text editor or Excel
3. You'll see two important values:
   - **Access key ID** (starts with `AKIA...`)
   - **Secret access key** (a long random string)

**⚠️ Security Note**: Keep these credentials secure! Never commit them to Git or share them with anyone.

### Configure AWS CLI

Open your terminal (macOS/Linux) or Command Prompt/PowerShell (Windows) and run:

```bash
aws configure
```

You'll be prompted for four values. Enter them as follows:

```
AWS Access Key ID [None]: <paste your Access Key ID from the CSV>
AWS Secret Access Key [None]: <paste your Secret Access Key from the CSV>
Default region name [None]: us-east-1
Default output format [None]: json
```

**Example:**
```
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-east-1
Default output format [None]: json
```

### Verify AWS Configuration

Test that your credentials work:

```bash
aws sts get-caller-identity
```

You should see output showing your AWS account details:

```json
{
    "UserId": "AIDAI...",
    "Account": "123456789012",
    "Arn": "arn:aws:iam::123456789012:user/your-username"
}
```

If you see an error, double-check that you copied the credentials correctly.

### Test S3 Access

Verify you can access the lab's S3 bucket:

```bash
aws s3 ls s3://YOUR_BUCKET_NAME/
```

You should see the folder structure:
```
                           PRE code/
                           PRE datasets/
                           PRE models/
                           PRE notebooks/
                           PRE outputs/
```

---

## Part 2: Claude Code Setup

## Step 3: Install Claude Code

Claude Code is an AI-powered coding assistant that can help you write, debug, and understand code.

### For macOS

**Option A: Using Homebrew (Recommended)**

```bash
brew install --cask claude-code
```

**Option B: Using the Install Script**

```bash
curl -fsSL https://claude.ai/install.sh | bash
```

After installation, **open a new terminal window** to ensure Claude Code is in your PATH.

### For Windows

**Option A: Using PowerShell**

Open PowerShell and run:

```powershell
irm https://claude.ai/install.ps1 | iex
```

**Option B: Using Command Prompt**

Open Command Prompt and run:

```cmd
curl -fsSL https://claude.ai/install.cmd -o install.cmd && install.cmd && del install.cmd
```

**Option C: Using winget (Windows 10/11)**

```powershell
winget install Anthropic.ClaudeCode
```

After installation, **close and reopen** your terminal/command prompt.

### Verify Claude Code Installation

Run:

```bash
claude --version
```

You should see the version number. If you get "command not found", try closing and reopening your terminal.

---

## Step 4: Configure Claude Code for AWS Billing

This step tells Claude Code to bill usage to AWS instead of requiring a personal subscription. We'll use a `.env` file so this configuration only applies to your lab projects.

### Understanding .env Files

A `.env` file stores environment variables for a specific project. This is better than global configuration because:
- It keeps AWS settings separate per project
- It won't interfere with other Claude Code usage
- It's easy to share settings with collaborators

### Create Your Project Folder

1. **Create a new folder for your research project**:
   
   ```bash
   # macOS/Linux
   mkdir ~/gormley-lab-project
   cd ~/gormley-lab-project
   
   # Windows (PowerShell)
   mkdir $HOME\gormley-lab-project
   cd $HOME\gormley-lab-project
   ```

2. **Open the folder in VS Code**:
   
   ```bash
   code .
   ```
   
   Or use File → Open Folder in VS Code

### Create a .env File

1. **Create a new file called `.env`**
   
   - In VS Code: Click "New File" or press Ctrl+N (Cmd+N on Mac)
   - Save it as `.env` (make sure the filename starts with a dot!)
   - VS Code might warn you about hidden files - that's okay, save it anyway

2. **Add these two lines to the `.env` file:**

   ```
   CLAUDE_CODE_USE_BEDROCK=1
   AWS_REGION=us-east-1
   ```

3. **Save the file** (Ctrl+S or Cmd+S)

### Create a Startup Script (Recommended)

To make starting Claude Code easier, create a simple script in your project folder:

**For macOS/Linux users:**

1. Create a new file called `start-claude.sh`
2. Add these lines:
   ```bash
   #!/bin/bash
   export $(cat .env | xargs)
   claude
   ```
3. Save the file
4. Open terminal in your project folder and make it executable:
   ```bash
   chmod +x start-claude.sh
   ```

Now you can start Claude Code by running:
```bash
./start-claude.sh
```

**For Windows users:**

1. Create a new file called `start-claude.bat`
2. Add these lines:
   ```cmd
   @echo off
   for /f "delims=" %%i in (.env) do set %%i
   claude
   ```
3. Save the file

Now you can start Claude Code by double-clicking `start-claude.bat` or running:
```cmd
start-claude.bat
```

### Create .gitignore

**Important**: Never commit your `.env` file to Git!

Create a `.gitignore` file in your project folder with this content:

```
.env
*.pem
*.key
credentials.csv
```

---

## Step 5: Verify Claude Code Setup

Let's make sure everything is configured correctly:

1. **Make sure you're in your project folder** (the one with the `.env` file)

2. **Start Claude Code using your startup script:**
   
   **macOS/Linux:**
   ```bash
   ./start-claude.sh
   ```
   
   **Windows:**
   ```cmd
   start-claude.bat
   ```

3. **Check the status** by typing:
   ```
   /status
   ```

4. **Look for these key indicators:**
   - `API provider: AWS Bedrock` ✅ (This means billing goes to AWS)
   - `AWS region: us-east-1` ✅
   - `Model: global.anthropic.claude-sonnet-4-5...` ✅

**Example of correct output:**
```
Version: 2.1.19
API provider: AWS Bedrock
AWS region: us-east-1
Model: Default (global.anthropic.claude-sonnet-4-5-20250929-v1:0)
```

If you see this, **you're ready to use Claude Code!** 🎉

You can exit for now by typing `exit` or pressing Ctrl+C.

---

## Part 3: SageMaker Setup

## Step 6: Install Python Dependencies

SageMaker requires Python and the AWS SDK for Python (boto3) along with the SageMaker Python SDK.

### Verify Python Installation

Check that Python 3.8 or later is installed:

```bash
python3 --version
```

or on Windows:

```cmd
python --version
```

If Python is not installed, download it from [python.org](https://www.python.org/downloads/).

### Create a Virtual Environment (Recommended)

It's best practice to use a virtual environment for your Python projects:

```bash
# macOS/Linux
python3 -m venv venv
source venv/bin/activate

# Windows
python -m venv venv
venv\Scripts\activate
```

You should see `(venv)` appear in your terminal prompt.

### Install Required Packages

```bash
pip install --upgrade pip
pip install boto3 sagemaker pandas numpy scikit-learn matplotlib
```

This installs:
- `boto3` - AWS SDK for Python
- `sagemaker` - SageMaker Python SDK
- `pandas`, `numpy`, `scikit-learn` - Common ML libraries
- `matplotlib` - Plotting library

### Verify Installation

```bash
python -c "import sagemaker; print(sagemaker.__version__)"
```

You should see the version number (e.g., `2.x.x`).

---

## Step 7: Configure SageMaker Access

### Understanding SageMaker Components

SageMaker needs a few pieces of information:

1. **S3 Bucket**: Where your data and models are stored (provided by your PI)
2. **IAM Role**: Permissions for SageMaker to access resources
3. **Region**: Where your jobs run (`us-east-1`)

### Important ARN Information

Your SageMaker jobs will use an execution role created by your PI. The ARN looks like this:

```
arn:aws:iam::YOUR_ACCOUNT_ID:role/GormleyLabSageMakerExecutionRole
```

**Note**: Your PI will provide you with the actual Account ID and S3 bucket name. You'll need both values for the configuration file below.

### Create a Configuration File

In your project folder, create a file called `sagemaker_config.py`:

```python
"""
SageMaker configuration for Gormley Lab
"""

import boto3
import sagemaker

# Lab's S3 bucket for data and models (get this from your PI)
BUCKET_NAME = 'YOUR_BUCKET_NAME'

# SageMaker execution role (get the Account ID from your PI)
SAGEMAKER_ROLE = 'arn:aws:iam::YOUR_ACCOUNT_ID:role/GormleyLabSageMakerExecutionRole'

# AWS Region
REGION = 'us-east-1'

# Initialize SageMaker session
def get_sagemaker_session():
    """Get a SageMaker session configured for the lab"""
    boto_session = boto3.Session(region_name=REGION)
    return sagemaker.Session(boto_session=boto_session)

# S3 paths for organization
PATHS = {
    'datasets': f's3://{BUCKET_NAME}/datasets/',
    'models': f's3://{BUCKET_NAME}/models/',
    'code': f's3://{BUCKET_NAME}/code/',
    'outputs': f's3://{BUCKET_NAME}/outputs/',
}

# Common instance types for different use cases
INSTANCE_TYPES = {
    'cpu_small': 'ml.m5.large',      # Testing, small datasets
    'cpu_medium': 'ml.m5.xlarge',    # Medium CPU training
    'gpu_budget': 'ml.g4dn.xlarge',  # Budget GPU training (T4)
    'gpu_single': 'ml.p3.2xlarge',   # Single V100 GPU
    'gpu_multi': 'ml.p3.8xlarge',    # 4x V100 GPUs
}
```

**Replace `ACCOUNT_ID`** in the `SAGEMAKER_ROLE` with the actual account ID provided by Adam.

---

## Step 8: Your First SageMaker Training Job

Let's create a complete example that trains a simple scikit-learn model using SageMaker.

### How SageMaker Training Works: The Two-File Pattern

Every SageMaker training job in this guide is built from two files that work together. Understanding this pattern will make both examples much clearer.

**The training script** (e.g., `train.py`) contains your actual machine learning code — loading data, defining a model, training it, evaluating it, and saving it. This is the file that runs on the remote SageMaker instance. You never call it directly from your local machine. Instead, SageMaker uploads it to a cloud instance and executes it there.

**The launcher script** (e.g., `run_training.py`) is what you run locally. Its job is to configure and submit a training job to SageMaker. It tells SageMaker which training script to use, what instance to run it on, what hyperparameters to pass, and where to store outputs. The key line connecting the two files is the `entry_point` parameter, which points the launcher at the training script by filename.

The flow looks like this:

```
You run locally:          run_training.py
                              │
                              │  tells SageMaker:
                              │    - entry_point='train.py'   ← which script to run
                              │    - instance_type            ← what hardware to use
                              │    - hyperparameters          ← passed as CLI args to train.py
                              │    - output_path              ← where to save the model in S3
                              ▼
SageMaker (remote instance):  train.py
                              │
                              │  receives hyperparameters as command-line arguments
                              │  trains the model
                              │  saves the model to the directory SageMaker specifies
                              ▼
                          Model artifact uploaded to S3
```

The hyperparameters defined in the launcher are passed to the training script as command-line arguments. That's why `train.py` uses `argparse` — it's how it receives values like `--n-estimators 100` from SageMaker. Similarly, SageMaker sets environment variables like `SM_MODEL_DIR` that tell the training script where to save its output. The training script reads these via `os.environ` as default values in argparse.

### Where Does S3 Fit In?

S3 is the shared storage layer that connects your local machine to the remote SageMaker instance. It plays two distinct roles in a training job:

**Input data (uploading before training):** Your training data lives in S3 so the remote instance can access it. Before running a job on your own dataset, you upload it to the lab bucket. When you call `.fit()` in the launcher, you point it at that S3 location. SageMaker then downloads the data onto the remote instance and makes it available to your training script via the path in the `SM_CHANNEL_TRAIN` environment variable.

**Output models (downloading after training):** When your training script saves a model, it writes to a local directory on the remote instance (`SM_MODEL_DIR`). After training completes, SageMaker automatically uploads everything in that directory to the S3 path you specified in `output_path`. That's how your trained model gets back to a place you can access it.

The full flow with S3 included looks like this:

```
Your data (local)
      │
      │  aws s3 cp data.csv s3://YOUR_BUCKET_NAME/datasets/
      ▼
S3 bucket (input)          ← data lives here before training
      │
      │  SageMaker downloads data onto remote instance
      ▼
You run locally:          run_training.py
                              │
                              │  tells SageMaker:
                              │    - entry_point='train.py'
                              │    - hyperparameters
                              │    - .fit({'train': 's3://...'})   ← where to get input data
                              │    - output_path='s3://...'        ← where to put the model
                              ▼
SageMaker (remote instance):  train.py
                              │  reads data from local path (copied from S3)
                              │  trains the model
                              │  saves model to SM_MODEL_DIR
                              ▼
S3 bucket (output)         ← trained model uploaded here automatically
                              │
                              │  aws s3 cp s3://...model.tar.gz ./
                              ▼
Your machine (local)       ← download and use the model
```

**The two examples below intentionally skip the upload step.** They use built-in datasets (`sklearn`'s iris dataset and `make_classification`) that are generated inside the training script itself, so there's nothing to upload beforehand. This keeps the examples simple and means you can run them immediately without preparing any data files. The `.fit()` call in both launchers is therefore called with no data argument.

When you're ready to train on your own data, two things change. First, upload your data to S3 before running the job:

```bash
aws s3 cp my_dataset.csv s3://YOUR_BUCKET_NAME/datasets/my_dataset.csv
```

Second, pass the S3 path into `.fit()` so SageMaker knows where to find it:

```python
# Instead of:
estimator.fit(wait=True)

# You do:
estimator.fit(
    {'train': 's3://YOUR_BUCKET_NAME/datasets/my_dataset.csv'},
    wait=True
)
```

SageMaker will download that file onto the remote instance and make it available at the path stored in `SM_CHANNEL_TRAIN`. Your training script then reads from that path instead of generating synthetic data. The `Working with S3 Data` section later in this guide covers upload and download operations in detail.

With that in mind, here's how it plays out in practice:

### Example 1: Train a Scikit-learn Model

This example trains a random forest classifier on the iris dataset.

#### Create the Training Script

Create a file called `train.py`. This is the script that will run on the remote SageMaker instance:

```python
"""
Simple scikit-learn training script for SageMaker
Trains a Random Forest classifier on the Iris dataset
"""

import argparse
import os
import joblib
import pandas as pd
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report

def train(args):
    """Main training function"""
    
    print("Loading data...")
    # This example uses a built-in dataset so no S3 upload is needed.
    # When using your own data, you would replace this block with code that
    # reads from args.train (the path where SageMaker downloads your S3 data).
    # See the "Where Does S3 Fit In?" section above for details.
    from sklearn.datasets import load_iris
    iris = load_iris()
    X, y = iris.data, iris.target
    
    # Split data
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.2, random_state=42
    )
    
    print(f"Training set size: {len(X_train)}")
    print(f"Test set size: {len(X_test)}")
    
    # Train model
    print(f"Training Random Forest with {args.n_estimators} trees...")
    model = RandomForestClassifier(
        n_estimators=args.n_estimators,
        max_depth=args.max_depth,
        random_state=42
    )
    model.fit(X_train, y_train)
    
    # Evaluate
    train_pred = model.predict(X_train)
    test_pred = model.predict(X_test)
    
    train_accuracy = accuracy_score(y_train, train_pred)
    test_accuracy = accuracy_score(y_test, test_pred)
    
    print(f"\nTraining accuracy: {train_accuracy:.4f}")
    print(f"Test accuracy: {test_accuracy:.4f}")
    print("\nClassification Report:")
    print(classification_report(y_test, test_pred, target_names=iris.target_names))
    
    # Save model
    model_path = os.path.join(args.model_dir, 'model.joblib')
    joblib.dump(model, model_path)
    print(f"\nModel saved to {model_path}")

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    
    # Hyperparameters
    parser.add_argument('--n-estimators', type=int, default=100)
    parser.add_argument('--max-depth', type=int, default=5)
    
    # SageMaker specific arguments
    parser.add_argument('--model-dir', type=str, default=os.environ.get('SM_MODEL_DIR', './model'))
    parser.add_argument('--train', type=str, default=os.environ.get('SM_CHANNEL_TRAIN', './data'))
    
    args = parser.parse_args()
    
    # Create model directory if it doesn't exist
    os.makedirs(args.model_dir, exist_ok=True)
    
    train(args)
```

Notice the bottom of `train.py`: the `argparse` block defines `--n-estimators` and `--max-depth` as command-line arguments, and reads `SM_MODEL_DIR` from the environment. These are how SageMaker communicates with this script — the launcher sets the hyperparameters, and SageMaker sets the environment variables. The training script doesn't need to know anything about S3 or cloud instances; it just trains a model and saves it to the directory it's told to use.

#### Create the SageMaker Job Script

Create a file called `run_training.py`. This is the script you run locally to submit the job. Watch for how `entry_point` and `hyperparameters` connect back to `train.py`:

```python
"""
Launch a SageMaker training job
"""

import boto3
import sagemaker
from sagemaker.sklearn import SKLearn
from datetime import datetime
import sagemaker_config as config

def main():
    # Initialize session
    sagemaker_session = config.get_sagemaker_session()
    
    # Create a unique job name with timestamp
    timestamp = datetime.now().strftime('%Y-%m-%d-%H-%M-%S')
    job_name = f'sklearn-iris-{timestamp}'
    
    print(f"Starting training job: {job_name}")
    print(f"Instance type: {config.INSTANCE_TYPES['cpu_small']}")
    
    # Create SKLearn estimator
    sklearn_estimator = SKLearn(
        entry_point='train.py',                    # Your training script
        role=config.SAGEMAKER_ROLE,                # IAM role
        instance_type=config.INSTANCE_TYPES['cpu_small'],  # Instance type
        instance_count=1,                          # Number of instances
        framework_version='1.2-1',                 # Scikit-learn version
        py_version='py3',                          # Python version
        sagemaker_session=sagemaker_session,
        hyperparameters={
            'n-estimators': 100,
            'max-depth': 5,
        },
        output_path=config.PATHS['models'],        # Where to save the model
        code_location=config.PATHS['code'],        # Where to upload code
        base_job_name='sklearn-iris',
        use_spot_instances=True,                   # Use spot for cost savings
        max_wait=3600,                             # Max wait time (1 hour)
        max_run=1800,                              # Max training time (30 min)
    )
    
    # Start training
    print("\nStarting training job...")
    print("This may take several minutes...")
    
    # No data argument here because train.py generates its own dataset.
    # When using your own data, upload it to S3 first, then pass the path:
    #   sklearn_estimator.fit({'train': 's3://YOUR_BUCKET_NAME/datasets/your_data.csv'}, wait=True)
    sklearn_estimator.fit(wait=True)
    
    print(f"\nTraining job completed!")
    print(f"Model artifacts saved to: {sklearn_estimator.model_data}")
    print(f"Job name: {sklearn_estimator.latest_training_job.name}")
    
    # You can now deploy this model if needed
    # predictor = sklearn_estimator.deploy(instance_type='ml.t2.medium', initial_instance_count=1)

if __name__ == '__main__':
    main()
```

To summarize the connection: `entry_point='train.py'` tells SageMaker which script to execute remotely, and the `hyperparameters` dictionary (`'n-estimators': 100`, `'max-depth': 5`) gets passed to that script as `--n-estimators 100 --max-depth 5` on the command line. The `output_path` tells SageMaker where in S3 to upload the model that `train.py` saves.

#### Run Your First Training Job

1. **Make sure you're in your project folder** with the virtual environment activated

2. **Verify all files are in place:**
   ```bash
   ls -la
   ```
   
   You should see:
   - `train.py` - Training script
   - `run_training.py` - Job launcher
   - `sagemaker_config.py` - Configuration
   - `.env` - Claude Code config
   - `venv/` - Virtual environment

3. **Run the training job:**
   ```bash
   python run_training.py
   ```

4. **Watch the output**. You'll see:
   - Job creation messages
   - Training logs streaming in real-time
   - Final accuracy metrics
   - Model artifact location in S3

**Expected output:**
```
Starting training job: sklearn-iris-2025-01-31-14-30-45
Instance type: ml.m5.large

Starting training job...
This may take several minutes...
2025-01-31 14:31:12 Starting - Starting the training job...
2025-01-31 14:31:45 Starting - Preparing the instances for training...
2025-01-31 14:32:30 Downloading - Downloading input data...
2025-01-31 14:32:45 Training - Training image download completed. Training in progress...
Loading data...
Training set size: 120
Test set size: 30
Training Random Forest with 100 trees...

Training accuracy: 1.0000
Test accuracy: 0.9667

Classification Report:
              precision    recall  f1-score   support
...

2025-01-31 14:33:15 Uploading - Uploading generated training model
2025-01-31 14:33:30 Completed - Training job completed

Training job completed!
Model artifacts saved to: s3://YOUR_BUCKET_NAME/models/sklearn-iris-2025-01-31-14-30-45/output/model.tar.gz
Job name: sklearn-iris-2025-01-31-14-30-45
```

### Example 2: PyTorch Neural Network Training

For deep learning, you'll often use PyTorch. Here's a more advanced example.

#### Create PyTorch Training Script

Create `train_pytorch.py`. This follows the same pattern as Example 1 — it's the script that runs remotely on SageMaker. The key differences are that it uses PyTorch instead of scikit-learn, and it's designed to take advantage of a GPU instance if one is available:

```python
"""
PyTorch training script for SageMaker
Simple neural network for classification
"""

import argparse
import os
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

class SimpleNN(nn.Module):
    """Simple feedforward neural network"""
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(SimpleNN, self).__init__()
        self.network = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(hidden_dim, output_dim)
        )
    
    def forward(self, x):
        return self.network(x)

def train_model(model, train_loader, criterion, optimizer, device):
    """Train for one epoch"""
    model.train()
    total_loss = 0
    correct = 0
    total = 0
    
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.to(device), target.to(device)
        
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
        
        total_loss += loss.item()
        _, predicted = output.max(1)
        total += target.size(0)
        correct += predicted.eq(target).sum().item()
    
    return total_loss / len(train_loader), 100. * correct / total

def test_model(model, test_loader, criterion, device):
    """Evaluate the model"""
    model.eval()
    test_loss = 0
    correct = 0
    total = 0
    
    with torch.no_grad():
        for data, target in test_loader:
            data, target = data.to(device), target.to(device)
            output = model(data)
            test_loss += criterion(output, target).item()
            _, predicted = output.max(1)
            total += target.size(0)
            correct += predicted.eq(target).sum().item()
    
    test_loss /= len(test_loader)
    accuracy = 100. * correct / total
    
    return test_loss, accuracy

def main(args):
    # Set device
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    print(f"Using device: {device}")
    
    # This example uses synthetic data so no S3 upload is needed.
    # When using your own data, replace this block with code that reads
    # from args.model_dir or the SM_CHANNEL_TRAIN path. See the
    # "Where Does S3 Fit In?" section in this guide for details.
    print("Generating synthetic dataset...")
    X, y = make_classification(
        n_samples=1000,
        n_features=20,
        n_informative=15,
        n_redundant=5,
        n_classes=3,
        random_state=42
    )
    
    # Split and scale
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.2, random_state=42
    )
    
    scaler = StandardScaler()
    X_train = scaler.fit_transform(X_train)
    X_test = scaler.transform(X_test)
    
    # Convert to PyTorch tensors
    X_train = torch.FloatTensor(X_train)
    y_train = torch.LongTensor(y_train)
    X_test = torch.FloatTensor(X_test)
    y_test = torch.LongTensor(y_test)
    
    # Create data loaders
    train_dataset = TensorDataset(X_train, y_train)
    test_dataset = TensorDataset(X_test, y_test)
    
    train_loader = DataLoader(train_dataset, batch_size=args.batch_size, shuffle=True)
    test_loader = DataLoader(test_dataset, batch_size=args.batch_size)
    
    # Create model
    model = SimpleNN(
        input_dim=20,
        hidden_dim=args.hidden_dim,
        output_dim=3
    ).to(device)
    
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=args.learning_rate)
    
    # Training loop
    print(f"\nTraining for {args.epochs} epochs...")
    for epoch in range(args.epochs):
        train_loss, train_acc = train_model(model, train_loader, criterion, optimizer, device)
        test_loss, test_acc = test_model(model, test_loader, criterion, device)
        
        print(f'Epoch {epoch+1}/{args.epochs}:')
        print(f'  Train Loss: {train_loss:.4f}, Train Acc: {train_acc:.2f}%')
        print(f'  Test Loss: {test_loss:.4f}, Test Acc: {test_acc:.2f}%')
    
    # Save model
    model_path = os.path.join(args.model_dir, 'model.pth')
    torch.save({
        'model_state_dict': model.state_dict(),
        'scaler_mean': scaler.mean_,
        'scaler_scale': scaler.scale_,
    }, model_path)
    print(f"\nModel saved to {model_path}")

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    
    # Hyperparameters
    parser.add_argument('--batch-size', type=int, default=32)
    parser.add_argument('--epochs', type=int, default=10)
    parser.add_argument('--learning-rate', type=float, default=0.001)
    parser.add_argument('--hidden-dim', type=int, default=64)
    
    # SageMaker specific
    parser.add_argument('--model-dir', type=str, default=os.environ.get('SM_MODEL_DIR', './model'))
    
    args = parser.parse_args()
    os.makedirs(args.model_dir, exist_ok=True)
    
    main(args)
```

#### Create PyTorch Job Launcher

Create `run_pytorch_training.py`. Same pattern as Example 1 — this is what you run locally. Note that `entry_point` now points to `train_pytorch.py`, and the hyperparameters match the arguments that `train_pytorch.py` expects (`--batch-size`, `--epochs`, etc.). The instance type is also upgraded to a GPU instance since PyTorch training benefits from one:

```python
"""
Launch a PyTorch training job on SageMaker
"""

import sagemaker
from sagemaker.pytorch import PyTorch
from datetime import datetime
import sagemaker_config as config

def main():
    # Initialize session
    sagemaker_session = config.get_sagemaker_session()
    
    timestamp = datetime.now().strftime('%Y-%m-%d-%H-%M-%S')
    job_name = f'pytorch-nn-{timestamp}'
    
    print(f"Starting PyTorch training job: {job_name}")
    
    # For GPU training, use 'gpu_budget' or 'gpu_single'
    # For CPU training (cheaper), use 'cpu_small'
    instance_type = config.INSTANCE_TYPES['gpu_budget']  # ml.g4dn.xlarge
    
    print(f"Instance type: {instance_type}")
    
    # Create PyTorch estimator
    pytorch_estimator = PyTorch(
        entry_point='train_pytorch.py',
        role=config.SAGEMAKER_ROLE,
        instance_type=instance_type,
        instance_count=1,
        framework_version='2.0',
        py_version='py310',
        sagemaker_session=sagemaker_session,
        hyperparameters={
            'batch-size': 32,
            'epochs': 20,
            'learning-rate': 0.001,
            'hidden-dim': 128,
        },
        output_path=config.PATHS['models'],
        code_location=config.PATHS['code'],
        base_job_name='pytorch-nn',
        use_spot_instances=True,
        max_wait=7200,     # 2 hours
        max_run=3600,      # 1 hour training time
    )
    
    print("\nStarting training job...")
    print("Training with GPU acceleration...")
    
    pytorch_estimator.fit(wait=True)
    
    print(f"\nTraining job completed!")
    print(f"Model artifacts: {pytorch_estimator.model_data}")

if __name__ == '__main__':
    main()
```

### Monitoring Your Training Jobs

#### Via AWS CLI

Check status of all your training jobs:

```bash
aws sagemaker list-training-jobs --max-results 10
```

Get details of a specific job:

```bash
aws sagemaker describe-training-job --training-job-name sklearn-iris-2025-01-31-14-30-45
```

Stop a running job (if needed):

```bash
aws sagemaker stop-training-job --training-job-name job-name-here
```

#### Via AWS Console

1. Go to AWS Console → SageMaker → Training jobs
2. You'll see all your jobs with status, duration, and cost
3. Click on a job to see detailed logs and metrics

#### Via Python

Create `check_jobs.py`:

```python
"""
Check status of recent SageMaker training jobs
"""

import boto3
from datetime import datetime, timedelta

sagemaker = boto3.client('sagemaker', region_name='us-east-1')

# Get jobs from the last 7 days
response = sagemaker.list_training_jobs(
    MaxResults=20,
    SortBy='CreationTime',
    SortOrder='Descending'
)

print("Recent Training Jobs:")
print("-" * 80)

for job in response['TrainingJobSummaries']:
    name = job['TrainingJobName']
    status = job['TrainingJobStatus']
    created = job['CreationTime'].strftime('%Y-%m-%d %H:%M:%S')
    
    # Calculate duration if job is complete
    if status in ['Completed', 'Failed', 'Stopped']:
        duration = job['TrainingEndTime'] - job['CreationTime']
        duration_str = str(duration).split('.')[0]  # Remove microseconds
    else:
        duration_str = "In progress"
    
    print(f"\nJob: {name}")
    print(f"  Status: {status}")
    print(f"  Created: {created}")
    print(f"  Duration: {duration_str}")
    
    if status == 'Failed':
        print(f"  Failure Reason: {job.get('FailureReason', 'Unknown')}")
```

Run it:

```bash
python check_jobs.py
```

---

## Working with S3 Data

### Uploading Data to S3

Upload a dataset for training:

```bash
# Upload a single file
aws s3 cp my_data.csv s3://YOUR_BUCKET_NAME/datasets/my_data.csv

# Upload a directory
aws s3 cp ./local_data_folder s3://YOUR_BUCKET_NAME/datasets/my_project/ --recursive

# Upload with progress
aws s3 cp large_file.zip s3://YOUR_BUCKET_NAME/datasets/ --progress
```

### Downloading Data from S3

```bash
# Download a file
aws s3 cp s3://YOUR_BUCKET_NAME/models/my_model.tar.gz ./

# Download a directory
aws s3 cp s3://YOUR_BUCKET_NAME/datasets/my_project/ ./local_folder/ --recursive

# List files in a path
aws s3 ls s3://YOUR_BUCKET_NAME/datasets/
```

### Python S3 Operations

Create `s3_utils.py`:

```python
"""
Helper functions for S3 operations
"""

import boto3
import os
from pathlib import Path

s3 = boto3.client('s3', region_name='us-east-1')
BUCKET = 'YOUR_BUCKET_NAME'  # Replace with your lab's bucket name

def upload_file(local_path, s3_key):
    """Upload a file to S3"""
    print(f"Uploading {local_path} to s3://{BUCKET}/{s3_key}")
    s3.upload_file(local_path, BUCKET, s3_key)
    print("Upload complete!")
    return f's3://{BUCKET}/{s3_key}'

def download_file(s3_key, local_path):
    """Download a file from S3"""
    print(f"Downloading s3://{BUCKET}/{s3_key} to {local_path}")
    s3.download_file(BUCKET, s3_key, local_path)
    print("Download complete!")

def upload_directory(local_dir, s3_prefix):
    """Upload a directory to S3"""
    local_dir = Path(local_dir)
    for file_path in local_dir.rglob('*'):
        if file_path.is_file():
            relative_path = file_path.relative_to(local_dir)
            s3_key = f"{s3_prefix}/{relative_path}"
            upload_file(str(file_path), s3_key)

def list_files(s3_prefix):
    """List files in an S3 prefix"""
    response = s3.list_objects_v2(Bucket=BUCKET, Prefix=s3_prefix)
    
    if 'Contents' not in response:
        print(f"No files found in s3://{BUCKET}/{s3_prefix}")
        return []
    
    files = [obj['Key'] for obj in response['Contents']]
    print(f"Found {len(files)} files in s3://{BUCKET}/{s3_prefix}")
    for f in files:
        print(f"  - {f}")
    return files

# Example usage
if __name__ == '__main__':
    # List datasets
    print("\nDatasets:")
    list_files('datasets/')
    
    # List models
    print("\nModels:")
    list_files('models/')
```

---

## Troubleshooting

### AWS CLI Issues

**Problem**: `aws: command not found`

**Solution**: 
- Close and reopen terminal after installation
- Verify PATH: `echo $PATH` (Mac/Linux) or `echo %PATH%` (Windows)
- Reinstall AWS CLI

**Problem**: `Unable to locate credentials`

**Solution**:
```bash
# Reconfigure
aws configure

# Verify
aws sts get-caller-identity
```

### Claude Code Issues

**Problem**: Not using AWS Bedrock

**Solution**:
1. Check `.env` file exists and has correct content
2. Use startup script to load environment
3. Verify with `/status` command

**Problem**: `Access Denied` error

**Solution**: Contact Adam - your IAM permissions may need adjustment

### SageMaker Issues

**Problem**: `ValidationException: Could not find model data`

**Solution**: Training job may have failed. Check CloudWatch logs:
```bash
aws sagemaker describe-training-job --training-job-name YOUR_JOB_NAME
```

**Problem**: `ResourceLimitExceeded`

**Solution**: You've hit instance limits. Either:
- Wait for other jobs to complete
- Use a different instance type
- Contact Adam to request limit increase

**Problem**: Training job stuck in "Starting" status

**Solution**: 
- Check if you're using spot instances (may take longer to start)
- Verify IAM role has correct permissions
- Check CloudWatch logs for errors

**Problem**: S3 access denied during training

**Solution**:
- Verify S3 bucket name in config is correct
- Ensure SageMaker execution role has S3 permissions
- Contact Adam if issue persists

### Python/Package Issues

**Problem**: `ModuleNotFoundError: No module named 'sagemaker'`

**Solution**:
```bash
# Make sure virtual environment is activated
source venv/bin/activate  # Mac/Linux
venv\Scripts\activate     # Windows

# Reinstall packages
pip install sagemaker boto3
```

**Problem**: `ImportError` for ML libraries

**Solution**:
```bash
pip install --upgrade scikit-learn pandas numpy torch
```

---

## Best Practices

### Security

- **Never commit credentials**: Add `.env`, `*.pem`, `*.key`, and `credentials.csv` to `.gitignore`
- **Rotate credentials**: If you suspect they're compromised, contact Adam immediately
- **Use IAM roles**: Don't hardcode credentials in code
- **Review permissions**: Only access resources you need

### Cost Management

#### For Training Jobs

- **Start small**: Test with `ml.m5.large` before using GPU instances
- **Use spot instances**: Save 70-90% with `use_spot_instances=True`
- **Set timeouts**: Always set `max_run` and `max_wait` to prevent runaway jobs
- **Monitor actively**: Check jobs regularly and stop if not progressing
- **Clean up**: Delete unnecessary models and outputs from S3

#### Instance Type Selection

Choose the right instance for your task:

| Use Case | Instance Type | Cost/Hour | When to Use |
|----------|--------------|-----------|-------------|
| Testing code | ml.m5.large | $0.115 | Debugging, small datasets |
| CPU training | ml.m5.xlarge | $0.23 | Medium datasets, no GPU needed |
| Budget GPU | ml.g4dn.xlarge | $0.736 | Small neural networks |
| Single GPU | ml.p3.2xlarge | $3.825 | Standard deep learning |
| Multi-GPU | ml.p3.8xlarge | $14.688 | Large models, big datasets |

**Tip**: Run your first training on a small instance type to catch bugs cheaply!

### Code Organization

```
your-project/
├── .env                    # Environment config (don't commit!)
├── .gitignore             # Git ignore file
├── start-claude.sh        # Startup script
├── sagemaker_config.py    # SageMaker configuration
├── requirements.txt       # Python dependencies
├── data/                  # Local data (for testing)
├── scripts/               # Training scripts
│   ├── train.py
│   ├── train_pytorch.py
│   └── preprocess.py
├── notebooks/             # Jupyter notebooks (if used)
├── utils/                 # Helper functions
│   ├── s3_utils.py
│   └── evaluation.py
└── experiments/           # Experiment tracking
    └── run_training.py
```

### Experiment Tracking

Keep a log of your experiments:

```python
# experiments/experiment_log.py
import json
from datetime import datetime

class ExperimentLogger:
    def __init__(self, log_file='experiments.json'):
        self.log_file = log_file
    
    def log_experiment(self, job_name, hyperparameters, notes=''):
        experiment = {
            'timestamp': datetime.now().isoformat(),
            'job_name': job_name,
            'hyperparameters': hyperparameters,
            'notes': notes
        }
        
        # Append to log file
        try:
            with open(self.log_file, 'r') as f:
                experiments = json.load(f)
        except FileNotFoundError:
            experiments = []
        
        experiments.append(experiment)
        
        with open(self.log_file, 'w') as f:
            json.dump(experiments, f, indent=2)
        
        print(f"Logged experiment: {job_name}")

# Usage in your training script:
# logger = ExperimentLogger()
# logger.log_experiment(
#     job_name='pytorch-nn-2025-01-31',
#     hyperparameters={'lr': 0.001, 'batch_size': 32},
#     notes='First attempt with new architecture'
# )
```

### Version Control

```bash
# Initialize git if you haven't
git init

# Create .gitignore
cat > .gitignore << EOF
.env
*.pem
*.key
credentials.csv
venv/
__pycache__/
*.pyc
.ipynb_checkpoints/
experiments.json
data/
EOF

# First commit
git add .
git commit -m "Initial project setup"
```

---

## Quick Reference Card

### Starting Your Work Session

1. **Navigate to project folder**:
   ```bash
   cd ~/gormley-lab-project
   ```

2. **Activate Python environment**:
   ```bash
   source venv/bin/activate  # Mac/Linux
   venv\Scripts\activate     # Windows
   ```

3. **Start Claude Code (optional)**:
   ```bash
   ./start-claude.sh  # Mac/Linux
   start-claude.bat   # Windows
   ```

### Common Commands

```bash
# AWS
aws sts get-caller-identity              # Verify credentials
aws s3 ls s3://YOUR_BUCKET_NAME/  # List bucket contents
aws sagemaker list-training-jobs         # List training jobs

# Python
python run_training.py                   # Start training job
python check_jobs.py                     # Check job status
python s3_utils.py                       # S3 operations

# Claude Code
/status                                  # Check configuration
/help                                    # Show commands
exit                                     # Exit Claude Code
```

### SageMaker Job Template

```python
from sagemaker.sklearn import SKLearn
import sagemaker_config as config

estimator = SKLearn(
    entry_point='train.py',
    role=config.SAGEMAKER_ROLE,
    instance_type=config.INSTANCE_TYPES['cpu_small'],
    instance_count=1,
    framework_version='1.2-1',
    sagemaker_session=config.get_sagemaker_session(),
    hyperparameters={'param': value},
    output_path=config.PATHS['models'],
    use_spot_instances=True,
    max_run=1800,
)

estimator.fit()
```

### Cost-Saving Checklist

- [ ] Start with small instance type for testing
- [ ] Enable spot instances (`use_spot_instances=True`)
- [ ] Set `max_run` timeout
- [ ] Stop jobs that aren't progressing
- [ ] Delete old model artifacts from S3
- [ ] Use CPU instances for non-deep-learning tasks

---

## Getting Help

### Resources

- **AWS SageMaker Documentation**: https://docs.aws.amazon.com/sagemaker/
- **SageMaker Python SDK**: https://sagemaker.readthedocs.io/
- **Claude Code Documentation**: https://code.claude.ai/docs
- **Lab Wiki**: [Your lab's internal documentation]

### Who to Contact

- **AWS access issues**: Contact Adam
- **Billing questions**: Contact Adam
- **SageMaker permissions**: Contact Adam
- **Technical ML questions**: Ask in lab Slack/Discord
- **Claude Code usage**: Use `/help` or ask Claude Code directly

### Before Asking for Help

1. Check this guide's troubleshooting section
2. Review CloudWatch logs for your job
3. Try searching the error message
4. Check AWS service health: https://status.aws.amazon.com/

When reporting issues, include:
- What you were trying to do
- The exact error message
- Relevant job name or file path
- Steps you've already tried

---

## Next Steps

Now that you're set up:

1. **Run the example training jobs** to get familiar with the workflow
2. **Upload your own dataset** to S3
3. **Modify the training scripts** for your research needs
4. **Track your experiments** systematically
5. **Share your findings** with the lab

### Advanced Topics to Explore

- **Hyperparameter tuning**: Automatic optimization of model parameters
- **Distributed training**: Multi-GPU and multi-instance training
- **Model deployment**: Hosting models for inference
- **SageMaker Pipelines**: Automating ML workflows
- **Custom containers**: Using specialized Docker images

### Learning Resources

- **AWS SageMaker Examples**: https://github.com/aws/amazon-sagemaker-examples
- **Fast.ai course**: Practical deep learning
- **Machine Learning Mastery**: Great tutorials for applied ML

---

## Appendix: Complete File Structure

After following this guide, your project should look like:

```
gormley-lab-project/
├── .env                           # Environment variables
├── .gitignore                     # Git ignore rules
├── start-claude.sh                # Claude Code startup (Mac/Linux)
├── start-claude.bat               # Claude Code startup (Windows)
├── requirements.txt               # Python dependencies
├── sagemaker_config.py           # SageMaker configuration
├── train.py                      # Scikit-learn training script
├── train_pytorch.py              # PyTorch training script
├── run_training.py               # Launch scikit-learn job
├── run_pytorch_training.py       # Launch PyTorch job
├── check_jobs.py                 # Monitor training jobs
├── s3_utils.py                   # S3 helper functions
├── venv/                         # Python virtual environment
└── README.md                     # Project documentation
```

Happy training! 🚀