eoy-pdf-sortandmerge/README.md

# End of Year Statements PDF Processor

A web application for sorting and merging PDF statements by address and page count, optimized for processing large batches of documents.

## Features

- Upload individual PDFs, ZIP archives, or entire folders
- Automatic address extraction and sorting
- Parallel processing for maximum speed
- Real-time progress tracking
- Web-based interface with progress bars and status updates
- Asynchronous processing to prevent timeouts
- **Microsoft Entra (Azure AD) authentication for secure access**

## Authentication

This application requires Microsoft Entra (Azure Active Directory) authentication. Before running the application, you must:

1. Set up an Azure AD app registration
2. Configure authentication settings in `.env`
3. Grant necessary API permissions

See [AZURE_AD_SETUP.md](AZURE_AD_SETUP.md) for detailed setup instructions.

## Server Recommendations

### VMware Virtual Machine Specifications

For optimal performance with large PDF batches (1000+ files), we recommend the following VM configuration:

#### Minimum Requirements
- **vCPUs**: 4 cores
- **RAM**: 8 GB
- **Storage**: 50 GB SSD
- **OS**: Ubuntu Server 22.04 LTS (or CentOS 8+)

#### Recommended for Production
- **vCPUs**: 8-16 cores (more cores = faster processing)
- **RAM**: 16-32 GB (PDFs load into memory)
- **Storage**: 100 GB NVMe SSD
- **Network**: 1 Gbps connection

#### High-Performance Setup (for 5000+ PDFs)
- **vCPUs**: 16-32 cores
- **RAM**: 64 GB
- **Storage**: 200 GB NVMe SSD
- **OS**: Ubuntu Server 22.04 LTS

### VMware Configuration Tips
1. Enable CPU hot-add and memory hot-plug for future upgrades
2. Use VMXNET3 network adapter for better performance
3. Allocate storage from a high-performance datastore
4. Enable VMware Tools for optimal integration

## Installation Instructions

### 1. VM Setup
1. Create a new VM in VMware with the specifications above
2. Install Ubuntu Server 22.04 LTS
3. Update the system:
   ```bash
   sudo apt update && sudo apt upgrade -y
   ```

### 2. Install Python and Dependencies
```bash
# Install Python 3.10+
sudo apt install python3 python3-pip python3-venv -y

# Create application directory
sudo mkdir -p /opt/pdf-processor
sudo chown $USER:$USER /opt/pdf-processor
cd /opt/pdf-processor

# Clone or copy the application files
# (Assuming you have the files ready)

# Create virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt
```

### 3. Install Production Server (Gunicorn)
```bash
pip install gunicorn

# Create systemd service
sudo nano /etc/systemd/system/pdf-processor.service
```

Add the following content:
```ini
[Unit]
Description=PDF Processor Web App
After=network.target

[Service]
User=your-username
Group=your-group
WorkingDirectory=/opt/pdf-processor
Environment="PATH=/opt/pdf-processor/venv/bin"
ExecStart=/opt/pdf-processor/venv/bin/gunicorn -w 4 -b 0.0.0.0:8000 app:app
Restart=always

[Install]
WantedBy=multi-user.target
```

```bash
# Enable and start service
sudo systemctl enable pdf-processor
sudo systemctl start pdf-processor

# Check status
sudo systemctl status pdf-processor
```

### 4. Configure Nginx (Optional, for Production)
```bash
sudo apt install nginx -y

# Create nginx config
sudo nano /etc/nginx/sites-available/pdf-processor
```

Add:
```nginx
server {
    listen 80;
    server_name your-server-ip;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
```

```bash
sudo ln -s /etc/nginx/sites-available/pdf-processor /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl restart nginx
```

### 5. Access the Application
- Open your web browser and navigate to `http://your-vm-ip:8000` (or port 80 if using Nginx)
- Upload your PDF files and monitor the processing progress

## Performance Tuning

### Gunicorn Workers
Adjust the number of workers based on your CPU cores:
```bash
gunicorn -w $(nproc) -b 0.0.0.0:8000 app:app
```

### Python Optimization
The app automatically uses all available CPU cores for parallel PDF processing. For very large batches, consider:

1. Increasing system file limits:
   ```bash
   sudo nano /etc/security/limits.conf
   # Add: * soft nofile 65536
   # Add: * hard nofile 65536
   ```

2. Tuning kernel parameters for better I/O:
   ```bash
   sudo nano /etc/sysctl.conf
   # Add: vm.dirty_ratio = 10
   # Add: vm.dirty_background_ratio = 5
   sudo sysctl -p
   ```

## Troubleshooting

### Common Issues
1. **413 Request Entity Too Large**: The upload limit is set to 1000TB, but check client-side limits
2. **Processing Stuck**: Check server logs with `sudo journalctl -u pdf-processor -f`
3. **Memory Issues**: Monitor with `htop` and increase RAM if needed
4. **Slow Processing**: Ensure all CPU cores are being utilized

### Monitoring
```bash
# Check CPU usage during processing
htop

# Monitor application logs
sudo journalctl -u pdf-processor -f

# Check disk I/O
iotop
```

## Security Considerations
- Run behind a reverse proxy (Nginx) in production
- Use HTTPS with SSL certificates
- Restrict access with firewall rules
- Regularly update dependencies
- Monitor for resource usage

## Support
For issues or questions, check the application logs and ensure your VM meets the minimum requirements.