eoy-pdf-sortandmerge/README.md

# End of Year Statements PDF Processor

A web application for sorting and merging PDF statements by address and page count, optimized for processing large batches of documents.

## Features

- Upload individual PDFs, ZIP archives, or entire folders
- Automatic address extraction and sorting
- Parallel processing for maximum speed
- Real-time progress tracking
- Web-based interface with progress bars and status updates
- Asynchronous processing to prevent timeouts
- **Microsoft Entra (Azure AD) authentication for secure access**

## Authentication

This application requires Microsoft Entra (Azure Active Directory) authentication. Before running the application, you must:

1. Set up an Azure AD app registration
2. Configure authentication settings in `.env`
3. Grant necessary API permissions

See [AZURE_AD_SETUP.md](AZURE_AD_SETUP.md) for detailed setup instructions.

## Server Recommendations

### VMware Virtual Machine Specifications

For optimal performance with large PDF batches (1000+ files), we recommend the following VM configuration:

#### Minimum Requirements
- **vCPUs**: 4 cores
- **RAM**: 8 GB
- **Storage**: 50 GB SSD
- **OS**: Ubuntu Server 22.04 LTS (or CentOS 8+)

#### Recommended for Production
- **vCPUs**: 8-16 cores (more cores = faster processing)
- **RAM**: 16-32 GB (PDFs load into memory)
- **Storage**: 100 GB NVMe SSD
- **Network**: 1 Gbps connection

#### High-Performance Setup (for 5000+ PDFs)
- **vCPUs**: 16-32 cores
- **RAM**: 64 GB
- **Storage**: 200 GB NVMe SSD
- **OS**: Ubuntu Server 22.04 LTS

### VMware Configuration Tips
1. Enable CPU hot-add and memory hot-plug for future upgrades
2. Use VMXNET3 network adapter for better performance
3. Allocate storage from a high-performance datastore
4. Enable VMware Tools for optimal integration

## Installation Instructions

### Option 1: Automated Ubuntu Installation (Recommended)

For Ubuntu servers, use the automated installation script that handles all dependencies, nginx configuration, and permissions:

```bash
# Download and run the installer
wget https://gitlab.com/trbc-dev/eoy-pdf-sortandmerge/-/raw/web-app/ubuntu-install.sh
chmod +x ubuntu-install.sh
./ubuntu-install.sh
```

The installer will:
- Install Python, nginx, and all dependencies
- Configure nginx with large file upload support (up to 100GB)
- Set up systemd service for automatic startup
- Configure Azure AD authentication
- Set proper file permissions for static assets
- Generate SSL certificates (optional)

**Important**: The installer supports uploads up to 100GB by default. If you need different limits, edit the nginx configuration after installation.

### Option 2: Manual Installation

#### 1. VM Setup
1. Create a new VM in VMware with the specifications above
2. Install Ubuntu Server 22.04 LTS
3. Update the system:
   ```bash
   sudo apt update && sudo apt upgrade -y
   ```

#### 2. Install Python and Dependencies
```bash
# Install Python 3.10+
sudo apt install python3 python3-pip python3-venv -y

# Create application directory
sudo mkdir -p /opt/pdf-processor
sudo chown $USER:$USER /opt/pdf-processor
cd /opt/pdf-processor

# Clone or copy the application files
# (Assuming you have the files ready)

# Create virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt
```

#### 3. Install Production Server (Gunicorn)
```bash
pip install gunicorn

# Create systemd service
sudo nano /etc/systemd/system/pdf-processor.service
```

Add the following content:
```ini
[Unit]
Description=PDF Processor Web App
After=network.target

[Service]
User=your-username
Group=your-group
WorkingDirectory=/opt/pdf-processor
Environment="PATH=/opt/pdf-processor/venv/bin"
ExecStart=/opt/pdf-processor/venv/bin/gunicorn -w 4 -b 0.0.0.0:8000 app:app
Restart=always

[Install]
WantedBy=multi-user.target
```

```bash
# Enable and start service
sudo systemctl enable pdf-processor
sudo systemctl start pdf-processor

# Check status
sudo systemctl status pdf-processor
```

### 4. Configure Nginx (Required for Production)

**Important**: Nginx must be configured to handle large file uploads. The default nginx configuration limits uploads to 1MB, which will cause "413 Request Entity Too Large" errors.

```bash
sudo apt install nginx -y

# Create nginx config with large file upload support
sudo nano /etc/nginx/sites-available/pdf-processor
```

Add the following configuration (adjust `client_max_body_size` based on your needs):

```nginx
server {
    listen 80;
    server_name your-server-ip;

    # Large file upload settings - CRITICAL for PDF processing
    client_max_body_size 100G;        # Allow uploads up to 100GB
    client_body_timeout 300s;         # 5 minute timeout for uploads
    client_header_timeout 60s;        # Header timeout

    # Security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "no-referrer-when-downgrade" always;
    add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;

    location / {
        proxy_pass http://127.0.0.1:5000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Timeout settings for long-running PDF processing
        proxy_connect_timeout 300s;
        proxy_send_timeout 300s;
        proxy_read_timeout 300s;

        # Disable buffering for large uploads
        proxy_buffering off;
        proxy_request_buffering off;
    }

    # Static files
    location /static/ {
        alias /path/to/your/app/static/;
        expires 1y;
        add_header Cache-Control "public, immutable";
    }
}
```

```bash
# Enable the site
sudo ln -s /etc/nginx/sites-available/pdf-processor /etc/nginx/sites-enabled/
sudo rm -f /etc/nginx/sites-enabled/default

# Test configuration
sudo nginx -t

# Restart nginx
sudo systemctl restart nginx
```

### 5. Access the Application
- **With automated installer**: The installer will provide the exact URL and setup next steps
- **Manual installation**: Open your web browser and navigate to `http://your-vm-ip` (nginx on port 80)
- **Direct access**: `http://your-vm-ip:5000` (Flask development server)

## Troubleshooting

### Common Issues

1. **413 Request Entity Too Large**: 
   - **Cause**: Nginx upload limit is too small (default is 1MB)
   - **Solution**: Ensure `client_max_body_size` is set to at least 100G in nginx config
   - **Check**: `sudo nginx -t && sudo systemctl restart nginx`

2. **Permission Denied on Static Files**:
   - **Cause**: Nginx can't access logo/images in static directory
   - **Solution**: Run the automated installer or manually set permissions:
     ```bash
     sudo usermod -a -G www-data $USER
     sudo chmod -R 755 ~/pdf-sort-merge/static/
     find ~/pdf-sort-merge/static/ -type f -exec sudo chmod 644 {} \;
     sudo systemctl restart nginx
     ```

3. **Processing Stuck**: 
   - **Check logs**: `sudo journalctl -u pdf-sort-merge -f` (automated install)
   - **Check logs**: `sudo journalctl -u pdf-processor -f` (manual install)

4. **Memory Issues**: Monitor with `htop` and increase RAM if needed

5. **Slow Processing**: Ensure all CPU cores are being utilized

### Installation Issues

- **Automated installer fails**: Check that you're running as a regular user (not root) with sudo privileges
- **Azure AD setup incomplete**: Follow [AZURE_AD_SETUP.md](AZURE_AD_SETUP.md) carefully
- **PyMuPDF installation fails**: The app will still work but PDF processing will be disabled

### Monitoring and Management

```bash
# Check CPU usage during processing
htop

# Monitor application logs (automated install)
sudo journalctl -u pdf-sort-merge -f

# Monitor application logs (manual install)
sudo journalctl -u pdf-processor -f

# Check disk I/O
iotop

# Service management (automated install)
~/pdf-sort-merge/manage.sh status
~/pdf-sort-merge/manage.sh logs
~/pdf-sort-merge/manage.sh restart
~/pdf-sort-merge/manage.sh update  # Pull latest code and restart
```

## Security Considerations
- Run behind a reverse proxy (Nginx) in production
- Use HTTPS with SSL certificates
- Restrict access with firewall rules
- Regularly update dependencies
- Monitor for resource usage

## Support
For issues or questions, check the application logs and ensure your VM meets the minimum requirements.