eoy-pdf-sortandmerge/README.md

End of Year Statements PDF Processor

A web application for sorting and merging PDF statements by address and page count, optimized for processing large batches of documents.

Features

• Upload individual PDFs, ZIP archives, or entire folders
• Automatic address extraction and sorting
• Parallel processing for maximum speed
• Real-time progress tracking
• Web-based interface with progress bars and status updates
• Asynchronous processing to prevent timeouts
• Microsoft Entra (Azure AD) authentication for secure access

Authentication

This application requires Microsoft Entra (Azure Active Directory) authentication. Before running the application, you must:

1. Set up an Azure AD app registration
2. Configure authentication settings in .env
3. Grant necessary API permissions

See AZURE_AD_SETUP.md for detailed setup instructions.

Server Recommendations

VMware Virtual Machine Specifications

For optimal performance with large PDF batches (1000+ files), we recommend the following VM configuration:

Minimum Requirements

• vCPUs: 4 cores
• RAM: 8 GB
• Storage: 50 GB SSD
• OS: Ubuntu Server 22.04 LTS (or CentOS 8+)

Recommended for Production

• vCPUs: 8-16 cores (more cores = faster processing)
• RAM: 16-32 GB (PDFs load into memory)
• Storage: 100 GB NVMe SSD
• Network: 1 Gbps connection

High-Performance Setup (for 5000+ PDFs)

• vCPUs: 16-32 cores
• RAM: 64 GB
• Storage: 200 GB NVMe SSD
• OS: Ubuntu Server 22.04 LTS

VMware Configuration Tips

1. Enable CPU hot-add and memory hot-plug for future upgrades
2. Use VMXNET3 network adapter for better performance
3. Allocate storage from a high-performance datastore
4. Enable VMware Tools for optimal integration

Installation Instructions

Option 1: Automated Ubuntu Installation (Recommended)

For Ubuntu servers, use the automated installation script that handles all dependencies, nginx configuration, and permissions:

# Download and run the installer
wget https://gitlab.com/trbc-dev/eoy-pdf-sortandmerge/-/raw/web-app/ubuntu-install.sh
chmod +x ubuntu-install.sh
./ubuntu-install.sh

The installer will:

• Install Python, nginx, and all dependencies
• Configure nginx with large file upload support (up to 100GB)
• Set up systemd service for automatic startup
• Configure Azure AD authentication
• Set proper file permissions for static assets
• Generate SSL certificates (optional)

Important: The installer supports uploads up to 100GB by default. If you need different limits, edit the nginx configuration after installation.

Option 2: Manual Installation

1. VM Setup

1. Create a new VM in VMware with the specifications above
2. Install Ubuntu Server 22.04 LTS
3. Update the system:

   sudo apt update && sudo apt upgrade -y
   

2. Install Python and Dependencies

# Install Python 3.10+
sudo apt install python3 python3-pip python3-venv -y

# Create application directory
sudo mkdir -p /opt/pdf-processor
sudo chown $USER:$USER /opt/pdf-processor
cd /opt/pdf-processor

# Clone or copy the application files
# (Assuming you have the files ready)

# Create virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

3. Install Production Server (Gunicorn)

pip install gunicorn

# Create systemd service
sudo nano /etc/systemd/system/pdf-processor.service

Add the following content:

[Unit]
Description=PDF Processor Web App
After=network.target

[Service]
User=your-username
Group=your-group
WorkingDirectory=/opt/pdf-processor
Environment="PATH=/opt/pdf-processor/venv/bin"
ExecStart=/opt/pdf-processor/venv/bin/gunicorn -w 4 -b 0.0.0.0:8000 app:app
Restart=always

[Install]
WantedBy=multi-user.target

# Enable and start service
sudo systemctl enable pdf-processor
sudo systemctl start pdf-processor

# Check status
sudo systemctl status pdf-processor

4. Configure Nginx (Required for Production)

Important: Nginx must be configured to handle large file uploads. The default nginx configuration limits uploads to 1MB, which will cause "413 Request Entity Too Large" errors.

sudo apt install nginx -y

# Create nginx config with large file upload support
sudo nano /etc/nginx/sites-available/pdf-processor

Add the following configuration (adjust client_max_body_size based on your needs):

server {
    listen 80;
    server_name your-server-ip;

    # Large file upload settings - CRITICAL for PDF processing
    client_max_body_size 100G;        # Allow uploads up to 100GB
    client_body_timeout 300s;         # 5 minute timeout for uploads
    client_header_timeout 60s;        # Header timeout

    # Security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "no-referrer-when-downgrade" always;
    add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;

    location / {
        proxy_pass http://127.0.0.1:5000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Timeout settings for long-running PDF processing
        proxy_connect_timeout 300s;
        proxy_send_timeout 300s;
        proxy_read_timeout 300s;

        # Disable buffering for large uploads
        proxy_buffering off;
        proxy_request_buffering off;
    }

    # Static files
    location /static/ {
        alias /path/to/your/app/static/;
        expires 1y;
        add_header Cache-Control "public, immutable";
    }
}

# Enable the site
sudo ln -s /etc/nginx/sites-available/pdf-processor /etc/nginx/sites-enabled/
sudo rm -f /etc/nginx/sites-enabled/default

# Test configuration
sudo nginx -t

# Restart nginx
sudo systemctl restart nginx

5. Access the Application

• With automated installer: The installer will provide the exact URL and setup next steps
• Manual installation: Open your web browser and navigate to http://your-vm-ip (nginx on port 80)
• Direct access: http://your-vm-ip:5000 (Flask development server)

Troubleshooting

Common Issues

1. 413 Request Entity Too Large:

   • Cause: Nginx upload limit is too small (default is 1MB)
   • Solution: Ensure client_max_body_size is set to at least 100G in nginx config
   • Check: sudo nginx -t && sudo systemctl restart nginx

2. Permission Denied on Static Files:

   • Cause: Nginx can't access logo/images in static directory
   • Solution: Run the automated installer or manually set permissions:

     sudo usermod -a -G www-data $USER
     sudo chmod -R 755 ~/pdf-sort-merge/static/
     find ~/pdf-sort-merge/static/ -type f -exec sudo chmod 644 {} \;
     sudo systemctl restart nginx
     

3. Processing Stuck:

   • Check logs: sudo journalctl -u pdf-sort-merge -f (automated install)
   • Check logs: sudo journalctl -u pdf-processor -f (manual install)

4. Memory Issues: Monitor with htop and increase RAM if needed

5. Slow Processing: Ensure all CPU cores are being utilized

Installation Issues

• Automated installer fails: Check that you're running as a regular user (not root) with sudo privileges
• Azure AD setup incomplete: Follow AZURE_AD_SETUP.md carefully
• PyMuPDF installation fails: The app will still work but PDF processing will be disabled

Monitoring and Management

# Check CPU usage during processing
htop

# Monitor application logs (automated install)
sudo journalctl -u pdf-sort-merge -f

# Monitor application logs (manual install)
sudo journalctl -u pdf-processor -f

# Check disk I/O
iotop

# Service management (automated install)
~/pdf-sort-merge/manage.sh status
~/pdf-sort-merge/manage.sh logs
~/pdf-sort-merge/manage.sh restart
~/pdf-sort-merge/manage.sh update  # Pull latest code and restart

Security Considerations

• Run behind a reverse proxy (Nginx) in production
• Use HTTPS with SSL certificates
• Restrict access with firewall rules
• Regularly update dependencies
• Monitor for resource usage

Support

For issues or questions, check the application logs and ensure your VM meets the minimum requirements.