End of Year Statements PDF Processor

A web application for sorting and merging PDF statements by address and page count, optimized for processing large batches of documents.

Features

• Upload individual PDFs, ZIP archives, or entire folders
• Automatic address extraction and sorting
• Parallel processing for maximum speed
• Real-time progress tracking
• Web-based interface with progress bars and status updates
• Asynchronous processing to prevent timeouts
• Microsoft Entra (Azure AD) authentication for secure access

Authentication

This application requires Microsoft Entra (Azure Active Directory) authentication. Before running the application, you must:

1. Set up an Azure AD app registration
2. Configure authentication settings in .env
3. Grant necessary API permissions

See AZURE_AD_SETUP.md for detailed setup instructions.

Server Recommendations

VMware Virtual Machine Specifications

For optimal performance with large PDF batches (1000+ files), we recommend the following VM configuration:

Minimum Requirements

• vCPUs: 4 cores
• RAM: 8 GB
• Storage: 50 GB SSD
• OS: Ubuntu Server 22.04 LTS (or CentOS 8+)

Recommended for Production

• vCPUs: 8-16 cores (more cores = faster processing)
• RAM: 16-32 GB (PDFs load into memory)
• Storage: 100 GB NVMe SSD
• Network: 1 Gbps connection

High-Performance Setup (for 5000+ PDFs)

• vCPUs: 16-32 cores
• RAM: 64 GB
• Storage: 200 GB NVMe SSD
• OS: Ubuntu Server 22.04 LTS

VMware Configuration Tips

1. Enable CPU hot-add and memory hot-plug for future upgrades
2. Use VMXNET3 network adapter for better performance
3. Allocate storage from a high-performance datastore
4. Enable VMware Tools for optimal integration

Installation Instructions

1. VM Setup

1. Create a new VM in VMware with the specifications above
2. Install Ubuntu Server 22.04 LTS
3. Update the system:

   sudo apt update && sudo apt upgrade -y
   

2. Install Python and Dependencies

# Install Python 3.10+
sudo apt install python3 python3-pip python3-venv -y

# Create application directory
sudo mkdir -p /opt/pdf-processor
sudo chown $USER:$USER /opt/pdf-processor
cd /opt/pdf-processor

# Clone or copy the application files
# (Assuming you have the files ready)

# Create virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

3. Install Production Server (Gunicorn)

pip install gunicorn

# Create systemd service
sudo nano /etc/systemd/system/pdf-processor.service

Add the following content:

[Unit]
Description=PDF Processor Web App
After=network.target

[Service]
User=your-username
Group=your-group
WorkingDirectory=/opt/pdf-processor
Environment="PATH=/opt/pdf-processor/venv/bin"
ExecStart=/opt/pdf-processor/venv/bin/gunicorn -w 4 -b 0.0.0.0:8000 app:app
Restart=always

[Install]
WantedBy=multi-user.target

# Enable and start service
sudo systemctl enable pdf-processor
sudo systemctl start pdf-processor

# Check status
sudo systemctl status pdf-processor

4. Configure Nginx (Optional, for Production)

sudo apt install nginx -y

# Create nginx config
sudo nano /etc/nginx/sites-available/pdf-processor

Add:

server {
    listen 80;
    server_name your-server-ip;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

sudo ln -s /etc/nginx/sites-available/pdf-processor /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl restart nginx

5. Access the Application

• Open your web browser and navigate to http://your-vm-ip:8000 (or port 80 if using Nginx)
• Upload your PDF files and monitor the processing progress

Performance Tuning

Gunicorn Workers

Adjust the number of workers based on your CPU cores:

gunicorn -w $(nproc) -b 0.0.0.0:8000 app:app

Python Optimization

The app automatically uses all available CPU cores for parallel PDF processing. For very large batches, consider:

1. Increasing system file limits:

   sudo nano /etc/security/limits.conf
   # Add: * soft nofile 65536
   # Add: * hard nofile 65536
   

2. Tuning kernel parameters for better I/O:

   sudo nano /etc/sysctl.conf
   # Add: vm.dirty_ratio = 10
   # Add: vm.dirty_background_ratio = 5
   sudo sysctl -p
   

Troubleshooting

Common Issues

1. 413 Request Entity Too Large: The upload limit is set to 1000TB, but check client-side limits
2. Processing Stuck: Check server logs with sudo journalctl -u pdf-processor -f
3. Memory Issues: Monitor with htop and increase RAM if needed
4. Slow Processing: Ensure all CPU cores are being utilized

Monitoring

# Check CPU usage during processing
htop

# Monitor application logs
sudo journalctl -u pdf-processor -f

# Check disk I/O
iotop

Security Considerations

• Run behind a reverse proxy (Nginx) in production
• Use HTTPS with SSL certificates
• Restrict access with firewall rules
• Regularly update dependencies
• Monitor for resource usage

Support

For issues or questions, check the application logs and ensure your VM meets the minimum requirements.