Update README with Ubuntu installer instructions and nginx large file upload configuration

README.md

@@ -54,7 +54,30 @@ For optimal performance with large PDF batches (1000+ files), we recommend the f
 
 ## Installation Instructions
 
-### 1. VM Setup
+### Option 1: Automated Ubuntu Installation (Recommended)
+
+For Ubuntu servers, use the automated installation script that handles all dependencies, nginx configuration, and permissions:
+
+```bash
+# Download and run the installer
+wget https://gitlab.com/trbc-dev/eoy-pdf-sortandmerge/-/raw/web-app/ubuntu-install.sh
+chmod +x ubuntu-install.sh
+./ubuntu-install.sh
+```
+
+The installer will:
+- Install Python, nginx, and all dependencies
+- Configure nginx with large file upload support (up to 100GB)
+- Set up systemd service for automatic startup
+- Configure Azure AD authentication
+- Set proper file permissions for static assets
+- Generate SSL certificates (optional)
+
+**Important**: The installer supports uploads up to 100GB by default. If you need different limits, edit the nginx configuration after installation.
+
+### Option 2: Manual Installation
+
+#### 1. VM Setup
 1. Create a new VM in VMware with the specifications above
 2. Install Ubuntu Server 22.04 LTS
 3. Update the system:
@@ -62,7 +85,7 @@ For optimal performance with large PDF batches (1000+ files), we recommend the f
    sudo apt update && sudo apt upgrade -y
    ```
 
-### 2. Install Python and Dependencies
+#### 2. Install Python and Dependencies
 ```bash
 # Install Python 3.10+
 sudo apt install python3 python3-pip python3-venv -y
@@ -83,7 +106,7 @@ source venv/bin/activate
 pip install -r requirements.txt
 ```
 
-### 3. Install Production Server (Gunicorn)
+#### 3. Install Production Server (Gunicorn)
 ```bash
 pip install gunicorn
 
@@ -118,84 +141,132 @@ sudo systemctl start pdf-processor
 sudo systemctl status pdf-processor
 ```
 
-### 4. Configure Nginx (Optional, for Production)
+### 4. Configure Nginx (Required for Production)
+
+**Important**: Nginx must be configured to handle large file uploads. The default nginx configuration limits uploads to 1MB, which will cause "413 Request Entity Too Large" errors.
+
 ```bash
 sudo apt install nginx -y
 
-# Create nginx config
+# Create nginx config with large file upload support
 sudo nano /etc/nginx/sites-available/pdf-processor
 ```
 
-Add:
+Add the following configuration (adjust `client_max_body_size` based on your needs):
+
 ```nginx
 server {
     listen 80;
     server_name your-server-ip;
 
+    # Large file upload settings - CRITICAL for PDF processing
+    client_max_body_size 100G;        # Allow uploads up to 100GB
+    client_body_timeout 300s;         # 5 minute timeout for uploads
+    client_header_timeout 60s;        # Header timeout
+
+    # Security headers
+    add_header X-Frame-Options "SAMEORIGIN" always;
+    add_header X-XSS-Protection "1; mode=block" always;
+    add_header X-Content-Type-Options "nosniff" always;
+    add_header Referrer-Policy "no-referrer-when-downgrade" always;
+    add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;
+
     location / {
-        proxy_pass http://127.0.0.1:8000;
+        proxy_pass http://127.0.0.1:5000;
         proxy_set_header Host $host;
         proxy_set_header X-Real-IP $remote_addr;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header X-Forwarded-Proto $scheme;
+
+        # Timeout settings for long-running PDF processing
+        proxy_connect_timeout 300s;
+        proxy_send_timeout 300s;
+        proxy_read_timeout 300s;
+
+        # Disable buffering for large uploads
+        proxy_buffering off;
+        proxy_request_buffering off;
+    }
+
+    # Static files
+    location /static/ {
+        alias /path/to/your/app/static/;
+        expires 1y;
+        add_header Cache-Control "public, immutable";
     }
 }
 ```
 
 ```bash
+# Enable the site
 sudo ln -s /etc/nginx/sites-available/pdf-processor /etc/nginx/sites-enabled/
+sudo rm -f /etc/nginx/sites-enabled/default
+
+# Test configuration
 sudo nginx -t
+
+# Restart nginx
 sudo systemctl restart nginx
 ```
 
 ### 5. Access the Application
-- Open your web browser and navigate to `http://your-vm-ip:8000` (or port 80 if using Nginx)
+- **With automated installer**: The installer will provide the exact URL and setup next steps
-- Upload your PDF files and monitor the processing progress
+- **Manual installation**: Open your web browser and navigate to `http://your-vm-ip` (nginx on port 80)
-
+- **Direct access**: `http://your-vm-ip:5000` (Flask development server)
-## Performance Tuning
-
-### Gunicorn Workers
-Adjust the number of workers based on your CPU cores:
-```bash
-gunicorn -w $(nproc) -b 0.0.0.0:8000 app:app
-```
-
-### Python Optimization
-The app automatically uses all available CPU cores for parallel PDF processing. For very large batches, consider:
-
-1. Increasing system file limits:
-   ```bash
-   sudo nano /etc/security/limits.conf
-   # Add: * soft nofile 65536
-   # Add: * hard nofile 65536
-   ```
-
-2. Tuning kernel parameters for better I/O:
-   ```bash
-   sudo nano /etc/sysctl.conf
-   # Add: vm.dirty_ratio = 10
-   # Add: vm.dirty_background_ratio = 5
-   sudo sysctl -p
-   ```
 
 ## Troubleshooting
 
 ### Common Issues
-1. **413 Request Entity Too Large**: The upload limit is set to 1000TB, but check client-side limits
-2. **Processing Stuck**: Check server logs with `sudo journalctl -u pdf-processor -f`
-3. **Memory Issues**: Monitor with `htop` and increase RAM if needed
-4. **Slow Processing**: Ensure all CPU cores are being utilized
 
-### Monitoring
+1. **413 Request Entity Too Large**: 
+   - **Cause**: Nginx upload limit is too small (default is 1MB)
+   - **Solution**: Ensure `client_max_body_size` is set to at least 100G in nginx config
+   - **Check**: `sudo nginx -t && sudo systemctl restart nginx`
+
+2. **Permission Denied on Static Files**:
+   - **Cause**: Nginx can't access logo/images in static directory
+   - **Solution**: Run the automated installer or manually set permissions:
+     ```bash
+     sudo usermod -a -G www-data $USER
+     sudo chmod -R 755 ~/pdf-sort-merge/static/
+     find ~/pdf-sort-merge/static/ -type f -exec sudo chmod 644 {} \;
+     sudo systemctl restart nginx
+     ```
+
+3. **Processing Stuck**: 
+   - **Check logs**: `sudo journalctl -u pdf-sort-merge -f` (automated install)
+   - **Check logs**: `sudo journalctl -u pdf-processor -f` (manual install)
+
+4. **Memory Issues**: Monitor with `htop` and increase RAM if needed
+
+5. **Slow Processing**: Ensure all CPU cores are being utilized
+
+### Installation Issues
+
+- **Automated installer fails**: Check that you're running as a regular user (not root) with sudo privileges
+- **Azure AD setup incomplete**: Follow [AZURE_AD_SETUP.md](AZURE_AD_SETUP.md) carefully
+- **PyMuPDF installation fails**: The app will still work but PDF processing will be disabled
+
+### Monitoring and Management
+
 ```bash
 # Check CPU usage during processing
 htop
 
-# Monitor application logs
+# Monitor application logs (automated install)
+sudo journalctl -u pdf-sort-merge -f
+
+# Monitor application logs (manual install)
 sudo journalctl -u pdf-processor -f
 
 # Check disk I/O
 iotop
+
+# Service management (automated install)
+~/pdf-sort-merge/manage.sh status
+~/pdf-sort-merge/manage.sh logs
+~/pdf-sort-merge/manage.sh restart
+~/pdf-sort-merge/manage.sh update  # Pull latest code and restart
 ```
 
 ## Security Considerations