diff --git a/README.md b/README.md index 426f94a..e4a5dd3 100644 --- a/README.md +++ b/README.md @@ -54,7 +54,30 @@ For optimal performance with large PDF batches (1000+ files), we recommend the f ## Installation Instructions -### 1. VM Setup +### Option 1: Automated Ubuntu Installation (Recommended) + +For Ubuntu servers, use the automated installation script that handles all dependencies, nginx configuration, and permissions: + +```bash +# Download and run the installer +wget https://gitlab.com/trbc-dev/eoy-pdf-sortandmerge/-/raw/web-app/ubuntu-install.sh +chmod +x ubuntu-install.sh +./ubuntu-install.sh +``` + +The installer will: +- Install Python, nginx, and all dependencies +- Configure nginx with large file upload support (up to 100GB) +- Set up systemd service for automatic startup +- Configure Azure AD authentication +- Set proper file permissions for static assets +- Generate SSL certificates (optional) + +**Important**: The installer supports uploads up to 100GB by default. If you need different limits, edit the nginx configuration after installation. + +### Option 2: Manual Installation + +#### 1. VM Setup 1. Create a new VM in VMware with the specifications above 2. Install Ubuntu Server 22.04 LTS 3. Update the system: @@ -62,7 +85,7 @@ For optimal performance with large PDF batches (1000+ files), we recommend the f sudo apt update && sudo apt upgrade -y ``` -### 2. Install Python and Dependencies +#### 2. Install Python and Dependencies ```bash # Install Python 3.10+ sudo apt install python3 python3-pip python3-venv -y @@ -83,7 +106,7 @@ source venv/bin/activate pip install -r requirements.txt ``` -### 3. Install Production Server (Gunicorn) +#### 3. Install Production Server (Gunicorn) ```bash pip install gunicorn @@ -118,84 +141,132 @@ sudo systemctl start pdf-processor sudo systemctl status pdf-processor ``` -### 4. Configure Nginx (Optional, for Production) +### 4. Configure Nginx (Required for Production) + +**Important**: Nginx must be configured to handle large file uploads. The default nginx configuration limits uploads to 1MB, which will cause "413 Request Entity Too Large" errors. + ```bash sudo apt install nginx -y -# Create nginx config +# Create nginx config with large file upload support sudo nano /etc/nginx/sites-available/pdf-processor ``` -Add: +Add the following configuration (adjust `client_max_body_size` based on your needs): + ```nginx server { listen 80; server_name your-server-ip; + # Large file upload settings - CRITICAL for PDF processing + client_max_body_size 100G; # Allow uploads up to 100GB + client_body_timeout 300s; # 5 minute timeout for uploads + client_header_timeout 60s; # Header timeout + + # Security headers + add_header X-Frame-Options "SAMEORIGIN" always; + add_header X-XSS-Protection "1; mode=block" always; + add_header X-Content-Type-Options "nosniff" always; + add_header Referrer-Policy "no-referrer-when-downgrade" always; + add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always; + location / { - proxy_pass http://127.0.0.1:8000; + proxy_pass http://127.0.0.1:5000; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; + + # Timeout settings for long-running PDF processing + proxy_connect_timeout 300s; + proxy_send_timeout 300s; + proxy_read_timeout 300s; + + # Disable buffering for large uploads + proxy_buffering off; + proxy_request_buffering off; + } + + # Static files + location /static/ { + alias /path/to/your/app/static/; + expires 1y; + add_header Cache-Control "public, immutable"; } } ``` ```bash +# Enable the site sudo ln -s /etc/nginx/sites-available/pdf-processor /etc/nginx/sites-enabled/ +sudo rm -f /etc/nginx/sites-enabled/default + +# Test configuration sudo nginx -t + +# Restart nginx sudo systemctl restart nginx ``` ### 5. Access the Application -- Open your web browser and navigate to `http://your-vm-ip:8000` (or port 80 if using Nginx) -- Upload your PDF files and monitor the processing progress - -## Performance Tuning - -### Gunicorn Workers -Adjust the number of workers based on your CPU cores: -```bash -gunicorn -w $(nproc) -b 0.0.0.0:8000 app:app -``` - -### Python Optimization -The app automatically uses all available CPU cores for parallel PDF processing. For very large batches, consider: - -1. Increasing system file limits: - ```bash - sudo nano /etc/security/limits.conf - # Add: * soft nofile 65536 - # Add: * hard nofile 65536 - ``` - -2. Tuning kernel parameters for better I/O: - ```bash - sudo nano /etc/sysctl.conf - # Add: vm.dirty_ratio = 10 - # Add: vm.dirty_background_ratio = 5 - sudo sysctl -p - ``` +- **With automated installer**: The installer will provide the exact URL and setup next steps +- **Manual installation**: Open your web browser and navigate to `http://your-vm-ip` (nginx on port 80) +- **Direct access**: `http://your-vm-ip:5000` (Flask development server) ## Troubleshooting ### Common Issues -1. **413 Request Entity Too Large**: The upload limit is set to 1000TB, but check client-side limits -2. **Processing Stuck**: Check server logs with `sudo journalctl -u pdf-processor -f` -3. **Memory Issues**: Monitor with `htop` and increase RAM if needed -4. **Slow Processing**: Ensure all CPU cores are being utilized -### Monitoring +1. **413 Request Entity Too Large**: + - **Cause**: Nginx upload limit is too small (default is 1MB) + - **Solution**: Ensure `client_max_body_size` is set to at least 100G in nginx config + - **Check**: `sudo nginx -t && sudo systemctl restart nginx` + +2. **Permission Denied on Static Files**: + - **Cause**: Nginx can't access logo/images in static directory + - **Solution**: Run the automated installer or manually set permissions: + ```bash + sudo usermod -a -G www-data $USER + sudo chmod -R 755 ~/pdf-sort-merge/static/ + find ~/pdf-sort-merge/static/ -type f -exec sudo chmod 644 {} \; + sudo systemctl restart nginx + ``` + +3. **Processing Stuck**: + - **Check logs**: `sudo journalctl -u pdf-sort-merge -f` (automated install) + - **Check logs**: `sudo journalctl -u pdf-processor -f` (manual install) + +4. **Memory Issues**: Monitor with `htop` and increase RAM if needed + +5. **Slow Processing**: Ensure all CPU cores are being utilized + +### Installation Issues + +- **Automated installer fails**: Check that you're running as a regular user (not root) with sudo privileges +- **Azure AD setup incomplete**: Follow [AZURE_AD_SETUP.md](AZURE_AD_SETUP.md) carefully +- **PyMuPDF installation fails**: The app will still work but PDF processing will be disabled + +### Monitoring and Management + ```bash # Check CPU usage during processing htop -# Monitor application logs +# Monitor application logs (automated install) +sudo journalctl -u pdf-sort-merge -f + +# Monitor application logs (manual install) sudo journalctl -u pdf-processor -f # Check disk I/O iotop + +# Service management (automated install) +~/pdf-sort-merge/manage.sh status +~/pdf-sort-merge/manage.sh logs +~/pdf-sort-merge/manage.sh restart +~/pdf-sort-merge/manage.sh update # Pull latest code and restart ``` ## Security Considerations