nate/eoy-pdf-sortandmerge
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update README with Ubuntu installer instructions and nginx large file upload configuration

Browse Source
This commit is contained in:
nmemmert committed 2025-11-06 08:49:34 -05:00
1 parent 6380577791
commit cc6c8e8455
1 file changed
+112 -41
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+112 -41
View File
@@ -54,7 +54,30 @@ For optimal performance with large PDF batches (1000+ files), we recommend the f
## Installation Instructions
### 1. VM Setup
### Option 1: Automated Ubuntu Installation (Recommended)
For Ubuntu servers, use the automated installation script that handles all dependencies, nginx configuration, and permissions:
```bash
# Download and run the installer
wget https://gitlab.com/trbc-dev/eoy-pdf-sortandmerge/-/raw/web-app/ubuntu-install.sh
chmod +x ubuntu-install.sh
./ubuntu-install.sh
```
The installer will:
- Install Python, nginx, and all dependencies
- Configure nginx with large file upload support (up to 100GB)
- Set up systemd service for automatic startup
- Configure Azure AD authentication
- Set proper file permissions for static assets
- Generate SSL certificates (optional)
**Important**: The installer supports uploads up to 100GB by default. If you need different limits, edit the nginx configuration after installation.
### Option 2: Manual Installation
#### 1. VM Setup
1. Create a new VM in VMware with the specifications above
2. Install Ubuntu Server 22.04 LTS
3. Update the system:
@@ -62,7 +85,7 @@ For optimal performance with large PDF batches (1000+ files), we recommend the f
sudo apt update && sudo apt upgrade -y
```
### 2. Install Python and Dependencies
#### 2. Install Python and Dependencies
```bash
# Install Python 3.10+
sudo apt install python3 python3-pip python3-venv -y
@@ -83,7 +106,7 @@ source venv/bin/activate
pip install -r requirements.txt
```
### 3. Install Production Server (Gunicorn)
#### 3. Install Production Server (Gunicorn)
```bash
pip install gunicorn
@@ -118,84 +141,132 @@ sudo systemctl start pdf-processor
sudo systemctl status pdf-processor
```
### 4. Configure Nginx (Optional, for Production)
### 4. Configure Nginx (Required for Production)
**Important**: Nginx must be configured to handle large file uploads. The default nginx configuration limits uploads to 1MB, which will cause "413 Request Entity Too Large" errors.
```bash
sudo apt install nginx -y
# Create nginx config
# Create nginx config with large file upload support
sudo nano /etc/nginx/sites-available/pdf-processor
```
Add:
Add the following configuration (adjust `client_max_body_size` based on your needs):
```nginx
server {
listen 80;
server_name your-server-ip;
# Large file upload settings - CRITICAL for PDF processing
client_max_body_size 100G; # Allow uploads up to 100GB
client_body_timeout 300s; # 5 minute timeout for uploads
client_header_timeout 60s; # Header timeout
# Security headers
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "no-referrer-when-downgrade" always;
add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;
location / {
proxy_pass http://127.0.0.1:8000;
proxy_pass http://127.0.0.1:5000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Timeout settings for long-running PDF processing
proxy_connect_timeout 300s;
proxy_send_timeout 300s;
proxy_read_timeout 300s;
# Disable buffering for large uploads
proxy_buffering off;
proxy_request_buffering off;
}
# Static files
location /static/ {
alias /path/to/your/app/static/;
expires 1y;
add_header Cache-Control "public, immutable";
}
}
```
```bash
# Enable the site
sudo ln -s /etc/nginx/sites-available/pdf-processor /etc/nginx/sites-enabled/
sudo rm -f /etc/nginx/sites-enabled/default
# Test configuration
sudo nginx -t
# Restart nginx
sudo systemctl restart nginx
```
### 5. Access the Application
- Open your web browser and navigate to `http://your-vm-ip:8000` (or port 80 if using Nginx)
- Upload your PDF files and monitor the processing progress
## Performance Tuning
### Gunicorn Workers
Adjust the number of workers based on your CPU cores:
```bash
gunicorn -w $(nproc) -b 0.0.0.0:8000 app:app
```
### Python Optimization
The app automatically uses all available CPU cores for parallel PDF processing. For very large batches, consider:
1. Increasing system file limits:
```bash
sudo nano /etc/security/limits.conf
# Add: * soft nofile 65536
# Add: * hard nofile 65536
```
2. Tuning kernel parameters for better I/O:
```bash
sudo nano /etc/sysctl.conf
# Add: vm.dirty_ratio = 10
# Add: vm.dirty_background_ratio = 5
sudo sysctl -p
```
- **With automated installer**: The installer will provide the exact URL and setup next steps
- **Manual installation**: Open your web browser and navigate to `http://your-vm-ip` (nginx on port 80)
- **Direct access**: `http://your-vm-ip:5000` (Flask development server)
## Troubleshooting
### Common Issues
1. **413 Request Entity Too Large**: The upload limit is set to 1000TB, but check client-side limits
2. **Processing Stuck**: Check server logs with `sudo journalctl -u pdf-processor -f`
3. **Memory Issues**: Monitor with `htop` and increase RAM if needed
4. **Slow Processing**: Ensure all CPU cores are being utilized
### Monitoring
1. **413 Request Entity Too Large**:
- **Cause**: Nginx upload limit is too small (default is 1MB)
- **Solution**: Ensure `client_max_body_size` is set to at least 100G in nginx config
- **Check**: `sudo nginx -t && sudo systemctl restart nginx`
2. **Permission Denied on Static Files**:
- **Cause**: Nginx can't access logo/images in static directory
- **Solution**: Run the automated installer or manually set permissions:
```bash
sudo usermod -a -G www-data $USER
sudo chmod -R 755 ~/pdf-sort-merge/static/
find ~/pdf-sort-merge/static/ -type f -exec sudo chmod 644 {} \;
sudo systemctl restart nginx
```
3. **Processing Stuck**:
- **Check logs**: `sudo journalctl -u pdf-sort-merge -f` (automated install)
- **Check logs**: `sudo journalctl -u pdf-processor -f` (manual install)
4. **Memory Issues**: Monitor with `htop` and increase RAM if needed
5. **Slow Processing**: Ensure all CPU cores are being utilized
### Installation Issues
- **Automated installer fails**: Check that you're running as a regular user (not root) with sudo privileges
- **Azure AD setup incomplete**: Follow [AZURE_AD_SETUP.md](AZURE_AD_SETUP.md) carefully
- **PyMuPDF installation fails**: The app will still work but PDF processing will be disabled
### Monitoring and Management
```bash
# Check CPU usage during processing
htop
# Monitor application logs
# Monitor application logs (automated install)
sudo journalctl -u pdf-sort-merge -f
# Monitor application logs (manual install)
sudo journalctl -u pdf-processor -f
# Check disk I/O
iotop
# Service management (automated install)
~/pdf-sort-merge/manage.sh status
~/pdf-sort-merge/manage.sh logs
~/pdf-sort-merge/manage.sh restart
~/pdf-sort-merge/manage.sh update # Pull latest code and restart
```
## Security Considerations