Complete PDF Sort and Merge Flask Application
- Full-featured web application for PDF processing - Microsoft Entra (Azure AD) authentication with PKCE - Parallel PDF sorting and merging capabilities - Job management system with real-time progress tracking - Modern web UI with responsive design - Comprehensive error handling and logging - Ubuntu installation script for easy deployment - Environment configuration for secure credential management
This commit is contained in:
nmemmert committed
1 parent
31c2d27a62
commit
144adf64c4
18 files changed
+2903
-2
No files matched your search
@@ -1,3 +1,209 @@
|
||||
# EOY PDF Sortandmerge
|
||||
# End of Year Statements PDF Processor
|
||||
|
||||
Project to sort, filter, and merge pdf files.
|
||||
A web application for sorting and merging PDF statements by address and page count, optimized for processing large batches of documents.
|
||||
|
||||
## Features
|
||||
|
||||
- Upload individual PDFs, ZIP archives, or entire folders
|
||||
- Automatic address extraction and sorting
|
||||
- Parallel processing for maximum speed
|
||||
- Real-time progress tracking
|
||||
- Web-based interface with progress bars and status updates
|
||||
- Asynchronous processing to prevent timeouts
|
||||
- **Microsoft Entra (Azure AD) authentication for secure access**
|
||||
|
||||
## Authentication
|
||||
|
||||
This application requires Microsoft Entra (Azure Active Directory) authentication. Before running the application, you must:
|
||||
|
||||
1. Set up an Azure AD app registration
|
||||
2. Configure authentication settings in `.env`
|
||||
3. Grant necessary API permissions
|
||||
|
||||
See [AZURE_AD_SETUP.md](AZURE_AD_SETUP.md) for detailed setup instructions.
|
||||
|
||||
## Server Recommendations
|
||||
|
||||
### VMware Virtual Machine Specifications
|
||||
|
||||
For optimal performance with large PDF batches (1000+ files), we recommend the following VM configuration:
|
||||
|
||||
#### Minimum Requirements
|
||||
- **vCPUs**: 4 cores
|
||||
- **RAM**: 8 GB
|
||||
- **Storage**: 50 GB SSD
|
||||
- **OS**: Ubuntu Server 22.04 LTS (or CentOS 8+)
|
||||
|
||||
#### Recommended for Production
|
||||
- **vCPUs**: 8-16 cores (more cores = faster processing)
|
||||
- **RAM**: 16-32 GB (PDFs load into memory)
|
||||
- **Storage**: 100 GB NVMe SSD
|
||||
- **Network**: 1 Gbps connection
|
||||
|
||||
#### High-Performance Setup (for 5000+ PDFs)
|
||||
- **vCPUs**: 16-32 cores
|
||||
- **RAM**: 64 GB
|
||||
- **Storage**: 200 GB NVMe SSD
|
||||
- **OS**: Ubuntu Server 22.04 LTS
|
||||
|
||||
### VMware Configuration Tips
|
||||
1. Enable CPU hot-add and memory hot-plug for future upgrades
|
||||
2. Use VMXNET3 network adapter for better performance
|
||||
3. Allocate storage from a high-performance datastore
|
||||
4. Enable VMware Tools for optimal integration
|
||||
|
||||
## Installation Instructions
|
||||
|
||||
### 1. VM Setup
|
||||
1. Create a new VM in VMware with the specifications above
|
||||
2. Install Ubuntu Server 22.04 LTS
|
||||
3. Update the system:
|
||||
```bash
|
||||
sudo apt update && sudo apt upgrade -y
|
||||
```
|
||||
|
||||
### 2. Install Python and Dependencies
|
||||
```bash
|
||||
# Install Python 3.10+
|
||||
sudo apt install python3 python3-pip python3-venv -y
|
||||
|
||||
# Create application directory
|
||||
sudo mkdir -p /opt/pdf-processor
|
||||
sudo chown $USER:$USER /opt/pdf-processor
|
||||
cd /opt/pdf-processor
|
||||
|
||||
# Clone or copy the application files
|
||||
# (Assuming you have the files ready)
|
||||
|
||||
# Create virtual environment
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 3. Install Production Server (Gunicorn)
|
||||
```bash
|
||||
pip install gunicorn
|
||||
|
||||
# Create systemd service
|
||||
sudo nano /etc/systemd/system/pdf-processor.service
|
||||
```
|
||||
|
||||
Add the following content:
|
||||
```ini
|
||||
[Unit]
|
||||
Description=PDF Processor Web App
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
User=your-username
|
||||
Group=your-group
|
||||
WorkingDirectory=/opt/pdf-processor
|
||||
Environment="PATH=/opt/pdf-processor/venv/bin"
|
||||
ExecStart=/opt/pdf-processor/venv/bin/gunicorn -w 4 -b 0.0.0.0:8000 app:app
|
||||
Restart=always
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
```bash
|
||||
# Enable and start service
|
||||
sudo systemctl enable pdf-processor
|
||||
sudo systemctl start pdf-processor
|
||||
|
||||
# Check status
|
||||
sudo systemctl status pdf-processor
|
||||
```
|
||||
|
||||
### 4. Configure Nginx (Optional, for Production)
|
||||
```bash
|
||||
sudo apt install nginx -y
|
||||
|
||||
# Create nginx config
|
||||
sudo nano /etc/nginx/sites-available/pdf-processor
|
||||
```
|
||||
|
||||
Add:
|
||||
```nginx
|
||||
server {
|
||||
listen 80;
|
||||
server_name your-server-ip;
|
||||
|
||||
location / {
|
||||
proxy_pass http://127.0.0.1:8000;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```bash
|
||||
sudo ln -s /etc/nginx/sites-available/pdf-processor /etc/nginx/sites-enabled/
|
||||
sudo nginx -t
|
||||
sudo systemctl restart nginx
|
||||
```
|
||||
|
||||
### 5. Access the Application
|
||||
- Open your web browser and navigate to `http://your-vm-ip:8000` (or port 80 if using Nginx)
|
||||
- Upload your PDF files and monitor the processing progress
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### Gunicorn Workers
|
||||
Adjust the number of workers based on your CPU cores:
|
||||
```bash
|
||||
gunicorn -w $(nproc) -b 0.0.0.0:8000 app:app
|
||||
```
|
||||
|
||||
### Python Optimization
|
||||
The app automatically uses all available CPU cores for parallel PDF processing. For very large batches, consider:
|
||||
|
||||
1. Increasing system file limits:
|
||||
```bash
|
||||
sudo nano /etc/security/limits.conf
|
||||
# Add: * soft nofile 65536
|
||||
# Add: * hard nofile 65536
|
||||
```
|
||||
|
||||
2. Tuning kernel parameters for better I/O:
|
||||
```bash
|
||||
sudo nano /etc/sysctl.conf
|
||||
# Add: vm.dirty_ratio = 10
|
||||
# Add: vm.dirty_background_ratio = 5
|
||||
sudo sysctl -p
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
1. **413 Request Entity Too Large**: The upload limit is set to 1000TB, but check client-side limits
|
||||
2. **Processing Stuck**: Check server logs with `sudo journalctl -u pdf-processor -f`
|
||||
3. **Memory Issues**: Monitor with `htop` and increase RAM if needed
|
||||
4. **Slow Processing**: Ensure all CPU cores are being utilized
|
||||
|
||||
### Monitoring
|
||||
```bash
|
||||
# Check CPU usage during processing
|
||||
htop
|
||||
|
||||
# Monitor application logs
|
||||
sudo journalctl -u pdf-processor -f
|
||||
|
||||
# Check disk I/O
|
||||
iotop
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
- Run behind a reverse proxy (Nginx) in production
|
||||
- Use HTTPS with SSL certificates
|
||||
- Restrict access with firewall rules
|
||||
- Regularly update dependencies
|
||||
- Monitor for resource usage
|
||||
|
||||
## Support
|
||||
For issues or questions, check the application logs and ensure your VM meets the minimum requirements.
|
||||
Reference in new issue
Block a user