348 lines
10 KiB
Markdown
348 lines
10 KiB
Markdown
# Import System V2 Architecture
|
|
|
|
## Overview
|
|
|
|
ImportServiceV2 is a refactored, database-driven import processing system that replaces the monolithic ImportProcessor.php with a modular, maintainable architecture.
|
|
|
|
## Key Features
|
|
|
|
- **Database-driven configuration**: Entity processing rules, validation, and handlers configured in `import_entities` table
|
|
- **Pluggable handlers**: Each entity type has its own handler class implementing `EntityHandlerInterface`
|
|
- **Queue support**: Large imports can be processed asynchronously via `ProcessLargeImportJob`
|
|
- **Validation**: Entity-level validation rules stored in database
|
|
- **Priority-based processing**: Entities processed in configured priority order
|
|
- **Extensible**: Easy to add new entity types without modifying core service
|
|
|
|
## Directory Structure
|
|
|
|
```
|
|
app/Services/Import/
|
|
├── Contracts/
|
|
│ └── EntityHandlerInterface.php # Handler contract
|
|
├── Handlers/
|
|
│ ├── ContractHandler.php # Contract entity handler
|
|
│ ├── AccountHandler.php # Account entity handler
|
|
│ ├── PaymentHandler.php # Payment handler (to be implemented)
|
|
│ ├── ActivityHandler.php # Activity handler (to be implemented)
|
|
│ └── ... # Additional handlers
|
|
├── BaseEntityHandler.php # Base handler with common logic
|
|
└── ImportServiceV2.php # Main import service
|
|
```
|
|
|
|
## Database Schema
|
|
|
|
### import_entities Table
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| id | bigint | Primary key |
|
|
| key | string | UI key (plural, e.g., "contracts") |
|
|
| canonical_root | string | Canonical root for processor (singular, e.g., "contract") |
|
|
| label | string | Human-readable label |
|
|
| fields | json | Array of field names |
|
|
| field_aliases | json | Field alias mappings |
|
|
| aliases | json | Root aliases |
|
|
| supports_multiple | boolean | Whether entity supports multiple items per row |
|
|
| meta | boolean | Whether entity is metadata |
|
|
| rules | json | Suggestion rules |
|
|
| ui | json | UI configuration |
|
|
| handler_class | string | Fully qualified handler class name |
|
|
| validation_rules | json | Laravel validation rules |
|
|
| processing_options | json | Handler-specific options |
|
|
| is_active | boolean | Whether entity is enabled |
|
|
| priority | integer | Processing priority (higher = first) |
|
|
| created_at | timestamp | Creation timestamp |
|
|
| updated_at | timestamp | Update timestamp |
|
|
|
|
## Handler Interface
|
|
|
|
All entity handlers must implement `EntityHandlerInterface`:
|
|
|
|
```php
|
|
interface EntityHandlerInterface
|
|
{
|
|
public function process(Import $import, array $mapped, array $raw, array $context = []): array;
|
|
public function validate(array $mapped): array;
|
|
public function getEntityClass(): string;
|
|
public function resolve(array $mapped, array $context = []): mixed;
|
|
}
|
|
```
|
|
|
|
### Handler Methods
|
|
|
|
- **process()**: Main processing method, returns result with action (inserted/updated/skipped) and entity
|
|
- **validate()**: Validates mapped data before processing
|
|
- **getEntityClass()**: Returns the model class name this handler manages
|
|
- **resolve()**: Resolves existing entity by key/reference
|
|
|
|
## Creating a New Handler
|
|
|
|
1. Create handler class extending `BaseEntityHandler`:
|
|
|
|
```php
|
|
<?php
|
|
|
|
namespace App\Services\Import\Handlers;
|
|
|
|
use App\Models\YourEntity;
|
|
use App\Models\Import;
|
|
use App\Services\Import\BaseEntityHandler;
|
|
|
|
class YourEntityHandler extends BaseEntityHandler
|
|
{
|
|
public function getEntityClass(): string
|
|
{
|
|
return YourEntity::class;
|
|
}
|
|
|
|
public function resolve(array $mapped, array $context = []): mixed
|
|
{
|
|
// Implement entity resolution logic
|
|
return YourEntity::where('key', $mapped['key'])->first();
|
|
}
|
|
|
|
public function process(Import $import, array $mapped, array $raw, array $context = []): array
|
|
{
|
|
$existing = $this->resolve($mapped, $context);
|
|
|
|
if ($existing) {
|
|
// Update logic
|
|
$payload = $this->buildPayload($mapped, $existing);
|
|
$appliedFields = $this->trackAppliedFields($existing, $payload);
|
|
|
|
if (empty($appliedFields)) {
|
|
return [
|
|
'action' => 'skipped',
|
|
'entity' => $existing,
|
|
'message' => 'No changes detected',
|
|
];
|
|
}
|
|
|
|
$existing->fill($payload);
|
|
$existing->save();
|
|
|
|
return [
|
|
'action' => 'updated',
|
|
'entity' => $existing,
|
|
'applied_fields' => $appliedFields,
|
|
];
|
|
}
|
|
|
|
// Create logic
|
|
$entity = new YourEntity;
|
|
$payload = $this->buildPayload($mapped, $entity);
|
|
$entity->fill($payload);
|
|
$entity->save();
|
|
|
|
return [
|
|
'action' => 'inserted',
|
|
'entity' => $entity,
|
|
'applied_fields' => array_keys($payload),
|
|
];
|
|
}
|
|
|
|
protected function buildPayload(array $mapped, $model): array
|
|
{
|
|
// Map fields to model attributes
|
|
return [
|
|
'field1' => $mapped['field1'] ?? null,
|
|
'field2' => $mapped['field2'] ?? null,
|
|
];
|
|
}
|
|
}
|
|
```
|
|
|
|
2. Add configuration to `import_entities` table:
|
|
|
|
```php
|
|
ImportEntity::create([
|
|
'key' => 'your_entities',
|
|
'canonical_root' => 'your_entity',
|
|
'label' => 'Your Entities',
|
|
'fields' => ['field1', 'field2'],
|
|
'handler_class' => \App\Services\Import\Handlers\YourEntityHandler::class,
|
|
'validation_rules' => [
|
|
'field1' => 'required|string',
|
|
'field2' => 'nullable|integer',
|
|
],
|
|
'processing_options' => [
|
|
'update_mode' => 'update',
|
|
],
|
|
'is_active' => true,
|
|
'priority' => 100,
|
|
]);
|
|
```
|
|
|
|
## Usage
|
|
|
|
### Synchronous Processing
|
|
|
|
```php
|
|
use App\Services\Import\ImportServiceV2;
|
|
|
|
$service = app(ImportServiceV2::class);
|
|
$results = $service->process($import, $user);
|
|
```
|
|
|
|
### Queue Processing (Large Imports)
|
|
|
|
```php
|
|
use App\Jobs\ProcessLargeImportJob;
|
|
|
|
ProcessLargeImportJob::dispatch($import, $user->id);
|
|
```
|
|
|
|
## Processing Options
|
|
|
|
Handler-specific options stored in `processing_options` JSON column:
|
|
|
|
### Contract Handler
|
|
- `update_mode`: 'update' | 'skip' | 'error'
|
|
- `create_missing`: boolean
|
|
|
|
### Account Handler
|
|
- `update_mode`: 'update' | 'skip'
|
|
- `require_contract`: boolean
|
|
|
|
### Payment Handler (planned)
|
|
- `deduplicate_by`: array of fields
|
|
- `create_booking`: boolean
|
|
- `create_activity`: boolean
|
|
|
|
## Migration Path
|
|
|
|
### Phase 1: Setup (Current)
|
|
- ✅ Create directory structure
|
|
- ✅ Add v2 columns to import_entities
|
|
- ✅ Create base interfaces and classes
|
|
- ✅ Implement ContractHandler and AccountHandler
|
|
- ✅ Create ProcessLargeImportJob
|
|
- ✅ Create seeder for entity configurations
|
|
|
|
### Phase 2: Implementation
|
|
- [ ] Implement remaining handlers (Payment, Activity, Person, Contacts)
|
|
- [ ] Add comprehensive tests
|
|
- [ ] Update controllers to use ImportServiceV2
|
|
- [ ] Add feature flag to toggle between v1 and v2
|
|
|
|
### Phase 3: Migration
|
|
- [ ] Run both systems in parallel
|
|
- [ ] Compare results and fix discrepancies
|
|
- [ ] Migrate all imports to v2
|
|
- [ ] Remove ImportProcessor.php (v1)
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
# Run migrations
|
|
php artisan migrate
|
|
|
|
# Seed entity configurations
|
|
php artisan db:seed --class=ImportEntitiesV2Seeder
|
|
|
|
# Run tests
|
|
php artisan test --filter=ImportServiceV2
|
|
```
|
|
|
|
## Benefits Over V1
|
|
|
|
1. **Maintainability**: Each entity has its own handler, easier to understand and modify
|
|
2. **Testability**: Handlers can be tested independently
|
|
3. **Extensibility**: New entities added without touching core service
|
|
4. **Configuration**: Business rules in database, no code deployment needed
|
|
5. **Queue Support**: Built-in queue support for large imports
|
|
6. **Validation**: Entity-level validation separate from processing logic
|
|
7. **Priority Control**: Process entities in configurable order
|
|
8. **Reusability**: Handlers can be reused across different import scenarios
|
|
|
|
## Simulation Service
|
|
|
|
ImportSimulationServiceV2 provides a way to preview what an import would do without persisting any data to the database. This is useful for:
|
|
- Validating mappings before processing
|
|
- Previewing create/update actions
|
|
- Detecting errors before running actual import
|
|
- Testing handler logic
|
|
|
|
### Usage
|
|
|
|
```php
|
|
use App\Services\Import\ImportSimulationServiceV2;
|
|
|
|
$service = app(ImportSimulationServiceV2::class);
|
|
|
|
// Simulate first 100 rows (default)
|
|
$result = $service->simulate($import);
|
|
|
|
// Simulate 50 rows with verbose output
|
|
$result = $service->simulate($import, limit: 50, verbose: true);
|
|
|
|
// Result structure:
|
|
// [
|
|
// 'success' => true,
|
|
// 'total_simulated' => 50,
|
|
// 'limit' => 50,
|
|
// 'summaries' => [
|
|
// 'contract' => ['create' => 10, 'update' => 5, 'skip' => 0, 'invalid' => 1],
|
|
// 'account' => ['create' => 20, 'update' => 3, 'skip' => 0, 'invalid' => 0],
|
|
// ],
|
|
// 'rows' => [
|
|
// [
|
|
// 'row_number' => 2,
|
|
// 'entities' => [
|
|
// 'contract' => [
|
|
// 'action' => 'update',
|
|
// 'reference' => 'CNT-001',
|
|
// 'existing_id' => 123,
|
|
// 'data' => ['reference', 'title', 'amount'],
|
|
// 'changes' => ['title' => ['old' => 'Old', 'new' => 'New']],
|
|
// ],
|
|
// ],
|
|
// 'warnings' => [],
|
|
// 'errors' => [],
|
|
// ],
|
|
// ],
|
|
// 'meta' => [
|
|
// 'has_header' => true,
|
|
// 'delimiter' => ',',
|
|
// 'mappings_count' => 8,
|
|
// ],
|
|
// ]
|
|
```
|
|
|
|
### CLI Command
|
|
|
|
```bash
|
|
# Simulate import with ID 123
|
|
php artisan import:simulate-v2 123
|
|
|
|
# Simulate with custom limit
|
|
php artisan import:simulate-v2 123 --limit=50
|
|
|
|
# Verbose mode shows field-level changes
|
|
php artisan import:simulate-v2 123 --verbose
|
|
```
|
|
|
|
### Action Types
|
|
|
|
- **create**: Entity doesn't exist, would be created
|
|
- **update**: Entity exists, would be updated
|
|
- **skip**: Entity exists but update_mode is 'skip'
|
|
- **invalid**: Validation failed
|
|
- **error**: Processing error occurred
|
|
|
|
### Comparison with V1 Simulation
|
|
|
|
| Feature | ImportSimulationService (V1) | ImportSimulationServiceV2 |
|
|
|---------|------------------------------|---------------------------|
|
|
| Handler-based | ❌ Hardcoded logic | ✅ Uses V2 handlers |
|
|
| Configuration | ❌ In code | ✅ From database |
|
|
| Validation | ❌ Manual | ✅ Handler validation |
|
|
| Extensibility | ❌ Modify service | ✅ Add handlers |
|
|
| Change detection | ✅ Yes | ✅ Yes |
|
|
| Priority ordering | ❌ Fixed | ✅ Configurable |
|
|
| Error handling | ✅ Basic | ✅ Comprehensive |
|
|
|
|
## Original ImportProcessor.php
|
|
|
|
The original file remains at `app/Services/ImportProcessor.php` and can be used as reference for implementing remaining handlers.
|