Distributed Marketplace Processing Architecture

1. System Profile and Target Stack

Architecture specification v2.2.0 targets Java 25, PHP 8.2.12, MySQL 8.0 and JavaScript.

The baseline cluster profile is defined around 13 active partners, 3 marketplaces per partner, 39 accounts and an average of 2,000 listings per account. Listing volume ranges from 20 to 20,000 and the initial cluster consists of 3 servers.

2. Leader Election, Lease and Split-Brain Protection

The cluster is designed to have at most one active leader at any time. Servers are hybrid-capable and may act as either Worker or Chief.

Leadership is lease-based rather than permanent. The specification defines a 15-second lease and a 5-second leader heartbeat. Each new leader election increments a monotonic fencing token; planning or write requests carrying an old token are designed to be rejected with HTTP 409.

The failover threshold is defined as 120 seconds. This limits the risk of a previously stalled leader resuming writes and creating split-brain behavior.

3. Capacity, Slot Isolation and Marketplace Quotas

Each server is designed for a maximum of 3 active tasks, while preventing multiple simultaneous jobs for the same marketplace on one server.

Cluster capacity grows with the N × 3 formula where N is the number of active servers. Three servers yield a theoretical ceiling of 9 concurrent tasks; adding a fourth server expands capacity to 12.

Marketplace-level cluster quotas are tied to the active server count so a single channel cannot consume all available cluster capacity.

4. Long-Running Tasks and Progressive Persistence

Large accounts may be assigned as one long-running crawl_task to a Worker. The task can run for hours and proceed as a stream across pages.

Instead of keeping the full result set in memory, the specification calls for periodic persistence through the PHP API. An example flush size is 100 records.

Atomic and idempotent progressive flush is still marked as a P0 pre-production item. It must not be presented as production-ready until implementation and verification are complete.

5. Task Heartbeat, Checkpoint and Resume

For long-running work, Workers are expected to send task heartbeat updates every 20 seconds carrying fields such as server_id, fencing_token, current_page, last_cursor_token and processed_count.

The checkpoint model is monotonic: processed_count, current_page and total_discovered_count must not move backwards. A stale heartbeat may still indicate liveness, but it must not overwrite newer progress.

After retry or reassignment, the task should resume from the latest valid checkpoint. Completing monotonic checkpoint updates in the PHP heartbeat endpoint is a P0 pre-production item.

6. Size-Aware Fair Scheduling

A 20-to-20,000 listing variance can cause small accounts to wait excessively under a simple FIFO scheduler.

The Aereo specification defines size-aware scheduling with aging protection to balance smaller jobs and accounts with high SLA delay. Cold-start discovery rules are also defined for accounts whose listing size is initially unknown.

The objective is to prevent the largest catalogs from continuously occupying every slot while still keeping new and small accounts fresh.

7. Java 25 Runtime and Concurrency

Java 25 virtual threads are the target model for HTTP- and pagination-heavy I/O operations, aiming to reduce conventional thread overhead and memory pressure during long-running network work.

The Java layer is not allowed to connect directly to MySQL; coordination and persistence are handled through the PHP REST API layer.

8. Service Security and Data Integrity

Task ownership and fencing checks are used to reject operations generated by stale Workers or former leaders.

Service calls define HMAC-SHA256, timestamp and nonce validation requirements to protect against replay and payload/method/path tampering.

Trusted-origin policy is restricted to the same scheme, host, effective port and trusted path prefix. Production HTTPS is required by the target policy and redirects are not automatically followed.

Prepared statements, transactions, FOR UPDATE / SKIP LOCKED and unique guards are part of the intended data-integrity model.

9. Test Strategy and Production Readiness Gate

Specification v2.2.0 states that CrawlTaskOrchestratorTest contains 42 executable diagnostic/contract tests and reports 42/42 PASS.

Some of these are deterministic contract simulations. The specification explicitly states that they do not replace real PHP + MySQL transaction/concurrency integration testing.

Before production readiness, initial election, failover boundaries, stale fencing, concurrent plan/acquire, heartbeat lease boundaries, retries, checkpoint regression, reassignment, idempotent completion and HMAC replay/tamper scenarios are defined as mandatory tests against real PHP endpoints and MySQL 8.

10. Open P0/P1 Work

P0: monotonic checkpoint update in the PHP heartbeat endpoint; atomic/idempotent 100-record progressive product flush; real PHP + MySQL concurrency/integration suite; production/test DB schema, index and unique-constraint verification.

P1 hardening: server-authoritative cluster policy and Java parity validation, task-completion single-flight/CAS, startup invariant validation, disabling insecure HTTP in production, and stronger composite identity/marketplace capability contracts where required.

P2 operational quality: CPU/RAM telemetry pipeline, typed API exception classes, configurable history capacity and improved nonce-cleanup strategy.