Payment Debugging Guide

This guide provides a systematic approach to investigating payment failures in Upvendo. Payments flow through multiple services and providers, so debugging requires tracing through several layers.

Start Here: The Diagnostics Tools

Before running any of the manual steps below, use the built-in tool runtime. Tools live in app/Tools/Read/ and app/Tools/Write/ and are auto-discovered into the registry by ToolServiceProvider::boot() (app/Providers/ToolServiceProvider.php:42). They are runnable from the CLI via php artisan tool:run (app/Console/Commands/ToolRunCommand.php).

Start with payment.diagnostics (app/Tools/Read/PaymentDiagnosticsTool.php) -- it self-describes as "Diagnose a payment: transaction state, provider status, webhook events, idempotency key state, and timeline. Use this as the first step when investigating payment failures."

Read-only tools (safe to run):

Tool name	Class
payment.diagnostics	App\Tools\Read\PaymentDiagnosticsTool
order.debug	App\Tools\Read\OrderDebugTool
webhook.inspect	App\Tools\Read\WebhookInspectTool
integration.health	App\Tools\Read\IntegrationHealthTool
queue.status	App\Tools\Read\QueueStatusTool
merchant.diagnostics	App\Tools\Read\MerchantDiagnosticsTool

Write tools (each declares a minimum role -- admin or super_admin):

Tool name	Class	Minimum role
payment.capture	App\Tools\Write\PaymentCaptureTool	super_admin
transaction.repair	App\Tools\Write\TransactionRepairTool	super_admin
webhook.replay	App\Tools\Write\WebhookReplayTool	admin
pos.resync	App\Tools\Write\POSResyncTool	admin
cache.clear	App\Tools\Write\CacheClearTool	admin

Quick Reference: Key Files

File	What to Look For
app/Services/Payment/PaymentService.php	Payment intent creation, terminal interaction
app/Services/Payment/PaymentCaptureService.php	Capture logic, post-payment processing
app/Services/Payment/TransactionService.php	Transaction creation, state management
app/Services/Orchestrators/KioskOrchestrator.php	Kiosk payment orchestration
app/Services/Orchestrators/OnlineOrderingOrchestrator.php	Online payment orchestration
app/Services/Orchestrators/VivaWebhookOrchestrator.php	Viva webhook processing
app/Services/Orchestrators/StripeWebhookOrchestrator.php	Stripe webhook processing
app/Services/Common/VivaWalletService.php	Viva Wallet API calls
app/Services/BackOffice/SquareIntegrationService.php	Square API calls

---

Investigation Flowchart

Step 1: Identify the Transaction

Start by finding the transaction in MongoDB:

// By order number (displayed to customer)
db.transactions.findOne({ order_no: "ORDER-123" })

// By idempotency key (from logs)
db.transactions.findOne({ "payment_snapshot.idempotency_key": "idemp_..." })

// By time range and location
db.transactions.find({
  location_id: ObjectId("..."),
  created_at: { $gte: ISODate("2025-03-27T10:00:00Z"), $lte: ISODate("2025-03-27T11:00:00Z") }
}).sort({ created_at: -1 })

Step 2: Check Transaction Status

Transaction status holds the capitalised App\Enums\OrderStatuses string values (app/Enums/OrderStatuses.php:8-18) -- a query for status: "unpaid" matches nothing.

Status	What It Means	Next Step
Pending	Transaction created, payment not initiated	Go to "Payment Not Initiated"
Awaiting Payment	Sent to terminal/checkout, waiting	Go to "Payment Stuck Awaiting"
Awaiting Capture	Preauthorized, not yet captured	Go to "Preauthorization Capture Failure"
Awaiting Invoice	Invoice/deferred payment method	Check the invoice flow
Complete	Payment captured successfully	Payment worked, check post-payment
Unpaid	Payment attempted but not verified	Go to "Payment Not Verified"
Cancelled	Transaction was cancelled	Check cancellation reason

The remaining enum cases (Queued, In Progress, Ready) are order-fulfilment statuses, not payment statuses.

Step 3: Check payment_snapshot

The payment_snapshot field contains provider-specific data:

// Viva Wallet fields
payment_snapshot: {
  idempotency_key: "idemp_...",
  sessionId: "idemp_...",
  transactionId: "...",      // Viva transaction ID
  orderCode: "...",           // Viva order code
  details: { ... },           // Full Viva transaction details
  sessionDetails: { ... },    // Terminal session info
  webhookData: { ... },       // Raw webhook payload
  capture: { ... },           // Capture response
  is_free: false,
  auto_success: false
}

// Square fields
payment_snapshot: {
  idempotency_key: "idemp_...",
  order_id: "...",
  webhookData: { status: "COMPLETED", ... }
}

---

Common Payment Failures

1. Payment Not Initiated (Stuck in Pending)

Symptoms: Transaction exists but terminal never prompted for payment.

Possible Causes:

• API error when calling PaymentService::processPaymentIntent()
• Terminal not reachable
• Device not properly configured

Investigation:

1. Check application logs around the transaction creation time
2. Look for errors from VivaWalletService::initiateTerminalSale() or SquareIntegrationService::createTerminalCheckout()
3. Verify the device has a valid terminal ID:
   • Viva: device.viva_wallet_terminal_id (app/RawModels/Device.php:54)
   • Stripe Terminal: device.stripe_terminal_id (app/RawModels/Device.php:70)
   • Square: device.external_data.square.device_code.terminal_device_id, read via Device::getSquareTerminalId() (app/RawModels/Device.php:213-216) -- there is no square_terminal_id field

2. Terminal Not Prompting for Payment

Symptoms: API call succeeds but physical terminal shows nothing.

Possible Causes:

• Terminal offline or disconnected
• Terminal paired to wrong account
• Terminal firmware needs update
• Network connectivity issue between terminal and provider

Investigation:

1. Check the terminal status in the provider's dashboard (Viva/Square)
2. Verify terminal serial number matches device configuration
3. Check if initiateTerminalSale() returned a successful response
4. For Viva: check if session was created (look at payment_snapshot.sessionId)

3. 409 Conflict on Terminal Sale

Symptoms: HttpException with 409 status from Viva Wallet API.

Cause: The idempotency key was already used for this terminal.

How the Code Handles It:

processVivaWalletPayment()
  -> initiateTerminalSale() throws 409
  -> Catch HttpException(409)
  -> updateIdempotencyKey() generates new key
  -> Retry initiateTerminalSale() with new key

If it still fails after retry:

• Check all_idempotency_keys on the transaction for key history
• The terminal may have a stuck session from a previous payment
• May need to manually abort via Viva dashboard

4. Payment Stuck in Awaiting Payment

Symptoms: Customer paid on terminal, but transaction never moved to Complete.

Possible Causes:

A. Webhook not received:

• Webhook URL misconfigured in provider dashboard
• Webhook signature verification failing
• Server returned error to webhook (provider stops retrying)

Investigation:

• Check provider dashboard for webhook delivery status
• Search logs for webhook-related errors
• Verify webhook secret in .env matches provider

B. Webhook received but capture failed:

• Search logs: "Payment capture failed" with the idempotency key
• Check for write conflicts (concurrent capture attempts) -- see Lifecycle Append Failures and Aborted Transactions under Diagnostic Commands: a quiet lifecycle append that hits a write conflict rethrows so the capture retries in a fresh session
• Verify tenant database resolution worked

C. Webhook received but transaction not found:

• Search logs: "Transaction not found for"
• Check if location ID in webhook data matches the transaction's location
• Verify tenant database was correctly resolved

5. Payment Shows "Unpaid" After Capture

Symptoms: capturePayment() ran but status is Unpaid instead of Complete.

Cause: The payment provider reported a non-success status.

For Viva Wallet (PaymentService.php:974-977, polling path):

$newStatus = match ($transactionDetails['statusId'] ?? null) {
    'F' => Enums\OrderStatuses::Complete->value,   // F = Finalized
    default => Enums\OrderStatuses::Unpaid->value  // Any other status
};

• Check payment_snapshot.details.statusId -- should be "F" for success
• If not "F", check the Viva Wallet dashboard for the actual transaction status
• The webhook path is a separate match on $webhookData['StatusId'] -- note the capital S (PaymentService.php:986-989) versus the polling path's statusId

For Square (PaymentService.php:926-929):

$newStatus = match ($webhookData['status'] ?? null) {
    'COMPLETED' => Enums\OrderStatuses::Complete->value,
    default => Enums\OrderStatuses::Unpaid->value
};

• Check payment_snapshot.webhookData.status -- should be "COMPLETED"

6. Duplicate Payment Capture

Symptoms: Post-payment actions (inventory, loyalty, KDS) run twice.

Cause: Two capture attempts race past the lock.

Prevention Mechanisms:

1. Distributed lock payment_verification_{idempotencyKey} -- 120s TTL on the capture path (PaymentCaptureService.php:364), 30s on the online-ordering verify path (OnlineOrderingOrchestrator.php:980)
2. Distributed lock in the Viva webhook job: payment_processing_{channel}_{key} (30s TTL -- ProcessVivaWebhookJob.php:178)
3. Viva duplicate detection: SHA-256 webhook hash comparison

Investigation:

• Check if capturePayment() was called from both webhook and polling
• Look for concurrent log entries with the same idempotency key
• Check whether processing exceeded the lock TTL (120s capture / 30s online verify)

7. Online Ordering Payment Not Verified

Symptoms: Customer completed payment on checkout page, but order never appears.

Investigation for Viva Wallet:

1. Check payment_snapshot.orderCode exists on the transaction
2. Search for POST /verify-payment in access logs
3. Check if Viva webhook (TRANSACTION_PAYMENT_CREATED) was received
4. Verify the transaction exists via Viva API: retrieveTransactionByOrderCode()

Investigation for Square:

1. Check Square dashboard for the payment status
2. Verify Square webhook was received
3. Check payment_snapshot.webhookData

8. Preauthorization Capture Failure

Symptoms: Online ordering payment preauthorized but capture fails.

Code Path: PaymentService::verifyPayment() -> VivaWalletService::captureTransaction()

Check:

• payment_snapshot.capture field for the capture response
• If capture.Success === false, the capture was rejected
• Log message: "Error capturing transaction: {idempotencyKey}"
• Common cause: preauth expired (too much time between auth and capture)

---

Diagnostic Commands

Check Transaction State

// Full transaction with payment details
db.transactions.findOne(
  { order_no: "ORDER-123" },
  { status: 1, payment_snapshot: 1, status_logs: 1, all_idempotency_keys: 1, invalid_idempotency_keys: 1, order_status: 1, receipt_no: 1 }
)

status_logs is the per-transaction lifecycle history (Transaction.php:95, read via getStatusLogs() at :789). Entries are appended by TransactionLifecycleLogService using the event names in app/Enums/TransactionLifecycleEvents.php -- it is usually the fastest way to see what actually happened and in what order.

Lifecycle Append Failures and Aborted Transactions

lifecycle_logs is a separate array from status_logs. It is written by the quiet lifecycle appenders -- TransactionLifecycleLogService::appendLifecycleQuietly() and TransactionRepository::appendLifecycleLogQuietly() -- which $push onto lifecycle_logs (TransactionRepository.php:1364).

That append is best-effort except when the failure means the caller's MongoDB transaction is already dead. Both wrappers classify the exception with App\Support\AbortedTransactionError::matches() and rethrow on a match, so the caller's executeWithTransactionRetry() (e.g. PaymentCaptureService.php:183) re-runs its closure in a fresh session instead of continuing inside a transaction whose writes can never commit. Every other failure is logged and stays quiet.

AbortedTransactionError matches four message substrings (app/Support/AbortedTransactionError.php:35-40):

• Write conflict
• WriteConflict
• TransientTransactionError
• has been aborted

Log strings to grep:

• TransactionLifecycleLogService: append failed (TransactionLifecycleLogService.php:87)
• TransactionRepository: failed to append lifecycle log (TransactionRepository.php:1393)

Either line followed by a slow capture ending in Transaction with { txnNumber: N } has been aborted is the signature this rethrow addresses (production 2026-07-26: append failed on a write conflict, then 3 retries and a 2.8s capture).

This classifier is deliberately not the same set as DBTransactionTrait::isRetryableTransactionError() (app/Traits/DBTransactionTrait.php:293-314). That one additionally matches replica-set failover and step-down errors -- NotWritablePrimary, PrimarySteppedDown, InterruptedAtShutdown, InterruptedDueToReplStateChange, TemporarilyUnavailable, UnknownTransactionCommitResult -- which can hit a plain non-transactional write and say nothing about a transaction. Those must not make a quiet appender rethrow.

Check for Stuck Locks

These entries live in the central mongodb database, collection caches (app/RawModels/Cache.php:9-11), and the key is stored in _id -- there is no key field, and they are not in the tenant DB you are attached to when inspecting db.transactions. Distributed locks always go to Mongo; plain cache entries (viva_webhook_recent_*, failed_sepa_invoice_*) short-circuit to Redis whenever the cache store is a RedisStore, and .env.example ships CACHE_STORE=redis -- check Redis first for those (app/Traits/CachingTrait.php:28 cacheDriverIsRedis(), short-circuits at :208putDatabaseCache() and :249 getDatabaseCache()).

// Find distributed locks related to payment (central `mongodb` DB)
db.caches.find({ _id: /^payment_verification_/ })
db.caches.find({ _id: /^payment_processing_/ })

// Clear stuck lock manually (use with caution)
db.caches.deleteOne({ _id: "payment_verification_idemp_xxx" })

Check Webhook Processing

// Viva webhook recent processing records (Redis when CACHE_STORE=redis)
db.caches.find({ _id: /^viva_webhook_recent_/ }).sort({ _id: -1 }).limit(10)

// Failed SEPA invoice cache (Redis when CACHE_STORE=redis)
db.caches.find({ _id: /^failed_sepa_invoice_/ })

Stuck Transactions

A scheduled command reconciles transactions that never reached a terminal state: pos:reconcile-stuck-transactions runs hourly, and pos:reconcile-stuck-transactions --prune runs daily at 03:00 (bootstrap/app.php:106-107). Records are tracked by the App\RawModels\StuckTransaction model.

---

Provider-Specific Debugging

Viva Wallet

Field	Where to Find	Purpose
sessionId	payment_snapshot.sessionId	Terminal session identifier
orderCode	payment_snapshot.orderCode	Viva order reference
transactionId	payment_snapshot.transactionId	Viva transaction reference
statusId	payment_snapshot.details.statusId	"F" = success, others = failure

Viva Status Codes:

• F = Finalized (success)
• A = Active (pending)
• E = Error
• C = Cancelled

Square

Field	Where to Find	Purpose
terminal_checkout_id	external_data.square.terminal_checkout_id	Square checkout reference
order_id	payment_snapshot.order_id	Square order reference
status	payment_snapshot.webhookData.status	"COMPLETED" = success

Square Terminal Statuses (app/Services/BackOffice/Square/SquareConstants.php:39-47):

• PENDING = Waiting for customer
• IN_PROGRESS = Customer interacting
• CANCEL_REQUESTED = Cancellation requested, not yet confirmed
• CANCELED = Cancelled (single "L" -- Square's spelling)
• COMPLETED = Payment successful

Stripe

Stripe covers two distinct flows:

Subscription billing -- for subscription payment issues:

1. Check Stripe dashboard for the subscription status
2. Look at invoice.payment_failed webhook handler
3. Check SEPA recovery logic in StripeWebhookOrchestrator

Stripe Terminal card payments -- PaymentService::processStripeTerminalPayment() (PaymentService.php:336) drives the reader identified by device.stripe_terminal_id, and the payment intent is finalised by StripeWebhookOrchestrator::handleSuccessfulTerminalPayment() (StripeWebhookOrchestrator.php:248), which resolves the device by stripe_terminal_id (:241).

---

Escalation Checklist

Before escalating a payment issue, gather:

1. Transaction _id and order_no
2. Transaction status and payment_snapshot contents
3. Device ID and location ID
4. Provider (Viva Wallet or Square)
5. Provider-side transaction/order reference
6. Relevant log entries (search by idempotency key)
7. Timestamp of the issue
8. Whether this is a one-off or recurring pattern