Troubleshooting

A practical runbook for diagnosing and recovering from e-invoicing failures. For background on the system design, see the Architecture. For Invopop API specifics, see the Invopop Integration.

Submission Failures

Symptoms

• Document stuck in issuing state

• Logger.error from invopop_http_client.ex in logs

• No webhook ever arrives

What Happens

When submit_invoice is called (Step 3 in the end-to-end flow), the Invopop HTTP client (invopop_http_client.ex) makes two calls: PUT /silo/v1/entries/{id} then PUT /transform/v1/jobs/{id}.

HTTP errors are mapped to tagged tuples:

HTTP status	Mapped to	Meaning
200-299	{:ok, body}	Success
401	{:error, :unauthorized}	Bad or expired API token
409	{:error, :conflict, body}	Silo entry already exists (use PATCH to update)
422	{:error, :unprocessable_entity, body}	GOBL validation failed
Other 4xx/5xx	{:error, :unexpected, %{status: ..., body: ...}}	Unexpected error

The Req HTTP adapter retries safe transient errors automatically (via retry: :safe_transient). There is no application-level retry loop for submission failures. If the SendToTaxAuthority command fails, the document stays in issuing state indefinitely.

Recovery Steps

1. Check logs for the HTTP client error: search for [EInvoicing] or invopop in error-level logs

2. Check the Invopop Console at console.invopop.com to see if the Silo entry and/or job were created

3. Fix the root cause (expired token, malformed GOBL, etc.)

4. Re-dispatch the command from the staging console (see Console Recipes below)

---

Webhook Processing Failures

Symptoms

• Invopop Console shows the job completed, but the document is still in issuing state in Abacus

• Kafka consumer lag increasing on the webhooks_incoming_invopop topic

• Kafka.Events.Pipeline.Blocked event emitted (visible in monitoring)

What Happens

The Kafka handler (Kafka.Handler.Webhooks.Incoming.Invopop) is wrapped with WithRetry, which provides exponential backoff:

• Initial delay: 1 second

• Factor: 2x

• Cap: 50 seconds

• Retries indefinitely until the message processes successfully

Validation failures in the handler raise exceptions (never return :ok), so they are always retried. This is by design — returning :ok on failure would silently ack and lose the message.

Common Causes

• Missing required fields in the webhook payload (schema mismatch)

• Database connectivity issues preventing event dispatch

• Downstream event handler failures (the dispatch handler or provider implementation)

Recovery Steps

1. Check Kafka consumer lag for the webhooks_incoming_invopop topic

2. Check logs for [Kafka.Handler.Webhooks.Incoming.Invopop] at error level

3. Check if Pipeline.Blocked event was emitted — this means the handler has been retrying for a while

4. If the issue is transient (DB connection), it will self-heal once the dependency recovers

5. If the issue is a code bug, deploy a fix — the message will be retried automatically

---

Silent Failure on Success Webhook

Symptoms

• Invopop Console shows the invoice was accepted by Verifactu

• The Kafka webhook message was processed (no consumer lag)

• The document is still in issuing state in Abacus — not marked as issued

This is the most insidious failure mode because there are no obvious errors.

What Happens

When Provider.on_invoice_event/3 receives a success webhook, it enters a with chain that:

1. Fetches the full Silo entry from Invopop (get_silo_entry/2)

2. Extracts the QR code URL from the entry stamps (fetch_qr_url/1)

3. Builds and dispatches the MarkAsAcceptedByTaxAuthority command

If step 1 or step 2 fails (e.g. the Invopop API is temporarily down, or the entry lacks the expected stamps), the function returns an error tuple without dispatching the command. Depending on how the error propagates through the event handler chain, it may or may not cause a retry.

Recovery Steps

1. Check the Invopop Console for the Silo entry — verify it has the expected stamps (verifactu-qr, verifactu-hash)

2. Fetch the entry manually via the Invopop API to see if the QR URL is present:

curl "https://api.invopop.com/silo/v1/entries/${ENTRY_UUID}" \
  -H "Authorization: Bearer ${API_TOKEN}"

1. Look for stamps in data.head.stamps[] — the verifactu-qr stamp should contain a URL

2. If the entry is valid, manually dispatch MarkAsAcceptedByTaxAuthority from the staging console (see recipes below)

3. If the entry is missing stamps, check the Invopop workflow status and contact Invopop support

---

Stuck in Issuing State

Symptoms

• Document has issuing_at set but no issued_at or rejected_by_tax_authority_at

• All modification commands on the document are rejected

• No webhook has been received for this document

Why This Happens

There is no timeout or watchdog for the issuing state. If Invopop never sends a webhook (due to a bug, network issue, or misconfigured workflow), the document remains locked indefinitely. No scheduled job exists to detect this.

How to Identify Stuck Documents

From the Abacus staging console:

alias Accounting.Application.ReadModels.Document, as: DocRM
import Ecto.Query

Shared.Infrastructure.Repo.all(
  from d in DocRM,
  where: not is_nil(d.issuing_at) and is_nil(d.issued_at) and is_nil(d.rejected_by_tax_authority_at),
  select: %{id: d.id, issuing_at: d.issuing_at, e_invoicing_reference_id: d.e_invoicing_reference_id}
)

Recovery Steps

1. Check the Invopop Console for the job/entry status using the e_invoicing_reference_id

2. If the job succeeded in Invopop but the webhook was lost:

   • Fetch the Silo entry to get the QR URL and hash

   • Manually dispatch MarkAsAcceptedByTaxAuthority (see recipes below)

3. If the job failed in Invopop:

   • Check the fault details in the Invopop Console

   • Manually dispatch MarkAsRejectedByTaxAuthority with the rejection reason

4. If the job does not exist in Invopop (submission failed silently):

   • Re-dispatch SendToTaxAuthority to resubmit

---

Common Invopop API Errors

Error	Cause	What to do
invalid check digit	Invalid Spanish CIF/NIF	Check the customer's tax ID in LVDAM. Fix the ID and resubmit
invalid format	Malformed tax ID	Verify the CIF/NIF format matches [A-Z]\d{8} (CIF) or \d{8}[A-Z] (NIF)
request duplicated	Same supplier + invoice code combination	LVDAM assigns invoice numbers — check if a previous submission used the same code. Use a new invoice number
party is not registered	Supplier not registered with Invopop	Complete the supplier registration flow in Invopop Console
unexpected data error	Bypass mode used with a credit note	Known Invopop bug. Submit credit notes without the bypass tag
schema validation	Missing required fields in bypass mode	Ensure all ext fields are present on both line-level taxes and totals-level rates
No existe el registro de facturacion (3002)	Changed invoice code after Verifactu accepted the original	Cannot change the code after acceptance. Issue a credit note to correct
entry already exists with same id	PUT on an existing Silo entry	Use PATCH to update the entry, or generate a new UUID v7
401 Unauthorized	Expired or invalid API token	Check the workspace e-invoicing configuration. Rotate the token in Invopop Console and update the workspace config
Network timeout	Invopop API unreachable	The Req adapter retries transient errors automatically. If persistent, check Invopop's status page

---

Staging Console Recipes

Connect to the staging Abacus console:

./bin/staging/console.sh

Check Workspace E-Invoicing Configuration

alias Accounting.Application.ReadModels.WorkspaceEInvoicingConfiguration, as: Config

Config |> Shared.Infrastructure.Repo.get("WORKSPACE_ID")

To list all configured workspaces:

import Ecto.Query

Shared.Infrastructure.Repo.all(
  from c in Config,
  where: not is_nil(c.e_invoicing_provider_type),
  select: %{id: c.id, provider: c.e_invoicing_provider_type}
)

Find Documents Stuck in Issuing

alias Accounting.Application.ReadModels.Document, as: DocRM
import Ecto.Query

Shared.Infrastructure.Repo.all(
  from d in DocRM,
  where: not is_nil(d.issuing_at) and is_nil(d.issued_at) and is_nil(d.rejected_by_tax_authority_at),
  select: %{
    id: d.id,
    issuing_at: d.issuing_at,
    e_invoicing_reference_id: d.e_invoicing_reference_id,
    workspace_id: d.workspace_id
  }
)

Manually Dispatch MarkAsAcceptedByTaxAuthority

Use this when a document is stuck in issuing but Invopop confirms it was accepted.

alias Accounting.Domain.Commands.Document.MarkAsAcceptedByTaxAuthority
alias Shared.Infrastructure.CommandBus

command = MarkAsAcceptedByTaxAuthority.new(
  id: "DOCUMENT_ID",
  accepted_at: DateTime.utc_now(),
  qr_url: "https://prewww2.aeat.es/...",
  submission_status: "complete"
)

CommandBus.dispatch(command, user_id: "manual-recovery", caused_by: :none)

To get the QR URL, fetch the Silo entry from the Invopop API and look in data.head.stamps[] for the verifactu-qr stamp. Use the timestamp from the webhook or Invopop entry for accepted_at, and submission_status from the job status (typically "complete").

Manually Dispatch MarkAsRejectedByTaxAuthority

Use this when a document is stuck in issuing and the Invopop job failed.

alias Accounting.Domain.Commands.Document.MarkAsRejectedByTaxAuthority
alias Shared.Infrastructure.CommandBus

command = MarkAsRejectedByTaxAuthority.new(
  id: "DOCUMENT_ID",
  rejected_at: DateTime.utc_now(),
  rejection_reason: "Description of the failure from Invopop"
)

CommandBus.dispatch(command, user_id: "manual-recovery", caused_by: :none)

Re-dispatch SendToTaxAuthority

Use this when the initial submission failed and needs to be retried.

alias Accounting.Domain.Commands.Document.SendToTaxAuthority
alias Shared.Infrastructure.CommandBus

command = SendToTaxAuthority.new(id: "DOCUMENT_ID")

CommandBus.dispatch(command, user_id: "manual-recovery", caused_by: :none)

Test an Event Handler Directly

Useful for verifying handler behaviour without waiting for real events:

alias Accounting.Application.EventHandlers.Document.OnIssuing.SendToTaxAuthority
alias Accounting.Domain.Entities.Document.Events.Issuing

event = %Issuing{
  id: "DOCUMENT_ID",
  send_to_tax_authority?: true
} |> Shared.Domain.Event.set_user_id("test")

SendToTaxAuthority.call(event)

---

Useful Log Searches

By Scenario

Scenario	Log source	What to search
Submission HTTP error	invopop_http_client.ex	[EInvoicing] at error level
E-invoicing decision (skip/proceed)	document_handler.ex	e-invoicing or configured at debug/info level
Webhook received	invopop.ex (Kafka handler)	webhooks_incoming_invopop at info level
Webhook dispatch	webhook_received/dispatch.ex	WebhookReceived at info/warning level
Provider event handling	invopop.ex (provider impl)	on_invoice_event at info level
Kafka retry/blocked	with_retry.ex	Pipeline.Blocked or retry at warning/error level
CDC sync	change_data_capture/handler.ex	sync_document or DocumentChanged at debug/info level
RPC calls to LVDAM	http_adapter.ex	Lvdam.Client or the endpoint path at info/error level

By Log Level

• Debug: Normal flow decisions (e-invoicing skip/proceed, CDC events)

• Info: Successful operations (submission sent, webhook processed, command dispatched)

• Warning: Non-fatal issues (sync_document failed, optional field missing)

• Error: Failures requiring attention (HTTP errors, command failures, validation errors)

Tips

• Filter by document ID to trace a single invoice through the entire flow

• Combine the document ID with e_invoicing_reference_id (Invopop Silo entry UUID) to cross-reference between Abacus logs and the Invopop Console

• If the document has a workspace_id, use it to check workspace configuration issues