Troubleshoot backup failures and other problems

Start from the symptom and the last operation before it, then follow the matching check. Most checks only read information: you sign in to the XOA-HL appliance in your browser as an administrator. Some checks need the appliance console or a guest console. Before you retry a failed update, import or backup, save the error and the logs.

Start from the symptom

For console problems, such as the controls or a blank or frozen screen, see Use the VM console.

Diagnose network connectivity

Use these read-only checks for an existing, ordinary guest network or for the lab VLAN. Sign in as an XOA-HL administrator, so that you can see every VM, pool and host.

Before you start

• Guest checks need an authorized session on the guest console. The owners of the switches, DNS and firewalls check those.
• Keep separate console access to the host available.
• Do not create a fault, and do not change the management network, to test this guide.
• Bonds, SDN, VLAN trunks inside guests and emergency network recovery are not covered here.

1. Find the failing path

Note the time with its time zone, the last operation that worked and the exact error. Also note the source machine, the destination name or address, the protocol and the service port. Save this starting state before you retry. A test that works from your laptop does not prove access from the appliance, a host or a guest.

2. Trace the guest interface to the host uplink

Only read these screens. Do not click editable network or MAC fields, or connection buttons.

1. Note the VM UUID, its power state, the host it runs on and its pool. Open the VM’s Network tab (/#/vms/<id>/network).
2. Copy the interface (VIF) UUID with the row’s copy button. Note the device number, MAC address, network, MTU and status.
3. Open the pool’s network view (/#/pools/<id>/network) and match the exact network UUID, not only its name.
4. Read Connected: it shows whether the VIF is attached. A detached VIF on a stopped VM is not a fault.
5. Read IP addresses: it comes from the guest metrics, per VIF device. A missing address does not prove that the guest is not configured. A shown address is not a live reachability test. Compare with the guest console and the guest agent checks.
6. Note the locking mode indicator and Allowed IPs separately from the reported addresses. They are traffic rules, not a form to give the guest its IP address. Save any warning, and ask the network owner to explain the rules that apply. Do not unlock the interface, clear restrictions, change the MAC or MTU, or turn off security as a shortcut.
7. In the pool network row, expand PIFs and find the host the VM runs on. That row’s VLAN summary comes from the pool master only.
8. Open that host’s Network tab (/#/hosts/<id>/network). Note the PIF UUID, device, MAC, network, VLAN, MTU, management flag and IP mode. Compare them with the approved cabling and switch records. A network can be internal on purpose, with no physical uplink: check the expected layout before you call it a fault.
9. Read the attachment and the physical link (carrier) separately. The host’s Connected button shows whether the PIF is attached. The physical link icon, and the PIF status in the expanded pool row, show the carrier.

Missing PIF metrics can also show as no carrier, so a negative status alone does not prove an unplugged cable. Refresh, and compare with what the host and the switch show. Do not toggle PIFs, edit VLAN, MTU or IP settings, delete networks or restart networking to probe the path.

3. Check guest settings and the service

1. On the guest console, find the interface by its MAC address. Its name in the guest may differ from the host’s device name.
2. With the guest operating system’s tools, read its address and prefix, routes and DNS settings. Compare them with the unique address and subnet that the network owner approved. On the lab VLAN, the guest normally sees an untagged interface.
3. Ask the guest or network administrators to test the agreed peer and application service from this exact guest. Note separately the DNS answers, the address family used, the destination, the protocol and port, and the result.
4. If a test by name and a test by address give different results, check DNS and the application or TLS identity before you conclude. HTTPS to a bare IP address can change the certificate or virtual host. Never bypass certificate checks to declare success.

A successful ping does not prove that a service works, and a failed ping does not prove isolation. Test only approved destinations. When isolation is required, also test the forbidden destinations that the owner defines.

A peer on the same host does not test the physical uplink. Tests across hosts, or outside the host, need identified peers and an approved switch path. Do not migrate a workload to improvise a test. If you see unexpected access to production, save the evidence and shut the test guest down normally. If management or storage is affected, use the separate host console and the owner’s recovery plan before any other change.

4. Check the fix

1. Give the owner the exact mapping, the times, the error and your comparison results.
2. Apply the agreed fix through the right guest, storage, host or network procedure. This guide does not prescribe a general reset.
3. Refresh the inventory. Repeat the same service test, from the same source to the same destination.
4. Repeat the required isolation checks, and check management and storage access.
5. Note what changed and the result.

Check: the same test now passes. A green indicator alone does not mean that the problem is solved. If the path is still unclear, keep the incident open and report a problem with sanitized details.

Diagnose a failed NFS backup

Use this only for the full backup job to NFS from the tutorial, signed in as an XOA-HL administrator. You work in the browser. Storage checks belong to the administrator of that storage. You do not need to cause a failure.

Before you start

• Note the job and schedule IDs, the source VM UUID, the remote ID, the expected run time with its time zone and the result of the last isolated restore.
• Incremental chains, replication and production recovery are not covered here. For other backup tests, see incremental backups and chain recovery, one file from a delta backup, a DR replica on another pool and an SMB remote on Windows Server.

1. Find the run and save its details

1. Open Backup → Overview (/#/backup/overview) and find the job. Look at the Logs table below it.
2. Clear the log search and status filters if needed. Find the run by its job and start time. The table shortens the job ID, so use the full IDs from the details, not only the name.
3. Select the run’s status button to open its details. Note the run id, jobId, scheduleId, the start and end times, and the status.
4. Before you change anything, save the JSON privately with the clipboard button or Download log. The downloaded log is not redacted: for a report, make a sanitized excerpt.
5. Clear the search in the details and select all task statuses. At first, the dialog can show only pending, failed or interrupted tasks, five per page. Check every page.
6. Find the VM UUID, then its snapshot, remote, export and transfer tasks. Save the errors, skip reasons and warnings at each level. Not every raw task is shown: if the tree looks incomplete, read the saved JSON.

Check: your notes identify one exact run, and the affected VM and remote. Missing logs or tasks do not prove that nothing ran.

2. Read the status before you retry

3. Separate the failed stage from surviving data

Open Backup → Restore (/#/backup/restore) to see the restore points and the restore logs.

Check: you can tell apart the failed stage, the restore points that survived, and the objects whose state is still unknown. A file or a listed point alone does not prove that the data can be recovered.

4. Fix the cause and check the result

1. To stop future runs, disable the exact lab schedule in Overview, then refresh to check its state. This does not cancel a running backup: the cancel button in Overview is disabled in this version. Restarting xo-server does not cancel or clean up a backup.
2. Fix the access, space or selection problem with its owner. Then check the job, schedule, VM and remote again.
3. Keep Delete first off, and keep the recovery copies you need outside this job’s retention. If you are unsure whether work is still running, or who owns partial objects, save the evidence and ask for help before you retry.
4. When no run is active and you understand the state, run this one-VM lab job by hand, as in the tutorial. The restart and force buttons in the details dialog run backups: they do not restart the guest and are not read-only checks. Do not use force. A retry can also apply retention to older points.
5. Check the new run and the restore points. Then restore a stopped copy, remove every NIC and check the sample data.
6. Leave the lab schedule disabled. Before you turn scheduled runs on again, note the original error, the fix, the new run and point IDs, and the restore result.

Check: the new run, the kept restore points and the isolated restore of the guest data all pass. If not, keep the incident open and report a problem with sanitized evidence.

Check the appliance from its console

These commands only read the state. Run them inside the appliance, not on the hypervisor host, from an authorized console or SSH session. Some of them need root privileges.

rpm -q xoa-hl nodejs
timedatectl
df -h
systemctl status xo-server redis xoa-hl-update.service --no-pager
systemctl list-timers 'xoa-hl-*'
journalctl -u xo-server -u xoa-hl-check-update.service -u xoa-hl-update.service -u xoa-hl-auto-update.service --since '1 hour ago' --no-pager
tail -n 100 /var/log/xoa-first-boot.log
tail -n 100 /var/lib/xoa-hl/update.log
cat /run/xoa-hl/status
cat /var/lib/xoa-hl/auto-update.status

• A missing status file before the first check, or after a restart, is not a failure on its own.
• systemctl status returns a nonzero code for an inactive or failed unit. Read the state that it shows.
• The automatic update status can describe an older run. Compare its UTC times.
• For what each status means, and for update and restart decisions, see Maintain and update the appliance.

Do not delete database files, logs or locks to make an error go away. If the disk is full, find the files that grow and save the evidence before you choose what to clean up.

Recover administrator access

Use this when no administrator can sign in: the password is lost, the device with the one-time code is lost, or an LDAP or SAML sign-in no longer works. You run a recovery command inside the appliance, as root, from the appliance console or SSH. Anyone with root access to the appliance can take over an XO account this way, so protect that access.

The command changes the account you name:

• It sets a new password.
• It removes the account’s one-time code.
• It makes the account an Admin.
• It removes the account’s LDAP or SAML links, so the account becomes local.
• It keeps the account’s other settings and its existing authentication tokens.

If no account has that name, the command creates a new local administrator.

1. Open the appliance console from XO Lite, or an SSH session, and become root.

2. Check that the database runs: systemctl is-active redis prints active.

3. Run the command with the account’s exact username, usually an email address. You do not need to stop xo-server.

   node /opt/xo/packages/xo-server/dist/recover-account-cli.mjs admin@example.net

4. At Password (leave empty for random):, type the new password and press Enter. The password is not shown. If you leave it empty, the command prints a random password: store it in your password manager, then clear the screen. Do not give the password on the command line, because it would stay in the shell history.

5. Wait for user admin@example.net has been successfully updated. If the message says created, the name did not match an existing account. Once signed in, check the name in Settings → Users and delete the extra account if you do not need it.

6. In a new private browser window, sign in with the account and its new password. No one-time code is asked.

7. On your user page, read Authentication tokens. If a device or a password was lost, select Forget all authentication tokens. It removes every token except the one of your current session.

8. If the account used a one-time code, set it up again on a device you control. If it used LDAP or SAML, it is now a local account: keep it as your local recovery administrator, or ask the identity owner before you link it again.

Check: the account signs in with the new password, Settings → Users lists it as Admin, its OTP column shows Not configured, and only the tokens you expect remain.

Common questions

Why did host maintenance leave VMs behind, or the host disabled? Save the task. Then check the host’s actual enabled state and where each VM runs. The host maintenance test handles evacuation and re-enabling separately. A partial evacuation or an early failure needs the host owner’s recovery plan. Do not bypass migration blocks, and do not repeat the toggle blindly.

Why did a host fail to join a pool? Save the task, the error and the inventories of both pools. The pool join test checks the exact IDs, patch differences, master, networks and storage before you confirm. The interface runs a forced join: do not repeat it, and do not eject a host to clear an unexplained failure.

Why was delegated VM creation refused? Follow the resource set and quota test. Compare the exact user, the selected set, the template, SR and network IDs, and the quota Used/Total. Seeing a template or a button does not prove that the server allows the action. Save the error, and look for a partial VM or disk before you retry. Do not raise quotas or grant Admin on the pool to clear an unexplained failure.

Why are updates listed while the check command returns 100? The packaged checker turns the dnf check-update exit code 100 into AVAILABLE, 0 into UP_TO_DATE, and any other code into ERROR. Listed updates are not a failed installation.

Does switching to stable undo a testing update? No. The channel chooses the repositories for later checks and installations. Switching to stable does not downgrade a version that is already installed.

Does a pool metadata backup prove that a host can be recovered? No. The pool metadata test checks only the export, retention and file identity. Restoring it on an isolated test pool is not covered yet.

Can a configuration export replace a VM backup or an appliance rollback? No. Read the configuration scope and recovery limits, and keep separate backups of your guests.

Report a problem

Open an XCP-HL issue. Give this information as text:

• The page URL and language, or the appliance route where the problem occurs.
• The image tag, the versions shown in About, the installed RPM and Node versions, and the stable or testing channel.
• The operation, the exact steps, the expected result and the actual error.
• The time with its time zone, the manual or automatic mode, the related service state and a sanitized log excerpt.
• Whether the problem repeats on a disposable appliance, and the last version that worked.

Remove passwords, tokens, cookies, private host names and addresses, and user or guest data from logs and screenshots. xo-server and first-boot logs can contain credentials. Never attach a configuration export, a TLS private key or a complete configuration file. Keep the original evidence privately.