.. _ipbx_troubleshoot:

********************
IPBX Troubleshooting
********************

Transfers using DTMF
====================

When transfering a call using DTMF (\*1) you get an *invalid extension* error when dialing the
extension.

The workaround to this problem is to create a preprocess subroutine and assign it to the destinations
where you have the problem.

Under :menuselection:`Services --> IPBX --> IPBX configuration --> Configuration files` add a new file
containing the following dialplan::

    [allow-transfer]
    exten = s,1,NoOp(## Setting transfer context ##)
    same = n,Set(__TRANSFER_CONTEXT=<internal-context>)
    same = n,Return()

Do not forget to substitute <internal-context> with your internal context.

Some places where you might want to add this preprocess subroutine is on queues and outgoing calls
to be able to transfer the called person to another extension.


.. _fax-detection:

Fax detection
=============

XiVO **does not currently support Fax detection**. The following describe a workaround to use this
feature. The behavior is to answer all incoming (external) call, wait for a number of seconds (4 in
this example) : if a fax is detected, receive it otherwise route the call normally.

.. note:: This workaround works only :

    * on incoming calls towards an User (and an User only),
    * if the incoming trunk is a DAHDI or a SIP trunk,
    * if the user has a voicemail which is activated and with the email field filled
    * XiVO >= 13.08 (needs asterisk 11)

    Be aware that this workaround will probably not survive any upgrade.

#. In the Web Interface and under :menuselection:`Services --> IPBX --> IPBX configuration -->
   Configuration files` add a new file named *fax-detection.conf* containing the following
   dialplan::

    ;; Fax Detection
    [pre-user-global-faxdetection]
    exten = s,1,NoOp(Answer call to be able to detect fax if call is external AND user has an email configured)
    same  =   n,GotoIf($["${XIVO_CALLORIGIN}" = "extern"]?:return)
    same  =   n,GotoIf(${XIVO_USEREMAIL}?:return)
    same  =   n,Set(FAXOPT(faxdetect)=yes) ; Activate dynamically fax detection
    same  =   n,Answer()
    same  =   n,Wait(4) ; You can change the number of seconds it will wait for fax (4 to 6 is good)
    same  =   n,Set(FAXOPT(faxdetect)=no) ; If no fax was detected deactivate dyamically fax detection (needed if you want directmedia to work)
    same  =   n(return),Return()

    exten = fax,1,NoOp(Fax detected from ${CALLERID(num)} towards ${XIVO_DSTNUM} - will be sent upon reception to ${XIVO_USEREMAIL})
    same  =     n,GotoIf($["${CHANNEL(channeltype)}" = "DAHDI"]?changeechocan:continue)
    same  =     n(changeechocan),Set(CHANNEL(echocan_mode)=fax) ; if chan type is dahdi set echo canceller in fax mode
    same  =     n(continue),Gosub(faxtomail,s,1(${XIVO_USEREMAIL}))

#. In the file :file:`/etc/xivo/asterisk/xivo_globals.conf` set the global user subroutine to
   ``pre-user-global-faxdetection`` : this subroutine will be executed each time a user is called::

    XIVO_PRESUBR_GLOBAL_USER = pre-user-global-faxdetection

#. Reload asterisk configuration (both for dialplan and dahdi)::

    asterisk -rx 'core reload'


.. _cti-ami-proxy:

CTI server is unexpectedly terminating
======================================

If you observes that your CTI server is sometimes unexpectedly terminating with the following
message in :file:`/var/log/xivo-ctid.log`::

    (WARNING) (main): AMI: CLOSING

Then you might be in the case where asterisk generates lots of data in a short period of time on the
AMI while the CTI server is busy processing other thing and is not actively reading from its AMI
connection. If the CTI server takes too much time before consuming some data from the AMI
connection, asterisk will close the AMI connection. The CTI server will terminate itself once it
detects the connection to the AMI has been lost.

There's a workaround to this problem called the ami-proxy, which is a process which buffers the AMI
connection between the CTI server and asterisk. This should only be used as a last resort solution,
since this increases the latency between the processes and does not fix the root issue.

To enable the ami-proxy, you must:

#. Add a file :file:`/etc/systemd/system/xivo-ctid.service.d/ami-proxy.conf`:

   .. code-block:: sh

      mkdir -p /etc/systemd/system/xivo-ctid.service.d
      cat >/etc/systemd/system/xivo-ctid.service.d/ami-proxy.conf <<EOF
      [Service]
      Environment=XIVO_CTID_AMI_PROXY=1
      EOF
      systemctl daemon-reload

#. Restart the CTI server::

      systemctl restart xivo-ctid.service

If you are on a XiVO cluster, you must do the same procedure on the slave if you want the ami-proxy
to also be enabled on the slave.

To disable the ami-proxy::

   rm /etc/systemd/system/xivo-ctid.service.d/ami-proxy.conf
   systemctl daemon-reload
   systemctl restart xivo-ctid.service


.. _postgresql_localization_errors:

PostgreSQL localization errors
==============================

The database and the underlying `database cluster`_ used by XiVO is sensitive to the system locale
configuration. The locale used by the database and the database cluster is set when XiVO is
installed. If you change your system locale without particular attention to PostgreSQL, you might
make the database and database cluster temporarily unusable.

.. _database cluster: https://www.postgresql.org/docs/17/creating-cluster.html

When working with locale and PostgreSQL, there's a few useful commands and things to know:

* ``locale -a`` to see the list of currently available locales on your system
* ``locale`` to display information about the current locale of your shell
* ``grep ^lc_ /var/lib/postgresql/17/data/postgresql.conf`` to see the locale configuration of your
  database cluster
* ``sudo -u postgres psql -l`` to see the locale of your databases
* the :file:`/etc/locale.gen` file and the associated ``locale-gen`` command to configure the
  available system locales
* ``systemctl restart postgresql.service`` to restart your database cluster
* the PostgreSQL log file located at :file:`/var/log/postgresql/postgresql-17-main.log`

.. note:: You can use any locale with XiVO as long as it uses an UTF-8 encoding.


Database cluster is not starting
--------------------------------

If the database cluster doesn't start and you have the following errors in your log file::

   LOG:  invalid value for parameter "lc_messages": "en_US.UTF-8"
   LOG:  invalid value for parameter "lc_monetary": "en_US.UTF-8"
   LOG:  invalid value for parameter "lc_numeric": "en_US.UTF-8"
   LOG:  invalid value for parameter "lc_time": "en_US.UTF-8"
   FATAL:  configuration file "/var/lib/postgresql/17/data/postgresql.conf" contains errors

Then this usually means that the locale that is configured in :file:`postgresql.conf` (here ``en_US.UTF-8``)
is not currently available on your system, i.e. does not show up the output of ``locale -a``. You
have two choices to fix this issue:

* either make the locale available by uncommenting it in the :file:`/etc/locale.gen` file and running
  ``locale-gen``
* or modify the :file:`/var/lib/postgresql/17/data/postgresql.conf` file to set the various ``lc_*``
  options to a locale that is available on your system

Once this is done, restart your database cluster.


Can't connect to the database
-----------------------------

If the database cluster is up but you get the following error when trying to connect to the
``asterisk`` database::

   FATAL:  database locale is incompatible with operating system
   DETAIL:  The database was initialized with LC_COLLATE "en_US.UTF-8",  which is not recognized by setlocale().
   HINT:  Recreate the database with another locale or install the missing locale.

Then this usually means that the database locale is not currently available on your system. You have
two choices to fix this issue:

* either make the locale available by uncommenting it in the :file:`/etc/locale.gen` file, running
  ``locale-gen`` and restarting your database cluster
* or :ref:`recreate the database using a different locale <postgres-change-locale>`


Error during the upgrade
------------------------

Then you are mostly in one of the cases described above. Check your log file.


Error while restoring a database backup
---------------------------------------

If during a database restore, you get the following error::

   pg_restore: [archiver (db)] Error while PROCESSING TOC:
   pg_restore: [archiver (db)] Error from TOC entry 4203; 1262 24745 DATABASE asterisk asterisk
   pg_restore: [archiver (db)] could not execute query: ERROR:  invalid locale name: "en_US.UTF-8"
       Command was: CREATE DATABASE asterisk WITH TEMPLATE = template0 ENCODING = 'UTF8' LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8';

Then this usually means that your database backup has a locale that is not currently available on
your system. You have two choices to fix this issue:

* either make the locale available by uncommenting it in the :file:`/etc/locale.gen` file, running
  ``locale-gen`` and restarting your database cluster
* or if you want to restore your backup using a different locale (for example ``fr_FR.UTF-8``),
  then restore your backup using the following commands instead::

     sudo -u postgres dropdb asterisk
     sudo -u postgres createdb -l fr_FR.UTF-8 -O asterisk -T template0 asterisk
     sudo -u postgres pg_restore -d asterisk asterisk-*.dump


Error during master-slave replication
-------------------------------------

Then the slave database is most likely not using an UTF-8 encoding. You'll need to
:ref:`recreate the database using a different locale <postgres-change-locale>`


.. _postgres-change-locale:

Changing the locale (LC_COLLATE and LC_CTYPE) of the database
-------------------------------------------------------------

If you have decided to change the locale of your database, you must:

* make sure that you have enough space on your hard drive, more precisely in the file system holding
  the :file:`/var/lib/postgresql` directory. You'll have, for a moment, two copies of the
  ``asterisk`` database.
* prepare for a service interruption. The procedure requires the services to be restarted twice,
  and the system performance will be degraded while the database with the new locale is being
  created, which can take a few hours if you have a really large database.
* make sure the new locale is available on your system, i.e. shows up in the output of ``locale -a``

Then use the following commands (replacing ``fr_FR.UTF-8`` by your locale)::

   xivo-service restart all
   sudo -u postgres createdb -l fr_FR.UTF-8 -O asterisk -T template0 asterisk_newlocale
   sudo -u postgres pg_dump asterisk | sudo -u postgres psql -d asterisk_newlocale
   xivo-service stop
   sudo -u postgres psql <<'EOF'
   DROP DATABASE asterisk;
   ALTER DATABASE asterisk_newlocale RENAME TO asterisk;
   EOF
   xivo-service start

You should also modify the :file:`/var/lib/postgresql/17/data/postgresql.conf` file to set the various
``lc_*`` options to the new locale value.

For more information, consult the `official documentation on PostgreSQL localization support
<https://www.postgresql.org/docs/17/charset.html>`_.


Replication issues on a PostgreSQL instance
===========================================

If your XiVO MDS replication is broken or stuck, you can recreate the replication slot
from the main XiVO server and force a full resynchronization of the MDS database.

.. warning::

  This procedure will completely remove the PostgreSQL data directory on the MDS node and
  trigger a full database resynchronization from the main XiVO server.

  Before starting, you should:

  make sure you have enough free disk space on both the main XiVO server and the MDS node.
  prepare for a temporary service interruption on the affected MDS node.
  verify that the PostgreSQL replication slot name matches the affected MDS instance.

On the XiVO main server
-----------------------

List the current replication slots::

  SELECT * FROM pg_replication_slots ;

Remove the replication slot corresponding to the affected MDS::

  SELECT pg_drop_replication_slot('main_mds01');

Replace main_mds01 with the actual replication slot name of your MDS node.

On the XiVO MDS server
----------------------

Stop the PostgreSQL container::

  xivo-dcomp stop db

Remove the PostgreSQL container::

  xivo-dcomp rm db

Delete the PostgreSQL data directory::

  rm -rf /var/lib/postgresql/17

Recreate and restart the PostgreSQL container::

  xivo-dcomp up -d

The MDS node should now automatically perform a full PostgreSQL resynchronization from the
main XiVO server.

Verification
------------

You can verify that the replication is working again by checking the PostgreSQL logs on both
servers and ensuring that the replication slot is recreated automatically on the main XiVO server.

On the XiVO main server, you can also verify the replication status with::

  SELECT * FROM pg_replication_slots;


Originate a call from the Asterisk console
==========================================

It is sometimes useful to ring a phone from the asterisk console. For example, if you want
to call the ``1234`` extension in context ``default``::

   channel originate Local/1234@default extension 42@xivo-callme

WebRTC
======

* `http.conf` - asterisk's webserver must accept connection from outside, the listen address must be updated, for the sake of
  simplicity let's use 0.0.0.0, you can also pick an address of one of the network interfaces:

::

    [general]
    enabled=yes
    bindaddr=0.0.0.0
    bindport=5039
    prefix=
    tlsenable=yes
    tlsbindaddr=127.0.0.1:5040
    tlscertfile=/usr/share/xivo-certs/server.crt
    tlsprivatekey=/usr/share/xivo-certs/server.key
    servername=XiVO PBX

Do not forget to reload the configuration by the `module reload http` command on the Asterisk CLI.

* `rtp.conf` - the ICE support must be activated (it is by default)

Check `rtp show settings` in the asterisk console if it says otherwise.

The configuration is reloaded by `module reload res_rtp_asterisk.so`.

* WebRTC requires DTLS keys to be generated in `/etc/asterisk/keys`. If you need to manually generate the DTLS
  certificates following instructions on the Asterisk Wiki: https://wiki.asterisk.org/wiki/display/AST/Secure+Calling+Tutorial.
  You just need to generate the TLS certificates (first call of `ast_tls_cert`), other steps are not necessary.
  Make sure asterisk can read files by executing: :code:`chown -R asterisk.asterisk /etc/asterisk/keys`


Call Permission and Transfers
=============================

Some Call Permission issues may occur in case of call transfers.
For example:

* Given user U1 with call permissions C1,
* Given user U2 with another call permissions set C2,
* When U1 calls U2 and transfers it somewhere
* Then, depending on the type of transfer it will take the call permissions C1 or C2.

Current behavior is descrbided in `bug 1944 <https://projects.xivo.solutions/issues/1944>`_.