FreeSWITCH has an option to cache directory lookups:mod_xml_curl
FreeSWITCH has an option to cache directory lookups so that xml_curl doesn't need to lookup the same user again, using the cacheable attribute in the returned xml (see http://wiki.freeswitch.org/wiki/Xml_curl#Caching_objects), and removed from cache using the xml_flush_cache API.
I would offer a bounty of $261 to add an expiration option so that cached results expire after the configured time, something like cacheable=true caches forever (or until you flush the cache), cachable=60 would cache for 60 seconds.
mod_xml_curl
Skip to end of metadata
- Created by John Boteler, last modified by Jim Lynch on 2018.05.13
Go to start of metadata
About
There is also some leaking into http://wiki.freeswitch.org/wiki/Channel_Variables , at some point that URL should also be updated. (original wiki note)
mod_xml_curl can be used to dynamically control the behaviour of FreeSWITCH.
This module is called when FreeSWITCH needs to fetch configuration that would normally be read from the static XML configuration files. Here are some typical use cases;
- Running multiple FreeSWITCH instances without having to maintain multiple server configurations
- Centralized configuration management without resorting to hacky shell scripts
- Dynamically populating configuration from a web application connected to a database
- Providing an easy way to automate configuration deployment of FreeSWITCH (for example, a hosted VoIP platform)
This module will give a seasoned developer the core tool he needs to get the job done, but this is only 1% of the work.
It works just like any other typical API, it will send your web application a POST request, and you would reply with a chunk of XML.
It will fetch any configuration that would have otherwise been loaded via the FreeSWITCH Configuration File.
If it receives a valid response from your web application, then it will load the configuration just like it would if you had put it into the FreeSWITCH Configuration File. If it receives an invalid or 404 not found response, then it will attempt to look for the file on disk instead.
However, you may notice that some fields repeat data in other fields, or that the structure is a bit confusing.
Let me tell you now, the module is not perfect, and in some cases you will have to figure it out with trial and error because not everything is documented yet.
Click to expand Table of Contents
Considerations
There are some important things you need to consider before using mod_xml_curl.
- Ask yourself why you need to use dynamically generated configuration files, and what you can't achieve without it.
- Will you serve the configuration from files on disk, or will you build them using a database schema?
- Will you generate the output using an XML library, or a templating language?
- How will you parse the requests, will you write a proper API parsing library?
- What hosting stack will you use to expose the web application to your FreeSWITCH instances?
- How will you prevent it from being abused by unauthorized users?
These are just a few of the questions you should be asking yourself. If you are not, then mod_xml_curl probably isn't for you!
You should also have a reasonable grasp on writing low latency web applications, as the performance of your web application will directly impact the performance of FreeSWITCH.
Installation
Edit modules.conf in your FreeSWITCH source directory, ensure the following line is uncommented, then recompile and install.
xml_int/mod_xml_curl
Then edit your Modules XML Configuration File inside your FreeSWITCH configuration directory, and place the following at the top of the file;
./freeswitch/conf/autoload_configs/modules.conf.xml
|
|
It is important that these goes at the top, otherwise things might not get loaded in the correct order. Also don't get modules.conf and modules.conf.xml mixed up, they both serve very different purposes!
Configuration
The module is split into 4 different bindings (otherwise referred to as 'sections'), which are;
- configuration - configuration items, for example mod_sofia
- directory - user directory (user authentication)
- dialplan - Dialplan items (call routing)
- phrases - phrases for the SAY api (Speech Phrase Management)
You can point a binding at as many different gateways as you like, and you can specify multiple bindings to keep the configuration tidy.
Here is a typical example of how you might configure this module;
./freeswitch/conf/autoload_configs/xml_curl.conf.xml
|
|
If you wanted to keep things compact/tidy, you can also use the following approach:
./freeswitch/conf/autoload_configs/xml_curl.conf.xml
|
|
You may also provide some extra parameters, such as authentication scheme and login credentials. Here is an example;
./freeswitch/conf/autoload_configs/xml_curl.conf.xml
|
|
The 'name' parameters shown in the above examples can be anything you want.
options
| gateway-url | URL for binding | http://myhost/xml_curl.php |
| gateway-credentials | Username/Password for URL | user:pass |
| auth-scheme | which auhtentification scheme to use. Supported values are: basic, digest, NTLM, GSS-NEGOTIATE or "any" for automatic detection | basic |
| method | Whether to use GET or POST. | GET |
| timeout | HTTP request timeout | 20 |
| enable-cacert-check | Whether to check server's SSL certificate against CA certificates (recommended for HTTPS) | true |
| enable-ssl-verifyhost | Whether to check server's SSL certificate matches the hostname of the URL (recommended for HTTPS) | true |
| ssl-cacert-file | Path to CA certificate(s) file. Note this appears to be required as the module doesn't automatically search for system CAs. | /etc/ssl/certs/ca-certificates.crt |
| ssl-cert-path | Path to client certificate | /etc/ssl/certs/fs_client.crt |
| ssl-key-path | Path to client private key | /etc/ssl/private/fs_client.key |
| ssl-key-password | Password for client private key | mysecret |
| ssl-version | Which version of SSL/TLS to use. Support values are "SSLv3" or "TLSv1" | TLSv1 |
| disable-100-continue | Disable 100 Continue in HTTP Expect header, for servers which dislike this option. | true |
| cookie-file | Path to file for storing cookies | /var/run/freeswitch/fs.cookies |
| use-dynamic-url | Undocumented | Undocumented |
| enable-post-var | Undocumented | Undocumented |
| bind-local | Name of network interface to use for HTTP requests, which can affect the source address/routing on multihomed servers. | eth1 |
API tips and tricks
You can use the following freeswitch command to help with debugging your configuration. It will return a filename with the resulting XML for each xml_curl query.
xml_curl debug_on
If you get an error like the following, then that means you're really serving up some big-time configs. No worries, just edit mod_xml_curl.c and adjust XML_CURL_MAX_BYTES as needed.
mod_xml_curl.c:121 Oversized file detected [1056100 bytes]
If you want to include a local file from disk, you would use the following approach (TODO: needs confirmation);
A point to note here, is to use timeout param in xml_curl config to prevent sofia profile from becoming unresponsive to REGISTER requests. Full explanation of this issue can be found here
Pre-processor directives and variables
Note: Right now, this section is based more on my evaluation of the FreeSWITCH source code than on actually using the mechanism. I'm putting it here now while it is fresh in my mind because I believe it to be accurate.
When mod_xml_curl retrieves an XML document, it saves that document to a temporary file, and then tells FreeSWITCH™ to process that file like it would process any other XML configuration file. This means that any X-PRE-PROCESS tags you place in your dynamic configuration will be processed before handing the resulting XML to the main configuration parser, just like your conf/* files.
Some care must be taken when setting variables using X-PRE-PROCESS tags. Setting a variable using set in a X-PRE-PROCESS tag sets a "core global" variable. In all XML configuration these variables are referenced by the form $${varname}, and are actually treated as preprocessor variables. That is, the same parser pass that processes X-PRE-PROCESS tags also substitutes these variables for their value at that very instant in time, and the resulting xml document is passed to the main configuration parser, and that is the configuration that is used. Simply put, if you do something like this somewhat forced example:
...... ... ... ...
Just pretend that is happening across two different SIP profiles so that it seems somewhat plausible. The first reference to $${global_codec_prefs} will expand to mu-law, and the second one (and all subsequent references in time) to $${global_codec_prefs} will expand to a-law. The first setting is unchanged.
This point deserves some discussion because it's important to know that, as truly global variables, values of variables set in conf/vars.xml or in any X-PRE-PROCESS tag are accessible to the XML you generate dynamically, which makes it OK to reference $${local_ip_v4} from your generated XML and the Right thing will happen. It's also possible to overwrite variables that were previously set in vars.xml or any X-PRE-PROCESS tag using your generated XML. Doing this, however will have absolutely no effect on any previously parsed XML that used that variable. Also, executing a reloadxml will cause the original vars.xml to be re-parsed, thus overwriting your overwrite. If this sounds confusing that is because you can probably create some confusing and difficult to track bugs in your configuration by doing this.
Having said all of this, we can distill the information down to the simple caveat that, while it is safe and good to reference static variables in your dynamic configuration, it is probably not safe to try to overwrite them. Treat them with the respect you'd treat any global variable and you should be fine.
API Request Fields
This gives an explanation of what individual request fields means.
Event variables
For a better explanation on event variables, see Event Fields
Channel variables
For a better explanation on channel variables, see Channel Variables
section
The target section of the request.
This can be one of the following;
- directory
- dialplan
- configuration
Section: configuration
Below is a list of all the variables expected within a directory section request.
Applicable common fields
TODO: Needs example
Non-applicable common fields
TODO: Needs example
Applicable context fields
TODO: Needs example
Section: directory
Below is a list of all the variables expected within a directory section request.
Applicable common fields
- sip_contact_user
- sip_contact_host
- sip_auth_username
- sip_auth_response
- sip_auth_qop
- sip_auth_nonce
- sip_auth_nc
- sip_auth_cnonce
- sip_auth_method
- sip_profile
- sip_to_user
- sip_to_host
- sip_to_port
- sip_user_agent
Non-applicable common fields
- hostname (use FreeSWITCH-Hostname instead)
- sip_from_host (use domain instead)
- sip_request_host (use domain instead)
- sip_to_host (use domain instead)
- sip_auth_realm (use domain instead)
- key_value (use domain instead)
- sip_auth_uri (unclean, debugging only)
- key_name (useless here)
- key (useless here)
- tag_name (useless here)
Applicable context fields
purpose
Once you start implementing things with XML cURL, you will find that several requests are fired on several different places on the code, specially with directory. The purpose field will tell you which one it is requesting so that you know what to reply. Given that the modules are the ones responsible for giving context to the information on the directory, the list of options will depend on the module. On the next section, you will find examples and explanations for each one of the modules and their requests.
mod_sofia
- gateways
All sofia profiles that have parse=true on their domain definitions will query the directory for their gateway definitions. Every time we issue a rescan on a profile or the profile is loading, we will receive one of these requests. Sometimes this request even gets sent on FreeSWITCH boot and to avoid this behavior when you only want it to send such a request purely when client sends a register then comment out the lines in the profile which goes under domains section if your not going to use under multi tenant mode.
hostname : FS-CUBO section : directory tag_name : key_name : key_value : Event-Name : REQUEST_PARAMS Core-UUID : 5ff05ea9-98ae-4469-82a8-595766d9b41e FreeSWITCH-Hostname : FS-CUBO FreeSWITCH-Switchname : FS-CUBO FreeSWITCH-IPv4 : 192.168.204.3 FreeSWITCH-IPv6 : ::1 Event-Date-Local : 2012-11-13 12:33:11 Event-Date-GMT : Tue, 13 Nov 2012 15:33:11 GMT Event-Date-Timestamp : 1352820791632159 Event-Calling-File : sofia.c Event-Calling-Function : launch_sofia_worker_thread Event-Calling-Line-Number : 2332 Event-Sequence : 26 purpose : gateways profile : internal
- network-list
Even though we classify this as being for mod_sofia, it really is a request made by the core. Every time the core is loading ACLs, you will get a request like the following:
hostname : FS-CUBO section : directory tag_name : domain key_name : name key_value : 192.168.204.3 Event-Name : GENERAL Core-UUID : 6f2c3a2c-ab56-44d1-ad53-f763a430b1b3 FreeSWITCH-Hostname : FS-CUBO FreeSWITCH-Switchname : FS-CUBO FreeSWITCH-IPv4 : 192.168.204.3 FreeSWITCH-IPv6 : ::1 Event-Date-Local : 2012-11-13 12:24:57 Event-Date-GMT : Tue, 13 Nov 2012 15:24:57 GMT Event-Date-Timestamp : 1352820297648064 Event-Calling-File : switch_core.c Event-Calling-Function : switch_load_network_lists Event-Calling-Line-Number : 1202 Event-Sequence : 475 domain : 192.168.204.3 purpose : network-list
- publish-vm
action
The action being performed on this directory request.
Valid options are;
- sip_auth (Used by mod_sofia to perform authentication checks)
- message-count (Used by mod_voicemail to check if messages are waiting - needs response example)
- reverse-auth-lookup (???)
user
TODO: Find out how/where this value is extracted
Username of the authenticating user, this should always be present for every directory request.
ip
TODO: Find out how/where this value is extracted
IP address of authenticating user
domain
TODO: Find out how/where this value is extracted
Target domain/realm of authenticating user, this should always be present for every directory request.
Section: Dialplan
Below is a list of all the variables expected within a dialplan section request.
Applicable common fields
TODO: Explanation needed
Non-applicable common fields
TODO: Explanation needed
Applicable context fields
TODO: Explanation needed
Sample xml_dialplan POST
Sample of what mod_xml_curl posts to the web server on a request for dialplan. Caller is 800 and callee is 9999, keys and values seperated by ::::
Expand source
XML Usage
Section: dialplan
This is used to send back configuration files relating to dialplans. This will be triggered every time there is a call.
Example
REQUEST:
section=dialplan&tag_name=&key_name=&key_value=&context=default&destination_number=556 &caller_id_name=FreeSwitch&caller_id_number=5555551212&network_addr=&ani=&aniii=&rdnis= &source=mod_portaudio&chan_name=PortAudio/556&uuid=b7f0b117-351f-9448-b60a-18ff91cbe183 &endpoint_disposition=ANSWER
RESPONSE:
Section: configuration
This is used to send back configuration files such as sofia.conf.
Request variables are;
- key_value (This is the filename of the configuration being requested, for example sofia.conf - note it does not have the trailing .xml extension)
- key_name (Seems to always contain the string name)
- section (This will always contain the string configuration, as this is how you detect what section is being requested)
- tag_name (Seems to be the same as section)
Example
An example request query string would be;
key_value=sofia.conf&key_name=name§ion=configuration&tag_name=configuration
The response should look identical to what the file would have looked like on disk, with the exception that you need to wrap it in a
Section: directory
You must ensure that the "accept-blind-reg" parameter is set to "false" in sofia.conf.xml, otherwise your web application will not get called.
This will be called every time there is an authentication request, and also under other circumstances (see below)
This particular section will also be called from different contexts, and as such you may not need to return the full configuration each time. In fact, responding with as little as necessary is advisable, in order to avoid CPU spikes related to processing large XML documents, which then could lead to adverse voice quality. You can determine what information needs to be returned by analysing the 'purpose' field.
Example: default
You will also receive requests where purpose is blank, and these are where you would send back user information, say during an authorization.
REQUEST:
[hostname] => testmachine [section] => directory [tag_name] => domain [key_name] => name [key_value] => domain1.awesomevoipdomain.faketld [Event-Name] => REQUEST_PARAMS [Core-UUID] => c5c8cbf4-60c3-45a2-b110-933da620cfd2 [FreeSWITCH-Hostname] => 25515_1_36308_177178 [FreeSWITCH-IPv4] => 192.168.1.10 [FreeSWITCH-IPv6] => ::1 [Event-Date-Local] => 2009-10-27 00:47:10 [Event-Date-GMT] => Tue, 27 Oct 2009 07:47:10 GMT [Event-Date-Timestamp] => 1256629630733916 [Event-Calling-File] => sofia_reg.c [Event-Calling-Function] => sofia_reg_parse_auth [Event-Calling-Line-Number] => 1671 [action] => sip_auth [sip_profile] => internal [sip_user_agent] => PolycomSoundPointIP-SPIP_320-UA/3.1.0.0084 [sip_auth_username] => 1004 [sip_auth_realm] => domain1.awesomevoipdomain.faketld [sip_auth_nonce] => 533c5264-12cb-4f8b-bcdb-5ecabe5e540f [sip_auth_uri] => sip:domain1.awesomevoipdomain.faketld:5060 [sip_contact_user] => 1004 [sip_contact_host] => 192.168.1.100 [sip_to_user] => 1004 [sip_to_host] => domain1.awesomevoipdomain.faketld [sip_from_user] => 1004 [sip_from_host] => domain1.awesomevoipdomain.faketld [sip_request_host] => domain1.awesomevoipdomain.faketld [sip_request_port] => 5060 [sip_auth_qop] => auth [sip_auth_cnonce] => hSVnPb32nA/OtkY [sip_auth_nc] => 00000001 [sip_auth_response] => 6e4e611d7593d52e02451b70900071d8 [sip_auth_method] => REGISTER [key] => id [user] => 1004 [domain] => domain1.awesomevoipdomain.faketld [ip] => 192.168.1.100
RESPONSE:
If you don't want to communicate passwords in plain text over your network, you can use the `a1-hash` param instead. Its value should be an md5sum containing `user:domain:password`.
For example, if you have a user 1002 on domain 127.0.0.1 with password 1234, you can generate the proper hash by:
echo -n 1002:127.0.0.1:1234 | md5sum
The proper param tag for this example is:
Example: purpose=gateways
REQUEST:
[hostname] => testmachine [section] => directory [tag_name] => [key_name] => [key_value] => [Event-Name] => REQUEST_PARAMS [Core-UUID] => c5c8cbf4-60c3-45a2-b110-933da620cfd2 [FreeSWITCH-Hostname] => testmachine [FreeSWITCH-IPv4] => 192.168.1.10 [FreeSWITCH-IPv6] => ::1 [Event-Date-Local] => 2009-10-27 00:52:52 [Event-Date-GMT] => Tue, 27 Oct 2009 07:52:52 GMT [Event-Date-Timestamp] => 1256629972839876 [Event-Calling-File] => sofia.c [Event-Calling-Function] => config_sofia [Event-Calling-Line-Number] => 3056 [purpose] => gateways [profile] => internal
RESPONSE:
Example: purpose=network-list
REQUEST:
[hostname] => black [section] => directory [tag_name] => domain [key_name] => name [key_value] => 192.168.0.2 [domain] => 192.168.0.2 [purpose] => network-list
RESPONSE:
TODO: Example needed
Example: action=message-count
mod_voicemail occasionally needs to look up a user's id.
The request example is below and can be responded to in the same way as an authorization request for simplicity.
REQUEST:
[hostname] => testmachine [section] => directory [tag_name] => domain [key_name] => name [key_value] => domain1.awesomevoipdomain.faketld [Event-Name] => GENERAL [Core-UUID] => c5c8cbf4-60c3-45a2-b110-933da620cfd2 [FreeSWITCH-Hostname] => testmachine [FreeSWITCH-IPv4] => 192.168.1.10 [FreeSWITCH-IPv6] => ::1 [Event-Date-Local] => 2009-10-27 00:47:40 [Event-Date-GMT] => Tue, 27 Oct 2009 07:47:40 GMT [Event-Date-Timestamp] => 1256629660158410 [Event-Calling-File] => mod_voicemail.c [Event-Calling-Function] => resolve_id [Event-Calling-Line-Number] => 1278 [action] => message-count [key] => id [user] => 1004 [domain] => domain1.awesomevoipdomain.faketld
Example: action=reverse-auth-lookup
Here is an example of when an endpoint requests reverse authentication for a request, using reverse-auth-lookup;
REQUEST:
["url"]=> "freeswitch/directory" ["section"]=> "directory" ["key_value"]=> "default" ["Event-Name"]=> "REQUEST_PARAMS" ["Action"]=> "reverse-auth-lookup" ["key"]=> "id" ["user"]=> "walker" ["domain"]=> "domain-name"
RESPONSE:
This tells FreeSWITCH™ what credentials to use for the challenge.
Section: not found
If your web application receives a request and you don't wish to serve a config, then you should respond with 200 status code and the following "not found" result.
If you return an empty response instead of the not found, you may see the following error;
[ERR] switch_xml.c:1534 switch_xml_locate() Error[[error near line 1]: root tag missing]
post_load configuration
Due to the way FreeSWITCH loads modules and configuration, there are occasions where it will request a configuration in the format of 'post_load'
One of these is post_load_switch.conf, FreeSWITCH will still attempt to request the normal file name (switch.conf) but will silently ignore the content, therefore you need to ensure that you serve the same content from both (ensuring that name attribute in
For example, a request for switch.conf would look like this;
And a request for post_load_switch.conf would look like this;
The other example of this is post_load_modules.conf, because modules.conf has to be initially loaded from disk for the module to be available in the first place.
Caching
If you want to reduce the amount of HTTP requests FreeSWITCH needs to make to your web server you can supply the cacheable="true" attribute which will allow FreeSWITCH to manage a memory table with the information. You will still see HTTP requests being made but should definitely notice a decrease with this attribute set.
Here is an example of using this feature;
If you then make a change to the directory you should run the xml_flush_cache command to clear, some examples:
# This clears items for user 1001 xml_flush_cache id 1002 domain-name # This clears all items xml_flush_cache
You may also wish to enable an expiry on the cache entries by setting the cacheable attribute to a numeirc value which would be the number of milliseconds, see http://jira.freeswitch.org/browse/FS-4871
will cache the result for 60 seconds (60000 milliseconds)
Alternative ways of storing static configurations
For installations with heavy usage this module can cause delays in call handling.
For example;
If the the desired intent merely is to centrally store and/or share a common static dialplan with multiple FreeSWITCH installations an alternative to mod_xml_curl would be to create and use a git repository, or automatically retrieving an XML document stored on the webserver by using the exec command within an X-PRE-PROCESS. Within your dialplan add:
(Important: Note the ' - '. This instructs wget to output the retrieved document to stdout.)
This method will include the dialplan stored on the server at the location where the X-PRE-PROCESS exists, the dialplan will only be fetched from the webserver on start-up and upon reloadxml.
Client implementations
PHP
- fs_curl - by Raymond (intralanman), last updated 2011
- FreeSWITCH-mod_xml-with-PHP - by avimar, last updated 2010.
- xml-curl - by trixter, last updated 2008.
- PHP example
Other languages
- Perl/CGI example - (by James Jones
) - Python/CGI example - (by Mike Fedyk
) - C# example
- Ruby dialplan example - very simple database (MySQL) driven Dialplan XML routing. (Jonathan Towne
) - Ruby directory example - very simple database (PostgreSQL) driven User Directory. (Jonathan Towne
) - C++ Directory Example - Reads from a database and serves the directory to Xml_Curl over HTTP.
- Ramaze Example - TinyQueue dynamic group dials. (By the rubyists
)
Lua
Lua can be called directly without a web server. See: Mod_lua/Serving_Configuration