Posts in Category "Install"

Connect Meeting RTMP VS/VIPs on Load-Balancers

This article applies to on-premise Adobe Connect servers running behind hardware-based load-balancing devices or SSL accelerators.

A common cause of performance problems in Adobe Connect Meetings stems from the improper configuration of the Virtual Server (VS) Virtual IP Address (VIP) handling Real Time Messaging Protocol (RTMP) traffic in on-premise Connect deployments.

An Adobe Connect Meeting Server is at least two servers in one (possibly more if AEM/Events and UV telephony are incorporated); it is at least always a Tomcat-based HTTP application server and an Adobe Media Server (AMS) using RTMP. The two servers are fully integrated to work together in tandem to support Adobe Connect Meetings.

The most popular load-balancing and SSL acceleration  option in the Adobe Connect on-premise enterprise is the F5 BIG-IP Local Traffic Manager (LTM). This tech-note will illustrate the proper configuration of an RTMP VIP supporting Adobe Connect Meeting on an F5 LTM. The concepts apply to any load-balancing device and SSL accelerator.

The first thing to note is that the general configuration of a Connect server or cluster running behind an SSL accelerator or load-balancing device always requires more then one VIP. There are no exceptions to this rule and any attempts at shortcuts will result in delayed deployments and support cases. Attempts to place all traffic on a single VS/VIP are as common as they are incapacitating. General Connect cluster architecture tech-notes are here:

Adobe® Connect™ server pools/clusters and hardware-based load-balancing devices with SSL acceleration

Adobe Connect Servers and Hardware-based Load-balancing Devices

A simple diagram of an Adobe Connect server behind an F5 LTM follows; see the two VS/VIPs and Fully Qualified Domain Names (FQDNs) for each on the LTM:

C9SSL

Below we add a server to show a basic Connect cluster VIP configuration; see how each Connect Meeting server has its own VS/VIP while one VS/VIP servers both HTTPS application servers.

C9SSLa

Note: Neither of these basic diagrams depicts advanced configurations such as the integration of the Adobe Experience Manager (AEM) Events module. This article focuses on the performance of the Adobe Connect Meeting RTMP VIP in its basic context.

There is usually not an option for RTMP in the VIP profile of a hardware-based load-balancing device. A basic TCP profile is the correct choice. Here it is depicted on an F5 BIP-IP LTM:

f5.fw

With detail:

f5a.fw

f5b.fw

f5c.fw

Note that the symptom for an improperly configured VS/VIP is either the inability to launch a Connect Meeting or excessive latency in the Meeting due to RTMP tunneling (RTMPT) encapsulated within HTTP when the RTMP VIP is blocked or inoperable.

The presence of a capitol “T” in the latency indicator of an Adobe Connect Meeting indicates tunneling as depicted in this tech-note:

Tunneling with RTMP encapsulated in HTTP (RTMPT) should be avoided as it causes latency

Further diagnosis is usually warranted by using the Connect Meeting Addin in logging mode as depicted here:

Enable Logging in the Meeting Addin

Also here:

Troubleshooting Verbose Meeting Addin Logging

When the RTMP VS/VIP profile is improperly configured, the Connect Meeting addin verbose log will show it clearly, particularly when it is compared with the server-side debug log.

Example snippet from a Connect Meeting addin verbose log:

18:51:55    16844    PLAYER_TRACE    SSL connection closed.
18:51:55    16844    PLAYER_TRACE    SSL DoSSLHandshake WaitHandshake not in ssl_active state. (State is 0.) Failing.
18:51:55    16844    PLAYER_TRACE    SSL DoSSLHandshake WaitForSocket not in ssl_active state, failing.
18:51:55    16844    PLAYER_TRACE    SSL Receive socket read error 0x0.
18:51:55    16844    ACTION_TRACE    5/10/2016 14:51:55.101 [DEBUG] breezeLive.main.FCSConnector [attempt 1 of 60] Trying fallback tunneling connection rtmps://onlinemeeting.connectexample.com:443/?rtmp://localhost:8506/meetingas3app/7/1234567/
18:51:55    17179    PLAYER_TRACE    NetConnectionIO::DoConnect rtmps protocol, HTTP(S) tunneling, tunnel open succeeded.

The corresponding snippet in the server debug log as well as the application logs will read: RTMPT and often reconnect=true:

               Line 23456: 2016-01-17  14:25:06              32260    (s)2641173          Asc-Room               IA_CONNECT      [dID:32, ticket:123456789xyz, phase:, uID:, name:]             New client connecting:  { ip=127.0.0.1, protocol=rtmpt, player=MAC 11,9,971,247, savedConnectionSpeed=undefined, reconnect=true }                        –

[11-05 15:08:05] FCSj_Worker:18 (INFO) params: {bytesdown=0, protocol=rtmpt, ticket=123456789xyz, status=C, reconnect=true, nickname=John Doe, action=register-client, role=v, bytesup=0, session-timeout=12}

Correct configuration of the RTMP VS/VIP is extremely important; a Connect Meeting VS/VIP must have a dedicated FQDN.  It must have its own SSL certificate if SSL is accelerated through the load-balancing device and the VS/VIP must not have an HTTP profile; a TCP profile is needed.

For some additional information about troubleshooting Connect architecture with reference to hardware-based load-balancing devises and SSL accelerators, see the following tech-notes:

The Adobe Connect Deployment Guide on the F5 Website needs Updating

Configuring application-level health monitors for Connect on BIG-IP Local Traffic Manager

9.5 Connect Edge Proxy Server Full Installer

The new 9.5 Connect Edge Proxy Server full installation procedure follows.

Note: This article applies to on-premise Connect customers who have purchased Edge Proxy servers and must install or upgrade to Connect version 9.5. The only possible exception to on-premise exclusivity may be for those who are hosted by a managed ISP that supports external Enterprise Proxy Edge Servers (this latter model is uncommon).

Step 1: Download the Connect Edge Server installer from the URL location provided by the Adobe Connect Support or Customer Service team;  extract and install it with local administrative permissions.

edgeintro.fw

edgeextract.fw

The first installation screen option allows language selection among the following:  English, French, German and Japanese. Click OK to proceed or x (in the upper right) to quit.

edge1a.fw

Step 2 displays the Welcome window in your selected language. Click ‘Next’ to proceed or ‘Quit’ to exit.

edge2a.fw

Note: If you have a previous version installed, this pop-up message will display:

edge2a.fw

Step 3 displays the License Agreement: The administrator conducting the installation must accept the agreement to proceed. Click ‘Previous’ to go to the previous panel, ‘Next’ to proceed or ‘Quit’ to exit.

edge3a.fw

Step 4 displays the option to Select Destination: This panel offers a browse button and facilitates choosing an installation directory. Choose the installation destination, click ‘Previous’ to go to the previous panel, ‘Next’ to proceed or ‘Quit’ to exit.

edge4a.fw

Note: If the destination directory for the installation selected in this panel already exists then the below warning will appear. Click ‘Yes’ to continue or ‘No’ to quit.

edge5a.fw

Note: If the target directory does not exist, this screen will display:

edge2b.fw

Step 5 presents Shortcut Creation options: This screen will facilitate creating the shortcuts in the Start menu. Click ‘Previous’ to go to the previous panel, ‘Next’ to proceed or ‘Quit’ to exit.

edge6a.fw

Step 6 presents a Summary: Click ‘Previous’ to go to the previous panel, ‘Next’ to proceed or ‘Quit’ to exit.

edge7a.fw

edge7a.fw

Step 7 presents a progress screen: This will occur when the installation starts. The installer will extract the files and will try to take a backup of any previous installation. During this panel a command prompt will occur if there were initially any edge services installed. It will create a backup in the same location where it originally existed, but will append “_backup” in the directory name. Wait for the processes to complete.

Note: A clean installation is highly recommended rather than any attempt at installing over older versions of the Edge.

edge8a.fw

After the processes are completed, click Next to proceed:

edge9a.fw

Step 8 offers a GUI, Edge Server Setup Configuration: This panel writes the Edge Server FQDN, the Connect server FQDN, the Cluster ID and the server ports into the Edge server configuration.

edge10b.fw

Example entries follow based on the sample deployment diagram below:

Edge Server Hostname: edge.company.com
Connect Server Hostname: connect.company.com
Edge cluster ID: edge.dmz-edge
Connect Server Normal Port: 80
Connect Server Secure Port: 443

 

95EdgeDMZstunnelOriginstunnel

Step 9 presents the Finish Panel: The installation has completed. Click ‘Done’ to finish the installation:

edge11a.fw

Post installation, the Edge config.ini file, based on our example will contain these relevant entries:

FCS_EDGE_HOST=edge.company.com
FCS_EDGE_REGISTER_HOST=connect.company.com
FCS_EDGE_CLUSTER_ID=edge.dmz-edge=1
FCS.HTTPCACHE_BREEZE_SERVER_NORMAL_PORT=connect.company.com:80
FCS.HTTPCACHE_BREEZE_SERVER_SECURE_PORT=connect.company.com:443

Note: Prior versions of the Connect Edge often employed (although never required) a custom.ini file in the Connect Edge server installation root directory for these entries. The custom.ini would then override the config.ini file in the \conf directory. Placing a custom.ini in the root installation directory is still an option as well as a hazard should one contain stale or wrong entries. The new Edge installer writes directly to config.in through the screen illustrated in step 8.

Once the Connect Edge Server is installed, it must be registered with the origin server or cluster for which it serves as proxy:

On the origin server, register the Edge server by adding the Edge server unique name into the host mappings section of the Connect Management Console; the settings will propagate throughout an origin server cluster from any one of the origin servers:

Start > Programs > Adobe Connect Server > Configure Adobe Connect Server

If the Edge is communicating with the origin server, then you will see a preregistration configuration under the server settings tab:

edgereg1.fw

Add the same unique Edge name into the host mapping fields as follows to register the Edge; this is a manual security mechanism to prevent unauthorized pirate Edge server registration:

edgereg2.fw

Note: The common identification variable in the custom.ini on the origin and the config.ini on the Edge is the cluster ID; following our example is dmz-edge=1 indication the first zone by name; add this to the custom.ini file on the origin(s).

Note: Even a single Edge server warrants its own cluster ID.

edge.dmz-edge=1

For additional information on Edge server deployments including maintenance and troubleshooting, see the articles on the Connect Users Community. Note that the custom.ini file is used in these articles to configure the Edge by overriding the config.ini. As aforementioned, while the new 9.5 installer writes directly to the config.ini, the custom.ini, when used as described will override the config.ini.

The first tutorial listed below discusses the reverse proxy use of the Edge and the second discusses the enterprise proxy use:

Adobe Connect Edge Server Deployment Options: part 1
Adobe Connect Edge Server Deployment Options: part 2

Generating Server-side Logs to Troubleshoot On-premise Connect Deployments

In order to diagnose unexpected behavior within Adobe Connect, it may be necessary for the Adobe Connect Support team to examine server-side logs from an on-premise Connect deployment. The logs directory is located in the Connect (or Breeze – it is not uncommon for Connect upgrades to reside in legacy Breeze directories) directory:

logsdir.fw

Within the logs directory there are sub-directories containing various logs:

logsdir1.fw

The most commonly requested log by the support team, is the debug.log. It can be found in the logs>support directory. With the services running, the current debug log will appear without a date at the top of the debug.log file list. The default rollover is 12 hours generating AM and PM logs each day:

logsdir2.fw

In order to make the debug.log file more useful for purposes of diagnosis, you can enable verbose logging by adding entries to the custom.ini file located in the Connect or Breeze version sub-directory. Here you see it located in a 9.3.1 directory under the Breeze root installation/upgrade directory:

logscustomini3.fw

Before editing the custom.ini file, be sure to create a backup copy of it. Add the following lines in order to enable verbose logging:

HTTP_TRACE=true
DB_LOG_ALL_QUERIES=true

Note that for versions of Connect 9.2 and prior, use yes instead of true:

HTTP_TRACE=yes
DB_LOG_ALL_QUERIES=yes

Save the custom.ini file (be careful not to accidentally change the file type to .txt) and during a scheduled maintenance window, cycle the Connect and AMS/FMS services in order to load the changes and begin verbose logging (note this will bring Connect down while the services cycle):

logssvcs4.fw

There are occasions when it may be prudent to provide more than one log for a more complete diagnosis. To provide a full sample of the various Connect logs without sending a massive historical sample of log files, you may simply stop the Connect services (during scheduled downtime as this will bring down Connect) and rename the entire log directory to log.old. Then upon starting the services back up, recreate the issue being diagnosed and then stop the services.

This activity will generate a new small log directory isolating the issue under scrutiny that you just reproduced in Connect: Zip/compress this new abbreviated log directory with all its fresh abbreviated sub-directories and provide it to the the Adobe Connect Support team to help expedite more exhaustive server-side log analysis. This option is particularly helpful when examining a cluster as each server will have a set of logs. When providing cluster logs, always label each compressed log folder to easily identify the server from which it came.

Note that often when diagnosing unexpected behavior in Adobe Connect Meetings, it may also be prudent to enable client-side Connect addin verbose logging as well.  The relevant client-side logging tech-notes are here:

Enable logging | Meeting Add-in

Troubleshooting Verbose Meeting Addin Logging

SQL Update Warning Message in 9.5.2 Patch can be Safely Ignored

While applying the SQL update in the 9.5.2 patch,  you might encounter a warning message as depicted below:

sql3

This is a general message and can be safely ignored while applying the patch. It is not expected to cause any unexpected inconsistencies in the environment later although full function testing is always prudent after and updates.

Here is the SQL update file that is used in the 9.5.2 patch for your reference :

sql2

Environment :  Adobe Connect Licensed Upgrade

 

Troubleshooting the Flash-less Connect Addin Launch Feature

In order for the Flash-less Connect Meeting Addin to work, you must install the latest Connect Addin as an executable. The reason behind this requirement is that the Addin executable setup inserts needed registry keys within the Windows platform that are necessary to launch the Meeting Addin without Flash. The default, Lightning Addin download by design does not have admin access and cannot configure the Windows client for Flash-less Meetings.

With reference to Macintosh clients, the same rule applies as well. The complete installer needs to be run locally on each client to enable launching the Connect Meeting Addin without Flash.

flashlesssnapshot

If the Flash-less launch continues to be problematic, check for older version of the Addin and uninstall them:

flashlessunsint.fw

If the problem still persists, check also in the registry as there may be residual “connectpro” entries there under HKEY_CLASSES_ROOT:

forceaddin-reg.fw

Manually delete these entries carefully.

New Adobe Connect Support Blog Subscription Option

Now you can stay on top of the new articles and posts by subscribing to the Adobe Connect Support Blog. Simply go to the Adobe Connect Support Blog home page and enter your email address and check off the categories about which you would like to be notified. Click “Subscribe me” and you will begin receiving  regular updates:

subscribe.fw

 

 

AEM/CQ JRE upgrade might not work if the correct default AEM/CQ folder is not selected while deploying 9.5.2

If you’re upgrading from 9.5 to 9.5.2 patch on Adobe Connect servers, please ensure the correct AEM/CQ installation folder is chosen while upgrading the JRE version for AEM/CQ servers else CQ jre might not correctly get upgraded.

Step 4 in the 9.5.2 Readme file for deployment instructions, mentions steps about upgrading JRE for your AEM/CQ servers.  On Step 4e, when you are required to choose the location parent directory of AEM/CQ, make sure to select the correct location where AEM/CQ folder is kept.

By default, it points to the 9.5 default installation directory(\Connect\9.5.0\ )for the install, but we need to make sure it points to root \Connect\ folder inside which AEM/CQ folder is kept and not the 9.5.0 folder.

This would ensure the AEM/CQ and Connect JREs have correctly upgraded and you should see the updated file and folder structure as mentioned in Step 4f  in the Verify Patch deployment instructions.

Affected Environment :   Licensed server 9.5.2 upgrade

If you’ve already applied the patch and suspect running into this problem :

To identify you are running into the same problem as described here, below is how the Connect installed folder structure would look like if somehow the incorrect AEM/CQ location folder is chosen while upgrading AEM/CQ  jre.

** This problem might get caused by the update_cq_jre.exe if not run via following the steps mentioned further below on this article. **

952-6

 

To rectify this problem, follow the below steps : 

Tomcat JRE Update

  • Please remove the current duplicate 9.5.0 folder completely created after running the patch which contains new jre7 folder.

 

  • Now run update_jre_tomcat.exe as an Administrator and on the prompt select the \Connect\9.5.0\  folder for the installation

952-2       952-3

 

AEM/CQ JRE Update

  • Now run update_cq_jre.exe as an Administrator and on the prompt select the root \Connect\ folder for the installation and NOT the 9.5.0 folder.

952-4

  • Finish the install and then it should show the correct jre7 folder created correctly under original 9.5.0 folder like below :

952-5

 

Identifying the right Visual C++ libraries for Connect 9.x releases

Environment :

Adobe Connect Licensed 9.x

If you’re planning to deploy a 9.x Patch/Full release in your Adobe Connect environment, make sure you have the following Visual C++ libraries installed on the Adobe Connect server :

Additionally, if you are installing 9.4.2f patch, the below additional runtime might be needed as well on the server to avoid any missing DLLs :

Important Information :

Above Visual C++ Runtimes are required for facilitating uploading of documents in Adobe Connect content library or sharing them in meeting rooms.

If you are experiencing any issues uploading any contents on the account or meeting room like ppt,pdf format etc, make sure to verify the above libraries should be installed on all the Adobe Connect servers.

Contact Support for any further information or queries.

 

Changing the License Serial Key in Connect

This article applies to on-premise and managed ISP Connect users. It does not apply to multi-tenancy hosted or ACMS.

On rare occasions it may be necessary to change the serial key in Adobe Connect. Here are the steps:

  1. Navigate to: \Connect_installation_directory\appserv\conf\config.ini and change the value of  SERIAL_KEY=  to reflect the new serial number
  2. In \Connect_installation_directory\custom.ini,  if there’s a serial key value listed (SERIAL_KEY=), replace it there as well.
  3. Using MSSQL Studio Express (or your choice of SQL editing options), view the serial key currently being used by Connect by running this command: SELECT * from pps_accounts WHERE name=’Enterprise Account’
  4. To get Connect to accept the new license you must change the serial key that is currently in the database by running this SQL command: UPDATE pps_accounts SET serial_key = ‘NEW_SERIAL_NUMBER’ WHERE serial_key = ‘OLD_SERIAL_NUMBER’
  5. Restart the services: Application Server (Connect) and the Meeting Server (AMS or FMS depending on the version of Connect) services.fw
  6. Open the Administration Console (port 8510 locally on any Connect server)

connconfig.fw

7. Go to License Settings and upload the new license file.

connconfiglic.fw

8. Restart the AppServer (Connect) and the Meeting server (AMS or FMS depending on version) again and the  new license file will be applied

services.fw

Troubleshooting: If there are any problems, do the following to troubleshoot:

  • Shut down the Connect and AMS or FMS Services
  • Open and verify \Connect_installation_directory\appserv\conf\config.ini and update the entry for SERIAL_KEY
  • Open and verify  \Connect_installation_directory\custom.ini and update  the entry for SERIAL_KEY
  • Open SQL Server and choose the Connect database and run the following script (replacing the text as appropriate):

“Input New Serial Key Here” with the New Serial Key but leaving the quotes.
DECLARE @NEW_SERIAL VARCHAR(32)
SET @NEW_SERIAL=’Input New Serial Key Here’

UPDATE PPS_CONFIG
SET VALUE = @NEW_SERIAL
WHERE SECTION=’cps’ AND NAME=’serial_key’

UPDATE PPS_ACCOUNTS
SET SERIAL_KEY = @NEW_SERIAL
WHERE ACCOUNT_ID=7

UPDATE PPS_ENUM_DATA_HOSTS
SET LICENSE = @NEW_SERIAL
WHERE HOST_ID > 0

db.fw_

  • Start the Connect and FMS services

Problems will ensue when the license is reducing the allowed usage of Connect (if you are downsizing) and you leave an overage in place. For example, if you have 100 meeting hosts assigned, and you are changing to a license that only allows 50 named meeting hosts then when you  apply the license you will get an error unless you have reduced the number to accommodate the new licensed restriction.

On-premise Connect without Webinars Throws an Error when Moving Meetings

On-premise Connect customers running version 9.5 who do not have the Webinar option enabled will see an error when trying to move meetings via the GUI in Connect Central:

Webinar95Bugon-prem.fw

This will be fixed in an upcoming release and in the meantime we have two workaround options: