Posts in Category "Install"

Meeting Connection Test Page Fails at Step Two

Issue: Meeting test page fails at step 2; there are many possible causes for this:

1. If it is failing when you attempt to connect through a remote edge but does not fail when you connect directly to an origin, then the most likely culprit is name resolution. Audit your remote Edge name-resolution against these tutorials:

Adobe Connect Edge Server Deployment Options: part 1

Adobe Connect Edge Server Deployment Options: part 2

2. MS Patch MS12-006 (installed on client workstation) has been problematic:

MS Update MS12-006 (http://technet.microsoft.com/en-us/security/bulletin/ms12-006) causes step two of the meeting test page to fail; more details on the Patch regarding TLS/SSL can be found here:  http://support.microsoft.com/kb/2643584

3. Other network and client-side constraints and configuration issues can cause this test to fail. In order to troubleshoot, try the following tests and gather the data from them:

For on-premise customers who are experiencing a failure at step two, try hitting the test page on the Adobe hosted service:

https://my.adobeconnect.com/common/help/en/support/meeting_test.htm

Firefox and IE should look like this:

FFMT.fw

Chrome looks like this; notice the absence of an addin:

ChromeMT.fw

To drill deeper, click on the Send Results link:

FFMTSendResults.fw

And then click on the Details link

FFMTSendResultsDetails.fw

FFMTSendResultsDetailsPV.fw

Note: Currently, as of the writing of this blog entry, in Connect 9.1, there is a known bug that prevents the addin version from appearing in the the meeting test-link output. In the screen capture above, you only see the Flash-player version. This is scheduled, tentatively to be fixed in Connect 9.2.

You may copy and paste the output from the details link to assist the Adobe Customer Care team with diagnosing meeting connection issues. Do not use the Send Results link on the meeting connection test page, but manually copy and paste the results into an email.

On the meeting test page, if you find that you need the addin, (excluding Chrome) simply click on the Downloads link:

FFMTDownloads.fw

Note that the Flash-player download link is also available:

FFMTDownloadsAddinFP.fw

The meeting test link has some limitations, for example, it will not diagnose tunneling RTMP(S) over HTTP(S). To figure out if you are tunneling and thereby experiencing additional latency and connection drops caused by something on the network blocking RTMP, (usually a proxy or firewall), look in the upper left corner of a live meeting room and see if there is a “T” in the output when you click on the green connection indicator:

.tunneljpgOctagon

All of this information is vital when trying to diagnose meeting connection issues.

Look for an upcoming blog article on diagnosing and traversing sources of network blockages of RTMP(S) resulting in tunneling RTMP(S) over HTTP(S) and causing increased latency in a meeting. You should have a lightening fast connection and it is very doable with the right configuration:

no-tunnel-star.fw

Stunnel does not Startup with Connect

Problem: stunnel does not start up with Connect

Although stunnel can be installed as a service, it doesn’t load the stunnel.conf file(!) one workaround is to not setup the services to run automatically but to auto-run these batch files at startup:

Note: This tech-note assumes stunnel is installed in c:\Connect\9.0.0.1\; be sure to adapt the scripts accordingly.

Origin server startup.bat:

@ECHO ON
net start FMS
net start FMSAdmin
net start ConnectPro
net start CPTelephonyService
c:\Connect\9.0.0.1\stunnel\stunnel.exe stunnel.conf
@ECHO OFF Origins stop.bat:

@ECHO ON
net stop ConnectPro
net stop CPTelephonyService
net stop FMSAdmin
net stop FMS /y
@ECHO OFF

If you have remote Edge servers, use these; they includes cache clearing maintenance.

Edges start.bat:

@ECHO ON
net start fms
ping 1.1.1.1 -n 1 -w 10000>nul
net start fmsadmin
c:\breeze\edgeserver\stunnel\stunnel.exe stunnel.conf
@ECHO OFF

Edges stop.bat:

@ECHO ON
net stop fmsadmin
ping 1.1.1.1 -n 1 -w 10000>nul
net stop fms
ping 1.1.1.1 -n 1 -w 20000>nul
del /Q /S c:\breeze\edgeserver\win32\cache\http\*.*
ping 1.1.1.1 -n 1 -w 10000>nul
@ECHO OFF

Run > gpedit.msc
Local Computer Policy > Computer Configuration > Windows Settings > Scripts (Startup/Shutdown)
Batch files are assigned as startup & shutdown scripts. This is in addition to being available to be run manually.

Providing Diagnostic Data to Expedite Solutions for Connect Meeting Issues

Issue: Anything that may happen during a meeting which has a pejorative effect on end-user experience.

Solution: In Connect 9.1 we have a great diagnostic option in the meeting room. You can immediately pull logs from any meeting to diagnose:

If you click Help>About Adobe Connect, while holding down the Ctrl key, the debug logs will appear int he meeting room and you will have the option to copy them to your clipboard.

log-mtg.fw

 

log-mtg1.fw

Sending me these, along with the RTMP string  Help> About Adobe Connect, while holding down the Shift key – this will be most helpful from the client experiencing the extreme latency.

rtmp-mtg.fw

Now if you want to take it even one step further and provide a client-side view of the meeting:

The instructions for enabling client-side logging are here: http://helpx.adobe.com/adobe-connect/kb/enable-logging-acrobat-connect-professional.html

Providing all this data along with the date and time (including timezone) and Meeting URL of any issue, will greatly expedite analysis and solution.

Resource Constraints cause Connection Read Error in Logs on Clustered Connect Servers

Issue: FCSj_IO:4 (x) – Connection read error: -1 LP: 5345 RP: 8506 URI: rtmp://localhost:8506/meetingapp/7/12345678

I have seen that in some VMWare environments that are very overtaxed for resources, latency between/among the clustered Connect servers on ports 8507 (and also 8506 though 8506 does not cause this error), can cause problems. Intra-cluster latency should never exceed 2-3ms. When it does we see intermittent errors and can also see crashes.

I had one unnamed customer who had a particularly weak infrastructure and  I could predict his crashes; he was doing back-ups and running other tasks at a certain time weekly that would severely hamper network connectivity for about an hour; these tasks were so all-consuming on the network, they turned every Connect cluster resource into an individual asset on its own island. The Connect logs bore this out and we knew with precision what was going on and could predict his call or email based on his maintenance schedule. He knew he needed to upgrade his infrastructure and in the meantime we worked out a reaction plan to deal with the issue; it included:

  • Place a higher than normal percentage of cache on each server to limit invoking shared storage during maintenance (see page 57)
  • Set the JDBC driver reconnection string for Database connectivity robustness
  • Plan heavy Connect usage around network and server maintenance activities and when possible, do your Connect server maintenance activities at the same time as well.

Connect & Unified Voice (UV) Traffic Flow Diagram

Issue: Plan for the flow of traffic to enable UV among the various components in any Connect deployment: Connect, Flash Media Gateway (FMG), Session Initiation Protocol (SIP)

There are numerous documents on the topic of Unified Voice (UV) with Connect:

This diagram shows the flow of traffic and the protocols used for UV with Connect and is offered as a planning and a troubleshooting tool; click on the diagram to expand it for viewing:

 

Connect_FMG_Flow

The Adobe Connect Deployment Guide on the F5 Website needs Updating

Issue: Be careful when following the Adobe Connect Deployment Guide posted on the F5 Website. While the article is be helpful, there are some ambiguities that can lead to trouble. I have tried to update their deployment guide but have not succeeded; the LTM is the most popular load-balancing device and SSL accelerator in the Connect niche and when it is set up properly it works splendidly. Here are corrections, updates and things to watch out for when deploying Connect behind an LTM:

1. Do not use an HTTP profile for an RTMP VIP. An HTTP profile for RTMP VIPs may affect playback of video as well as break remote Edge connectivity. Remember that you have two servers running on each box, a Tomcat application server and an FMS server. Do not treat the FMS server as though it were an application server; RTMP is a streaming protocol that requires a TCP profile at the HLD VIP.

2. Use the health monitor documented here for LTM.

3. Do not use session-awareness or stick-sessions even if you use SSL. The Round Robin algorithm should float freely to the Tomcat application pool.

4. Do not use Nagle’s Algorithm with SSL; it will have a negative effect on performance.

Review this general Connect pool/cluster configuration tutorial before configuring BIG-IP LTM with Connect: Adobe® Connect™ server pools/clusters and hardware-based load-balancing devices with SSL acceleration

Brad’s Short-list for Connect Cluster SaaS Monitoring Options

There are many options on the monitoring theme that are worth considering when trying to decide how to keep trach of Connect server resources in a cluster. Articles describing clustered environments are on the Connect Users Community : http://www.connectusers.com  Simply search the User’s Community using the keywords: cluster, pool, edge, SSL, etc.

To effectively monitor your Connect cluster SaaS options can sometimes be cost effective than home-spun solutions; here are some staff picks with some commentary:

Sumologic- It resembles Splunk. The main difference is that Sumologic is hosted and managed externally and Splunk is hosted and managed on-premise. With Sumologic, there is not any need for software licensing, hardware investments or internal administrator expertise.  Splunk offers a similar service called splunk>storm, but it is not as mature as Sumologic and lacks some of the alerting capability found in Sumologic.

Loggly - An alternative to Sumologic could be Loggly which offers a similar service; it seems that the alerting service is not exactly built in.  It requires a little more work and is called AlertBirds.

Note: It is possible to take an on-premise option like Cacti and port it to Sumologic, so you could effectively kill 2 birds with one stone.  You can setup a forwarder in 30 seconds and be searching the logs in no time at all.

Monitis – Provides capabilities similar to those of Nagios along with external monitoring.  The Monitis community writes custom monitors thereby enriching the options.

LogicMonitor – An alternative to Monitis could be something like LogicMonitor.  You may be able to port your existing Nagios checks over to it (check and verify).  This si a simple solution, installing the monitor and having basic checks like CPU, Memory, Bandwidth, Disk Usage, Disk IO and external ping, http, https and udp monitors setup would take all of 20 minutes.

Pingdom- An alternative to RedAlert at a lesser cost.  It is trusted by millions and is easy to use and has more endpoints than comparable options.  It takes five minute setup.

The beauty of a SaaS monitoring solution is that you do not need to worry about scaling your monitoring solution every time you scale your Connect architecture.  You can have a single solution for 20 Connect Clusters vs having to add Cacti servers, Nagios servers, Splunk architecture and licensing to handle the additional monitoring needs commensurate with expansion.  With a SaaS solution, there in not any build-out time.  You can literally have 20 monitors up and running in under an hour, and work on adding additional ones at your leisure in between casts with your new Deceiver 8 Fly Combo.

With reference to basic on-premise monitoring, make sure you use standard perfmon counters for things like CPU, Memory etc. For meeting count and meeting user monitoring you may use the FMSAdmin API with scripts to make various calls and then parse the data and pass it to an option such as Cacti.  To insure robustness, the FMSAdmin service should be restarted routinely. You could also use similar counters to pull data directly from the Connect database, but this is not without risk as Connect updaters and upgrades can introduce changes that may require rework of your custom counters.

Adobe Connect Servers and Hardware-based Load-balancing Devices

This updated article offers a best-practice configuration of a basic Connect pool/cluster behind a high-end, application-aware HLD such as F5 BIG-IP LTM. This article does not discuss SSL-acceleration. This article does not describe all the possible configurations, but offers a general working example of a basic HTML/RTMP non-SSL cluster/pool of Connect servers.

Adobe Connect Server Pools and Hardware-based Load-balancing Devices: http://www.connectusers.com/tutorials/2009/04/load_balancing/index.php

Connect license the upload fails: License file has a quota of type ‘bandwidth-quota’

Issue:  Upon uploading an updated connect license (perhaps after expansion or upgrade) the upload fails with the error:  License file has a quota of type ‘bandwidth-quota’ (account) with a limit less than amount currently used

Workaround: Edit the Connect SQL database as follows:
1.      Document the current value for ‘USED’ field in the pps_acl_quotas value for the QUOTA_ID that is set to 8.
2.      Execute the follow UPDATE statement to set the value to a lower value
a.      update pps_acl_quotas set used = 900000000 where quota_id = 8
b.      Note: For example if the LIMIT value is 1000000000 you could use the following: UPDATE pps_acl_quotas USED = 900000000 where Quota_ID = 8
3.      Upload the License.txt file in Connect
4.      If you don’t get the upload error
5.      Re-open the pps_acl_quotas table and verify that the LIMIT for QUOTA_ID is not 3500 (this will be the new concurrent user limit with the new license.txt file)
6.      If you  get the error when uploading, run the SQL statement, replace the 900000000 with the original number from step 1.

Connect 9.1 Installer Increments Sub-directories

Issue: The Connect 9.1.1 installer increments directories when re-installing

Method: Re-install 9.1.1 over an existing 9.1.1 installation (not an upgrade)

Result::  When you re-install 9.1.1, it adds a 911 sub-directory within the existing 911 directory

Expected: Re-install into original 911 directory

Workaround: After re-installing examine all edited files in the original directory and make sure to add any changes that did not populate (custom.ini, server.xml, etc.) and drive on. This issue is scheduled to be fixed in an upcoming updater, possibly 9.2.