Archive for November, 2013

Pulling Applicable Logs for Telephony / UV Issues

Typically when on-premise Adobe Connect customers experience telephony issues or UV related issues in their Adobe Connect Meeting rooms, Adobe Support will request specific logs from the customer’s environment.  These logs may come from different places depending on your Connect deployment.  Below is a good guide to understand how to grab just the applicable logs that Connect Support will need.

Typically, for telephony or UV issues these logs will be:

  • application.log
  • core.log (for FMG/UV if applicable)
  • SipLog (for FMG/UV if applicable)
  • TelephonyService.log

and any of the following (‘adaptor logs’ as I’ll refer to it as in this article), depending on the Telephony Integration/Provider you are using:

  • Arkadin_Adaptor.log
  • Avaya_Adaptor.log
  • Intercall_Adaptor.log
  • MeetingOne_Adaptor.log
  • MeetingPlace_Adaptor.log
  • Premiere_Adaptor.log
  • Premiere_EMEA_Adaptor.log

One thing to understand is that if you have a ‘cluster’ of Adobe Connect origin servers, your telephony services for the meeting in question may or may not open on the same server as the meeting itself.  So you could have a meeting running on one node in the cluster, while the actual telephony services are running on another.  Also, if you are running an FMG server (for UV), the FMG services may or may not be on the same server that is running the actual meeting.

Here’s how to collect the applicable logs to submit to Adobe Support if they request the ‘telephony’ logs for a meeting:

For Adobe Connect 9.x:

Application.log (also sometimes referred to as the ‘Meeting log’)

This is the log that contains all the logging for the meeting itself.  It will contain the telephony service connection in which you can identify what server the telephony services actually connected on.  It will also tell you  (if applicable) what server the FMG connection happened on.  This log will also show all the user activity in the meeting (including phone numbers, etc.). This is very crucial for Adobe to obtain when troubleshooting an audio issue in an Adobe Connect Meeting.

It is located in: [Root Connect Install Directory]\logs\support\apps\_defaultVHost_\meetingas3app\7\{sco-id} where ‘sco-id’ is the actual numeric value of the Meeting room’s sco-id.

To find the sco-id of the meeting room, you can log into Adobe Connect’s web manager and navigate to the Meeting Information page in the UI.  The URL of that page will look something like this:

http://myConnectURL/admin/meeting/sco/info?account-id=7&principal-id=11033&sco-id=11115&tab-id=11003

(where the values for principal-id, sco-id-, tab-id are obviously just example values for the sake of this article)

The ‘sco-id‘ from that URL will be the name of the folder you are looking for in the above directory for finding the application.log.

Note: If you have a cluster for Adobe Connect that contains multiple servers, you may have to search every server in the cluster for this file.  You may also notice that if the meeting room in question is used over and over, it will open eventually on all the servers in the cluster, so you may in fact have the sco-id folder present on more than one (or all) of the servers, but the actual application.log for the session you are looking for will only be on one server.  Each session will get it’s own Application.log.  So you will see multiple (eventually) versions of the application.log in this folder over time.  You need to look at the timestamp of the logs to make sure you are getting the right one for the day/time the session in question occurred.  Once you identify the right application.log for the meeting session in question, you need to open it up and look for the following logging looking similar to this:

(you can search the log for the values in red):

2013-11-19	10:21:32	94648	(s)2641173	get-telephony-service API result obtained.	-
2013-11-19	10:21:32	94648	(s)2641173	get-telephony-service API successful	-
2013-11-19	10:21:32	94648	(s)2641173	  .get-telephony-service result object:  [Object]	-
2013-11-19	10:21:32	94648	(s)2641173	    \\	-
2013-11-19	10:21:32	94648	(s)2641173	    .host [Object]	-
2013-11-19	10:21:32	94648	(s)2641173	        \\	-
2013-11-19	10:21:32	94648	(s)2641173	        .external-name [string]= connect01.mycompany.com	-
2013-11-19	10:21:32	94648	(s)2641173	        .ip [string]= 10.11.123.44	-
2013-11-19	10:21:32	94648	(s)2641173	        .name [string]= connecthost01	-
2013-11-19	10:21:32	94648	(s)2641173	        .service-host-id [string]= 15647477	-
2013-11-19	10:21:32	94648	(s)2641173	    .subcode [undefined]	-
2013-11-19	10:21:32	94648	(s)2641173	    .code [string]= ok	-
2013-11-19	10:21:32	94648	(s)2641173	hostName: connecthost01, hostIp: 10.11.123.44	-

where the hostName will be the server in which the telephony services opened on.  The logging snippet will contain that hostName and the hostIP as well as the external name (FQDN) of that meeting server.  The above examples in BLUE are just merely generic values for the sake of this article.

** IF applicable (if you are using UV), also look for the following (you can search the log for the values in red):

 

2013-11-19	10:21:44	94648	(s)2641173	FMGServiceConnector.asc:: get-fmg-service API onResult obtained	-
2013-11-19	10:21:44	94648	(s)2641173	FMGServiceConnector.asc:: get-fmg-service API successful	-
2013-11-19	10:21:44	94648	(s)2641173	FMGServiceConnector.asc:: fmgHostName=fmghost02, fmgHostIp=10.11.123.456	-
2013-11-19	10:21:44	94648	(s)2641173	FMGServiceConnector.asc:: FMG client has already connected, connected ip= 10.11.123.456	-
2013-11-19	10:21:44	94648	(s)2641173	FMGServiceConnector.asc:: Setting connected fmg ip:10.11.123.456	-
2013-11-19	10:21:44	94648	(s)2641173	FMGServiceConnector.asc:: fmgHostName saved = fmghost02	-

where the fmgHostName and fmg ip address will be visible.  The above examples in BLUE are just merely generic values for the sake of this article.

 

Once you identify the server the Telephony Services opened on (and if applicable, UV), move on to getting those logs…

TelephonyService.log

This is the log that contains all the logging for the Adobe Connect Telephony Services.

It is located in: [Root Connect Install Directory]\Breeze\logs\telephony

Note: If you have a cluster for Adobe Connect that contains multiple servers, refer to the info above on making sure you know what server to pull this log from.

{XXXXX}_Adaptor.log

This is the log (depending on your telephony provider) that includes all the logging for the communication between Adobe Connect and the telephony provider’s bridge.  It will contain the communication back and forth and the messages sent to and from the telephony provider using that provider’s API, as well as applicable responses. The ‘XXXXX’ value in the name will obviously be the name of the provider (e.g. Arkadin, PGI, Intercall, etc.).

It is located in: [Root Connect Install Directory]\Breeze\logs\telephony

Note: If you have a cluster for Adobe Connect that contains multiple servers, refer to the info above on making sure you know what server to pull this log from.  It will be the same server that the TelephonyServices.log will come from.

Of course in some circumstances, Adobe Support will also need to work with the telephony provider to obtain logging from their side, should the Adobe Connect logs not yield enough data to continue with the investigation.

core.log 

This is one of two applicable FMG related logs.  If you are using UV, this will be necessary to obtain for support to review the FMG logging.

It is located in: [Root Connect Install Directory]\Flash Media Gateway\{FMG version}\log

Note: If you have a cluster for Adobe Connect that contains multiple servers, refer to the info above on making sure you know what server to pull this log from.

SipLog (this will be date/timestamped in the actual name of the log)

This will be the log that will contain the communication between FMG and the SIP server/service.  If you are using UV, this will be necessary to obtain for support to review the FMG logging.

It is located in: [Root Connect Install Directory]\Flash Media Gateway\{FMG version}\log

Note: If you have a cluster for Adobe Connect that contains multiple servers, refer to the info above on making sure you know what server to pull this log from.

 

7.x Recordings Not Playing after Upgrading to 9.1.1

If you have recently upgraded an Adobe Connect on premise deployment from version 7.x to 9.1.1 (with various 8.x steps possibly in-between), you may encounter an issue where older recordings no longer launch.  However, newly created recordings open and playback without issue.

If this is the case, please check the following directories…

First check:

[Root Connect Install]\9.1.1\appserv\common\meeting\shell

This directory should contain the following SWF files:

  • BreezeUIComponents.swf
  • CorePodCollection.swf
  • meeting.swf
  • meeting_sgn.swf
  • shell.swf
  • shell_sgn.swf
  • StamperSymbols.swf

If they do NOT exist in that directory (but just the ‘breezeLive’ folder and xml files are the only files present), please download the files from here and place into the directory.

Then check:

[Root Connect Install]\9.1.1\appserv\common\meeting\launcher

This directory should contain the following SWF files:

  • listener.swf
  • openmeeting.swf
  • openmeetingversioncheck.swf

If they do NOT exist in that directory, please download the files from here and place into the directory.

After confirming these files are now in those directories, retry launching an older Connect recording.  No restart is required.

Event Module Tutorial Collection

The below is a great collection of Event Module tutorials by both Adobe and our some of our partners.  These cover everything from administration, creation, migration, reporting, best practices, and API integration.

 

Creating an Event in Adobe Connect 9
by video2brain
In this tutorial, you’ll see how to create a new event in Adobe Connect 9 and add it to the Event Catalog

Creating and Editing Event Templates
by Alistair Lee
In this video tutorial, Alistair Lee shows you how to create, edit and manage event templates – new to Adobe Connect 9.

Event Administration in Adobe Connect 9
by Alistair Lee
In this video tutorial, Alistair Lee walks through the new features available to Event Administrators in Adobe Connect 9

Adobe Connect 9: Event Migration Guide
by Alistair Lee, Adobe Systems
This tutorial features a 23-page PDF guide on Adobe Connect 9 events. It highlights the new features introduced in version 9 of Adobe Connect and discusses the impact on events that have migrated.

Creating a Two Person Event Template with Adobe Connect
by Alistair Lee, Adobe Systems
In this tutorial, you’ll see how to quickly create a new event template that features two or more speakers.

Adobe Connect Events Overview
by Alistair Lee, Adobe Systems
In this video, Alistair Lee walks through an overview of the new Adobe Connect Events module.

Adobe Connect best practices for large events and seminars
by Rocky Mitarai, Adobe Systems
This checklist is an invaluable resource for any producers of large webinars and events on Adobe Connect.

Resetting the Default Event Templates in Adobe Connect
by Alistair Lee, Adobe Systems
This tutorial walks through the steps required to reset a default template.

Campaign Tracking in Adobe Connect 9.1
by Alistair Lee, Adobe Systems
In this video tutorial, Alistair walks through some of the new features in Adobe Connect 9.1 that make measuring campaigns and optimizing your promotional channels more intuitive.

Event Registration using Adobe Connect API’s
by Dustin Tauer, Easel Solutions
In this tutorial, Dustin covers how to use the Adobe Connect API’s to register a user for an event.

 

Meeting Add-in | Mac OS X Mavericks 10.9, 10.8, 10.7 with Safari 6.1 or 7

In case you have not seen, we have published a brief article on the issues with using the Adobe Connect Meeting Add-in with Mac OSX Mavericks and newer versions of  the Safari browser.

Details and instructions (in an additional linked article) on how to suppress the restrictions put in place by the newer browsers can be found here:

http://helpx.adobe.com/adobe-connect/kb/connect-add-in-osx-mavericks-109.html

Upgrade to Adobe Connect 9.1 causes Avaya Adaptor to be broken

Problem :

If you’ve upgraded your Adobe Connect server to version 9.1 and you already have Avaya adaptor configured, you might find it broken after the upgrade.

You might also run into this issue if you have a fresh installation of 9.1 and you are configuring Avaya adaptor for the first time

Reason :

The adaptor path is incorrect in the telephony configuration files in version 9.1

Environments Affected : Connect 9.1.1 Licensed

Solution :

  • Locate the following folder on your Adobe Connect 9.1.1 root installation : {Connect-Root}\9.1.1\TelephonyService\conf
  • Create a backup copy of  telephony-settings.xml file
  • Open the file in a text editor and locate the following lines for Avaya adaptor : <telephony-adaptor class-name=”com.macromedia.breeze_ext.telephony.AvayaAdaptor” enabled=”true” id=”avaya-adaptor”>
  • Replace the line with the following : <telephony-adaptor class-name=”com.macromedia.breeze_ext.telephony.Avaya.AvayaAdaptor” enabled=”true” id=”avaya-adaptor”>
  • Save the file and reopen the file with IE to make sure there are no errors.
  • Next, Create a backup copy of  telephony-capabilities.xml file
  • Open the file in a text editor and locate the following lines for Avaya adaptor : <telephony-adaptor class-name=”com.macromedia.breeze_ext.telephony.AvayaAdaptor” enabled=”true” id=”avaya-adaptor”>
  • Replace the line with the following : <telephony-adaptor class-name=”com.macromedia.breeze_ext.telephony.Avaya.AvayaAdaptor” enabled=”true” id=”avaya-adaptor”>
  • Save the file and reopen the file with IE to make sure there are no errors.
  • Restart the Adobe Connect & Telephony services.

 

Connect on VMWare – some deployment tips

Issue: VMWare is ubiquitous in the enterprise and while it opens up huge potential for management of the Connect infrastructure, it must be planned and executed with an eye toward robustness.

This advice is gleaned from conversations with senior persons on our operations team as well as from support cases generated by various customers with on-premise VMWare deployments of Connect.

One of the most important and often overlooked variables about virtualization is to make certain that  VMware is compatible with all the underlying components of the server and network architecture. The infrastructure supporting VMWare must be verified by VMware under their Hardware Certification Program or Partner Verified and Supported Products (PSVP) program; be sure to use certified hardware.

Here is the link to the compatibility reference:  http://www.vmware.com/resources/compatibility

With Connect you must consider both Tomcat and  FMS; the former can run on most anything, while the latter is a bit more demanding; RTMP can be acutely;y affected by latency and packet transmissions. If you notice unpredicted latency or a surprise crash of FMS with Connect 9.1, a good test would be to check the network components; sniff for packet transmission issues – have the vNIC of the guest VMs configured to use VMXNET3; this is a good place to start.

With reference to recommendations and best practices, it really depends on the VMware infrastructure adopted. The following references serve as a guide for an enhanced environment:

Enterprise Java Applications on VMware – Best Practices Guide: http://www.vmware.com/resources/techresources/1087

Best Practices for Performance Tuning of Latency-Sensitive Workloads in vSphere VMs: https://www.vmware.com/resources/techresources/10220

Performance Best Practices for VMware vSphere 5.1: https://www.vmware.com/resources/techresources/10329

The key with Network Storage is speed. If you lose connectivity to the shared storage then only what is cached on the origins will be available.

Shared storage requirements

  • Disk specs: 10,000–15,000 RPM — Fibre Channel preferred
  • Network link: TCP/IP — 1GB I/O throughput or better
  • Controller: Dual controllers with Active/Active multipatch capability
  • Protocol: CIFS or equivalent

Avoid, virtualizing the Connect database if possible.

I have seen that in some customer-based VMWare environments that are overtaxed, that latency among the servers on 8507 (and 8506), can cause problems. Intra-cluster latency (server to server communication) should never exceed 2-3ms. When it does we see intermittent crashes. I had one customer who had a particularly weak infrastructure and for whom I could predict his crashes; he was doing back-ups and running other tasks at a certain time weekly that would tax and hamper network connectivity for about an hour; these tasks were so all-consuming on the network, they turned every cluster resource into an individual asset on its own island. The log traces bore this out and we knew with precision what was going on. He knew he needed to upgrade his infrastructure and in the meantime we worked out a reaction plan to deal with the issue; it included:

  1. Place a higher than normal percentage of cache on each server to limit invoking shared storage
  2. Set the JDBC driver reconnection string for Database connectivity
  3. Plan Connect usage around these maintenance activities and when possible, do Connect maintenance activities at the same time as well – not very difficult as these were after hours, but being a  global operation, still not a given.

Presenter 9.0.2 Now Available

Presenter 9.0.2 is now available as an updater here:

http://www.adobe.com/support/downloads/product.jsp?product=153&platform=Windows

This is the update for existing Presenter 9.0 customers.

The full installer for Adobe Presenter 9.0.2 will be made live on November 18th – which means the existing build of Presenter 9.0 will be replaced on the Adobe online store, trial downloads, our Licensing system, and DVDs, with Presenter (9.0 + 9.0.2) consolidated installer.

The normal Presenter release notes have therefore been updated to include Presenter 9.0 PLUS Presenter 9.0.2 features and is live now. View the release notes here.

Release notes specific to Presenter 9.0.2 are found here and point out Presenter 9.0.2 details separately.

You can also download and install the update directly from inside of Adobe Presenter 9 by going to the Adobe Presenter > Help > Updates menu (pictured below).

 

presenter902

 

 

In the end (after updating), you should see the following version when going to: Adobe Presenter > Help > About Adobe Presenter…

presenter902a

 

 

Adobe Connect Mobile 2.2 Now Available

We have released Adobe Connect Mobile 2.2 for iOS and Android devices, and this new version is now live on iTunes and Google Play stores.

With this latest release, we’ve continued to focus on enabling the ability to fully host meetings from any device, as well as enable mobile learning.  The key enhancements we have added in this release include:

·         Widescreen (16:9) webcam viewing support

·         Manage audio conferencing from all devices (not just tablets)

·         Enhanced mobile learning, with additional quiz types in virtual classes

·         Optimized support for high-definition Android and iOS Retina displays

·         And more – More information on features and updates can be found in the release notes here.

Check out the official blog post that announces this.

 

Presentation Edited in more than one Version of Presenter Hangs on Playback when Published

Issue: Presentation edited using more than one version of Presenter hangs on playback when published.

There is more than one possible cause for this symptom, however when passing around a Presentation for editing by various contributors, you may run into backward-compatibility issues with text highlighting. Text highlighting was introduced in Presenter 9.0 and is not backward-compatible to Presenter 8.1, or earlier versions of Presenter.

Workaround: Rather than rework the content in a single version of Presenter, when text highlighting causes the hanging problem, you may delete the cctexthighlighting entry from the vconfig.xml file in the published output:

  • Publish the problematic content locally into a zip and extract the zip file.
  • Go to “<published output>/data/vconfig.xml”

Untitled-1.fw

  • Open the vconfig.xml file using any XML editor (I used Dreamweaver)
  • Delete the following variable in the vconfig.xml file: <uishow name=”cctexthighlighting” value=”true”/>

Untitled-2.fw

  • RE-zip the locally published Presentation
  • Publish the edited and zipped Presentation to Connect:

Untitled-1.fw

Untitled-3.fw

Play the published Presentation either as standalone content or shared in a Connect Meeting Share Pod.

Tunneling with RTMP encapsulated in HTTP (RTMPT) should be avoided as it causes latency

Tunneling with RTMP encapsulated in HTTP or RTMPT should be avoided as it causes latency that can have a negative impact on user experience in a Connect meeting. In rare circumstances,the latency commensurate with tunneling RTMP encapsulated in HTTP, can become so acute that it renders Connect unusable for affected clients. The performance hit commensurate with tunneling is one of the primary reasons we continue to deploy Connect Edge servers as they often can replace third-party proxy servers that are often the cause of tunneling latency..

While the amount of acceptable latency depends on what one is doing in the room; RTMPT tunneling affects most activities. Some activities, such as screen-sharing are more bandwidth intensive than other activities such as presenting an uploaded PowerPoint from within a meeting room; The high latency commensurate with RTMPT tunneling would affect the former more than the latter. VoIP is often the first thing to make the effects of high latency felt. Here is some feedback from a test I did while on site with a client dealing with tunneling because of their refusal to pipe RTMP around a third-party proxy:.

External user tunneling during test:
Spike at 3.10/3.02 sec
Latency 403/405 ms up to 3.53/3.52 sec up .064 down 118
When latency peaks to 2.6/2.4 sec I get a mild interruption to the audio V
Video pauses momentarily when the latency spikes

Internal user with direct connection:
2 msec / 1 msec Up 0.08 kbits down 127 kbits
No pauses, delays or spikes

Tunneling should only be considered as a fallback mechanism or safety net to allow connections when RTMP is blocked due to something unplanned or for a few remote clients who must negotiate specific network obstacles. When RTMPT is the default by network design, you will need to limit your activities within Connect to those feature that use the least bandwidth.

The picture below shows a direct connection over RTMPS on 443 is being blocked somewhere on the client’s network and the fallback mechanism built into Connect of tunneling RTMP encapsulated in HTTPS is the fallback path. This is usually caused by either proxy servers or firewalls or both – any application-aware appliance on one’s network that sees the RTMPS traffic on 443 and recognizes that it is not HTTPS is a potential obstacle; RTMPS traffic is on port 443 and while it is disguised as HTTPS, it still may be blocked. The result is tunneling, indicated by the “T” in the output:

.tunneljpgOctagon

Compare with a connection without tunneling:

no-tunnel

The recommended steps for anyone experiencing persistent tunneling, is for their network engineers to trust the source IP addresses of the Adobe hosted, ACMS, managed ISP or on-premise Connect/FMS server VIPs in order to prevent the blocking of RTMPS traffic. RTMPS is not supported by any third-party proxy server. Static bypass works well to solve this issue. The problem stems from network policies that require all traffic to go through a proxy. The result is tunneling with commensurate high latency and drops. RTMPS must be allowed to stream around a proxy to avoid the overhead and latency of tunneling encapsulated within HTTPS. Attempts to cache the stream add no value.

Other options will depend on the capabilities of the third-party proxy servers in the affected client infrastructure. Blue Coat ProxySG is one of the popular proxy server options in our niche. In cases of latency invoked by tunneling RTMP encapsulated in HTTP on a network that employs Blue Coat ProxySG servers, sniff tests done by support representatives have indicated that when an affected client attempts to connect to an Adobe Connect meeting, those clients would establish both explicit HTTP connections based on PAC file settings in the system registry to the Blue Coat ProxySG pool through a hardware-based load balancing device (HLD) and transparent HTTP and SSL connections through Blue Coat ProxySG via WCCP GRE redirect to several Adobe Connect servers. The problem manifests with RTMPS when the clients attempt to establish an SSL connection directly to the destination host without going through PAC file proxy settings. Since a Blue Coat ProxySG is commonly configured to perform an SSL intercept on both explicit and transparent HTTPS traffic, upon examining the content after decrypting the SSL payload from the clients, the Blue Coat ProxySG will return an exception and close the connection because the request doesn’t contain an HTTP component and cannot be parsed for policy evaluation. As a workaround, other than using static bypass, it is possible to create a proxy service with the destination set to the Adobe Connect server IP range on port 443 and to set the proxy setting to TCP-Tunnel with Early Intercept enabled. This will allows Blue Coat ProxySG to intercept and tunnel the traffic without considering whether it is RTMPS or HTTPS.

Watch for a more comprehensive article on this topic forthcoming.