Posts in Category "Clustering"

Generating Server-side Logs to Troubleshoot On-premise Connect Deployments

In order to diagnose unexpected behavior within Adobe Connect, it may be necessary for the Adobe Connect Support team to examine server-side logs from an on-premise Connect deployment. The logs directory is located in the Connect (or Breeze – it is not uncommon for Connect upgrades to reside in legacy Breeze directories) directory:

logsdir.fw

Within the logs directory there are sub-directories containing various logs:

logsdir1.fw

The most commonly requested log by the support team, is the debug.log. It can be found in the logs>support directory. With the services running, the current debug log will appear without a date at the top of the debug.log file list. The default rollover is 12 hours generating AM and PM logs each day:

logsdir2.fw

In order to make the debug.log file more useful for purposes of diagnosis, you can enable verbose logging by adding entries to the custom.ini file located in the Connect or Breeze version sub-directory. Here you see it located in a 9.3.1 directory under the Breeze root installation/upgrade directory:

logscustomini3.fw

Before editing the custom.ini file, be sure to create a backup copy of it. Add the following lines in order to enable verbose logging:

HTTP_TRACE=true
DB_LOG_ALL_QUERIES=true

Note that for versions of Connect 9.2 and prior, use yes instead of true:

HTTP_TRACE=yes
DB_LOG_ALL_QUERIES=yes

Save the custom.ini file (be careful not to accidentally change the file type to .txt) and during a scheduled maintenance window, cycle the Connect and AMS/FMS services in order to load the changes and begin verbose logging (note this will bring Connect down while the services cycle):

logssvcs4.fw

There are occasions when it may be prudent to provide more than one log for a more complete diagnosis. To provide a full sample of the various Connect logs without sending a massive historical sample of log files, you may simply stop the Connect services (during scheduled downtime as this will bring down Connect) and rename the entire log directory to log.old. Then upon starting the services back up, recreate the issue being diagnosed and then stop the services.

This activity will generate a new small log directory isolating the issue under scrutiny that you just reproduced in Connect: Zip/compress this new abbreviated log directory with all its fresh abbreviated sub-directories and provide it to the the Adobe Connect Support team to help expedite more exhaustive server-side log analysis. This option is particularly helpful when examining a cluster as each server will have a set of logs. When providing cluster logs, always label each compressed log folder to easily identify the server from which it came.

Note that often when diagnosing unexpected behavior in Adobe Connect Meetings, it may also be prudent to enable client-side Connect addin verbose logging as well.  The relevant client-side logging tech-notes are here:

Enable logging | Meeting Add-in

Troubleshooting Verbose Meeting Addin Logging

New Adobe Connect Support Blog Subscription Option

Now you can stay on top of the new articles and posts by subscribing to the Adobe Connect Support Blog. Simply go to the Adobe Connect Support Blog home page and enter your email address and check off the categories about which you would like to be notified. Click “Subscribe me” and you will begin receiving  regular updates:

subscribe.fw

 

 

Behind the Curtain: Making Multiple Connect Meetings or Seminars Appear as One

On those occasions when a Meeting invitation may attract more participants than expected or planned for at the last minute so that you are unable to increase Seminar capacity in a timely manner, a skilled host can use two or more Connect Meeting rooms and project them to participants as though it were one room as an emergency workaround. Here is a basic outline of how to split a large meeting onto multiple servers. It is prudent to not just have more than one Meeting in these cases, but also to make sure each Meeting is hosted on a separate server in a cluster to add robustness to the meeting. Load-balancing is a wonderful thing and you should always use it to its fullest.

Assume an example of a three-server cluster/pool of Connect servers and that you want to split a Connect Meeting onto all three servers; a simple 3-server cluster is depicted here to use as an example:

C9SSLCluster3Simple

For a working example, let’s place a Connect Meeting room hosted on each server; to do this you will need three separate URLs: One URL for each 1/3rd  of your attendees. Getting the attendees distributed among the three rooms can be tricky. One effective technique is to either send out three different invitations, with each targeting 1/3rd of your audience and each offering a different URL, or just point everyone to a page with  all three URLs and request/instruct the participants to alphabetically arrange themselves in subsets of users by URL selection. That way it is not random; I have seen this technique work fine; here are sample meeting URLs based on our picture above:

http://connect.domain.com/splitmeeting1
http://connect.domain.com/splitmeeting2
http://connect.domain.com/splitmeeting3

To make certain the each meeting is hosted on a separate server (rather than all three on one as load-balancing could easily prescribe), it will require some effort to keep entering and leaving the room until your meeting lands on the server you want. Using multiple browsers may be helpful as well. Working on this well in advance of the meeting is prudent as there is a session timeout factor to consider. The load balancing algorithm will eventually get the sessions distributed but it may take some effort.

The way to tell which server you are on is simple: In any meeting room click Help and while holding down the shift key click About Adobe Connect. This will pop up an RTMP string that will identify the server that Meeting is hosted on and also which server a client is coming through as each client can be using multiple servers (just to add not only to the complexity, but also the overall robustness).

Here is what the RTMP strings might look like for each of the three servers in our simple example above ( I am inserting some URL parameters from a hosted meeting as I write this in order to create our hypothetical example RTMP strings – rtmp://arfms3.adobeconnect.com:1935/?rtmp://pcparapp07:8506/meetingas3app/89676385/630888204/)

rtmps:// connectmtg01.domain.com/?rtmp://connapp01:8506/meetingas1app/847483075/1086833045/
rtmps:// connectmtg02.domain.com/?rtmp://connapp02:8506/meetingas2app/847483076/1086833046/
rtmps:// connectmtg03.domain.com/?rtmp://connapp03:8506/meetingas3app/847483077/1086833047/

The first name in the string (connectmtg0#) is the built-in Connect Edge server and the second name (connapp0#)  is the Connect origin server  hosting the meeting (each Connect servers runs both AMS/FMS and Tomcat together). The second name is the important one for our technique of splitting the attendees onto separate meeting servers.

In the hypothetical RTMP string samples above, I have made these artificially neat and tidy, the truth is that the first part of the string can be any of the three for any meeting participant regardless of the application server hosting the meeting. For example, you could come in to connapp01 through connectmtg03 – any combination is possible. Load balancing is done at more than one level as Connect leverages both a hardware-based load-balancing device and also its own internal clustering capabilities; combinations for various clients (including the hosts and presenters) in our example cluster depicted  above might include:

rtmps:// connectmtg01.domain.com/?rtmp://connapp02:8506/meetingas2app/847483076/1086833046/
rtmps:// connectmtg02.domain.com/?rtmp://connapp02:8506/meetingas2app/847483076/1086833046/
rtmps:// connectmtg01.domain.com/?rtmp://connapp03:8506/meetingas3app/847483077/1086833047/
rtmps:// connectmtg03.domain.com/?rtmp://connapp03:8506/meetingas3app/847483077/1086833047/
rtmps:// connectmtg02.domain.com/?rtmp://connapp01:8506/meetingas1app/847483075/1086833045/
rtmps:// connectmtg03.domain.com/?rtmp://connapp01:8506/meetingas1app/847483075/1086833045/

The key to remember is that the second name is the one that matters; a distribution of participants approximating 1/3rd on each server is the goal targeting: connapp01, connapp02 and connapp03. After this is set-up, the pre-meeting preparation part is complete (this should be done at least one hour prior to the meeting).

Next comes the creative hosting venture during the split meeting: As the host, you will need all three meetings open in front of you to manage them as one. From the perspective of the participants, there is only one meeting (ignore the host behind the curtain). Be sure to hide the Attendee List Pod in the Presenter-only area as it will only present those participants in that specific Connect Meeting thereby allowing a peek behind the curtain or misrepresenting the size of the entire three meeting combination.

And here is where the techniques are very much up to you:

  • Splitting video among the three rooms is possible using a third-party option, one we have used successfully is: Splitcam.com.
  • For audio, if using integrated audio, be sure to use the same integrated telephony number for all three rooms.
  • If using VoIP, then allow one speaker only at a time to send audio via VoIP.

Some ways in which you can limit the amount of data being processed in your room and to improve the overall performance of these sessions are:

  • Optimize room bandwidth. In a Connect Meeting, at the top of the screen click on MEETING > Preferences. Under the preferences menu you are able to adjust screen sharing, video and VoIP quality setting separately.
  • Turn off cameras whenever they are not in use.
  • When in use, multiple cameras should probably be set to SLOW images (depending on how many and other variables).
  • Turn off VoIP if not talking.
  • Participants should directly connect to the fastest internet connection available and be on a dedicated DSL connection, at a minimum.
  • No clients or hosts on wireless – allow no exceptions.
  • Shut down Email, instant messaging, and any programs NOT being used for the presentation.
  • Shut down any VPNs as a VPN will potentially destroy the possibility for success.

When large Connect Meetings or Seminars become commonplace in your enterprise, this cumbersome workaround quickly becomes impractical and you should increase your Seminar or Webinar licensed capacity as needed to avoid this complexity and manual work. With that said however, this technique will work in a bind and will provide a robust Connect Meeting experience for a very large audience even if it challenges a seasoned Connect Meeting host.

Changing the License Serial Key in Connect

This article applies to on-premise and managed ISP Connect users. It does not apply to multi-tenancy hosted or ACMS.

On rare occasions it may be necessary to change the serial key in Adobe Connect. Here are the steps:

  1. Navigate to: \Connect_installation_directory\appserv\conf\config.ini and change the value of  SERIAL_KEY=  to reflect the new serial number
  2. In \Connect_installation_directory\custom.ini,  if there’s a serial key value listed (SERIAL_KEY=), replace it there as well.
  3. Using MSSQL Studio Express (or your choice of SQL editing options), view the serial key currently being used by Connect by running this command: SELECT * from pps_accounts WHERE name=’Enterprise Account’
  4. To get Connect to accept the new license you must change the serial key that is currently in the database by running this SQL command: UPDATE pps_accounts SET serial_key = ‘NEW_SERIAL_NUMBER’ WHERE serial_key = ‘OLD_SERIAL_NUMBER’
  5. Restart the services: Application Server (Connect) and the Meeting Server (AMS or FMS depending on the version of Connect) services.fw
  6. Open the Administration Console (port 8510 locally on any Connect server)

connconfig.fw

7. Go to License Settings and upload the new license file.

connconfiglic.fw

8. Restart the AppServer (Connect) and the Meeting server (AMS or FMS depending on version) again and the  new license file will be applied

services.fw

Troubleshooting: If there are any problems, do the following to troubleshoot:

  • Shut down the Connect and AMS or FMS Services
  • Open and verify \Connect_installation_directory\appserv\conf\config.ini and update the entry for SERIAL_KEY
  • Open and verify  \Connect_installation_directory\custom.ini and update  the entry for SERIAL_KEY
  • Open SQL Server and choose the Connect database and run the following script (replacing the text as appropriate):

“Input New Serial Key Here” with the New Serial Key but leaving the quotes.
DECLARE @NEW_SERIAL VARCHAR(32)
SET @NEW_SERIAL=’Input New Serial Key Here’

UPDATE PPS_CONFIG
SET VALUE = @NEW_SERIAL
WHERE SECTION=’cps’ AND NAME=’serial_key’

UPDATE PPS_ACCOUNTS
SET SERIAL_KEY = @NEW_SERIAL
WHERE ACCOUNT_ID=7

UPDATE PPS_ENUM_DATA_HOSTS
SET LICENSE = @NEW_SERIAL
WHERE HOST_ID > 0

db.fw_

  • Start the Connect and FMS services

Problems will ensue when the license is reducing the allowed usage of Connect (if you are downsizing) and you leave an overage in place. For example, if you have 100 meeting hosts assigned, and you are changing to a license that only allows 50 named meeting hosts then when you  apply the license you will get an error unless you have reduced the number to accommodate the new licensed restriction.

Connect 9.5 Edge Server Installation Instructions

Connect 9.5 server installation instructions:

  1. Create a folder <Installation_Directory>/950/edgeserver
  2. Download the Edge 9.5 (based on AMS 5) installer
  3. Run the self-extracting .exe file downloaded in step#2 to <Installation_Directory>/950/edgeserver
  4. Refer the following articles for deployment options:
    1. http://www.connectusers.com/tutorials/2011/06/edge_server_deploy/
    2. http://www.connectusers.com/tutorials/2011/06/edge_server_deploy2/
  5. Run <Installation_Directory>/950/edgeserver/win32/vcredist_x64.exe
  6. Run the following commands as administrator:
    1. cd <Installation_Directory>/950/edgeserver/win32AMSAdmin.exe -install
    2. AMSMaster.exe -install
    3. sc start amsadmin
    4. sc start ams
  7. Confirm that services “Adobe Media Administration Server” and “Adobe Media Server (AMS)” are running
  8. If services need to run using specific user credentials, then be sure to set the credentials in service properties and restart the services

Troubleshooting Verbose Meeting Addin Logging

On occasion it can be difficult to get verbose addin logging to work. The tech-note describing how to set it up is here: Enable logging | Meeting Add-in

The tech-note correctly describes where to place the customized mms.cfg file for use with both 64 bit and 32 bit Windows clients as well as for the Mac OS.

If after following the instructions in the tech-note, you still do not see any verbose addin logs, one possible cause is that there may be an additional mms.cfg file in an alternate location on the client that is blocking the log creation process. To remedy this, add the customized debug mms.cfg to the following locations after renaming any existing mms.cfg files (to allow them to be restored after verbose logging or debugging is complete):

Here are the locations (more than in the tech-note):

  • Windows (32 bit) :

In: C:\Windows\System32\Macromed\Flash\mms.cfg
or C:\Windows\System32\mms.cfg

  • Windows 7 (64 bit):

In: c:\Windows\SysWOW64\Macromed\Flash\mms.cfg
or c:\Windows\SysWOW64\mms.cfg

After placing the mms.cfg in both folders, be sure to close all addin browsers and then to open the addin only in the one Meeting that you wish to troubleshoot.

On-premise Connect Installation Hangs Connecting to the Database

Symptoms: Installing with clean images on servers, the Connect Installation with the appropriate local Administrator permissions seemed to be successful but upon clicking “Done” its hangs indefinitely. Restarting the services does not help and the Connect Configuration Console on the local Connect server will not come up. Rebooting the VM will not bring Connect up. In the error.log, it reads:

“Start up error: java.lang.Exception: invalid backup folder: \\connectsharedstorage\connect.” START_UP    START_UP_ERROR….

Note: replace connectsharedstorage\connect with your UNC path to shared storage.

Solution: This error indicates that shared storage is expected by the database but is not configured on the Connect server. This may inadvertently be overlooked during an upgrade instance when a new server (perhaps with a new OS) replaces an older server. The fresh Connect installation, upon pointing to an existing upgraded database that has possibly been updated by script or maybe by the older server image, is expecting shared storage to be in place, but it is not yet configured on the new Connect server. To get past this, edit the Shared Storage entry in the PPS_Config table of the Connect Database to “NULL” and restart the services.

Configuring Secure SQL with Connect

It may be prudent to secure the connection between the Adobe Connect application servers and the SQL database.

Begin with the SQL server and then move onto the Connect server(s); if your SQL server is shared then begin with a change request to the DBA who has charge over the shared SQL environment. If your SQL database is already secure, you may skip Part I.

Part I. Securing the MS SQL Database Server:

First open the Certificates snap-in:

1. Open the MMC console, click Start, and then click Run; In the Run dialog box type:  MMC
2. From the  File menu, click Add/Remove Snap-in….
3. Click Add, and then click Certificates. Click Add again.
4. You are prompted to open the snap-in for the current user account, the service account or for the computer account. Select the Computer Account.
5. Select Local Computer, and then click Finish.
6. Click Close in the Add Standalone Snap-in dialog box.
7. Click OK in the Add/Remove Snap-in dialog box. Your installed certificates are located in the Certificates folder in the Personal container.

Use the MMC snap-in to install the certificate on the server:

  1. Click to select the Personal folder in the left-hand pane.
  2. Right-click in the right-hand pane, point to All Tasks, and then click Request New Certificate….
  3. The Certificate Request Wizard dialog box opens. Click Next. Select Certificate type is “computer”.
  4. In the Friendly Name text box you can type a friendly name for the certificate or leave the text box blank, and then complete the wizard. After the wizard finishes, you will see the certificate in the folder with the fully qualified computer domain name.

You are done now with installation of certificate on the SQL server, next you will need to export the certificate so that the same can be imported in the Connect application server.

  1. Open MMC, and then locate your certificate in the Personal folder.
  2. Right-click the certificate name, and then click Open.
  3. Review the Certification Path tab. Note the top most item.
  4. Navigate to the Trusted Root Certification Authorities folder, and then locate the Certificate Authority noted in step 3..
  5. Right-click CA, point to All Tasks, and then click Export.
  6. Select all the defaults, and then save the exported file to a location where the Connect application server can gain access to it.

Configure SSL encryption in the MS SQL instance:

1. On the SQL server start menu open Microsoft SQL Server>Configuration Tools> SQL Server Configuration Manager:

SQLsecure1.fw

2. Expand SQL Server Network Configuration, then right-click Protocols for MSSQLSERVER, and choose Properties. Select the Flags tab and change the Force Encryption setting to Yes.

sqlserverencryptionstep2

3. Under the Certificate tab, choose the certificate created earlier from the drop down list:

SQLsecure4

The database is now ready for secure connection with the Connect application server.

Part II. Configure the Connect application server to support a secure SQL connection:

Importing the certificate onto the Connect application server

  1. Copy the certificate from MS SQL Database server to the Connect application server(s) or to an accessible share.
  2. Navigate the Connect application sever by using the MMC snap-in, and then browse to the Trusted Root Certification Authorities folder.
  3. Right-click the Trusted Root Certification Authorities folder, point to All Tasks, and then click Import.
  4. Browse, and then select the certificate (.cer file) that you copied in step 1. Select the defaults to complete the remaining part of the wizard.

Create a Trust Store

1.  Be sure to have java installed on your Connect application server; at the command prompt, navigate to the bin directory of your JRE, and execute the following command:

keytool -import -file  <certificate file path> -alias firstCA -keystore <any name for trust store>
Note: This step will queue for a password, create and record a password for future reference.

2. In the ConnectProSvc.conf in the appserv\conf directory, add the following entries in the list of JAVA arguments:

wrapper.java.additional.28=-Djavax.net.ssl.trustStore= <path of Trust Store file created in step 1>
wrapper.java.additional.29=-Djavax.net.ssl.trustStorePassword=<password you created in step 1>

Configure the secure connection in Connect:

1. In custom.ini file under the root Connect installation directory, add the following entries:

DB_ENCRYPTION_METHOD=SSL
DB_VALIDATE_SERVER_CERTIFICATE=true

2. Cycle the services or reboot the server:

Adobe Connect Service
Flash Media Service

Note: For secure LDAP or LDAPS with Connect and for additional granularity around the paths and keystore see the following tech-note: Configure Connect Directory Services to use LDAPS

On-premise Adobe Connect Servers and Java

The question occasionally comes up: May I freely update Java on my Adobe Connect on-premise server?

And actually this question should be split into two questions:

  • What version of Java may I use?
  • What update of that version may I use?

It is important to keep these two questions separated because going from 1.6.0_37 to 1.6.0_45 is different from (more trivial than) moving from Java 6 to Java 8 (whereby compatibility issues could result).

With reference to the shipping version with Connect, our standard is Java 7, and has been since 9.1.1. Since we make every effort to keep Adobe Connect up to date with its surrounding infrastructure,we will evaluate a move to Java 8 going forward, as eventually public Java 7 updates will come to an end.

With reference to the updaters, Oracle releases quarterly Critical Patch Updates (see http://www.oracle.com/technetwork/topics/security/alerts-086861.html), and we have been striving to keep up with these (although our release cycle does often mean that we are one or two quarters behind so as to allow for time to fully test). The version being packaged with 9.4 is 1.7.0_71, the Critical Patch Update from Oct 2014.

While we  don’t believe there is any particular compatibility issue with moving from JRE 6 to JRE 7, nevertheless we do not recommend that customers update the JRE separately from Connect itself. There are multiple reasons for this:

  • We have uncovered JRE bugs in the past during our performance/longevity tests.
  • We also moved from 32-bit JRE to 64-bit JRE and this necessitated sizing changes (heap size etc.).

Heap size is an important variable that warrants performance testing to ensure that the sizing is adequate for the target JRE version. All this is due diligence is done as part of our packaged Connect builds; by updating outside of our quality assurance and performance testing cycles, you add unnecessary risk. It is best to take full advantage the battery of testing accomplished by the Adobe Connect engineering team; by upgrading the JRE separately, you will create an infrastructure with variables that have not been fully tested and thereby assume commensurate risk.

Make Certain that Content is Replicated Across All Servers in a Connect Cluster

Occasionally a specific piece of content may be intermittently available in a cluster. It could be Presenter or Captivate published on-demand content or even content within a Meeting room. Sometimes in these cases, the content published on one server is not replicated to all servers in the cluster. There are a few quick things to check:

First: Note that with Adobe Connect 9, the installer includes a cluster option. If you begin with a single server installation and expand later to a clustered environment by adding a server or servers, you will need to manually make the following change in the /appserv/conf/server.xml file in order to enable communication over port 8507 among clustered servers. It is prudent to double check this in the server.xml file after installing even if the cluster option was selected during installation:

<Executor name=”clusterThreadPool”
namePrefix=”cluster-8507-” maxThreads=”150″
minSpareThreads=”5″/>

<!– Define a non-SSL HTTP/1.1 Connector on port 8507 –>
<!– Used for HTTP access for intra-Cluster communications. –>
<!– Equivalent to JRun CLUSTER_PORT –>
<!– Uncomment for clustered deployments
<Connector port=”8507″ protocol=”HTTP/1.1″
executor=”clusterThreadPool”
enableLookups=”false”
acceptCount=”100″
connectionTimeout=”20000″
URIEncoding=”utf-8″/>

Second: Test the 8507 port communications on each server: From a command prompt on each server, type netstat –an|find “8507” and check to be sure that 8507 is active and listening on each.

netstat -an|find “8507”

netstat.fw

Use telnet to test connectivity on  8507 between Connect servers. Use telnet to check both IP and machine-name as well.

telnet server-machine-name 8507

telnet 8507.fw

Note: The machine name appears to the left of the FQDN under the Connect Servers Setting on port 8510 locally on any server in the cluster; here I have artificially designated them as server1 and server2.

serversettings

Be sure to check telnet connectivity from and to every server in the cluster:

telnet 8507.fw

If the IP works with telnet and the machine-name does not work, it may be necessary to add entries in DNS or add hosts files to each server:

etc-host.fw

Check the software-based firewall on the server to see if it is potentially blocking replication traffic:

netsh firewall show config

firewallsftwr.fw

win-firewall-svc.fw

Note: Connect does not support dual stack ipv6 and ipv4 on the same server.

Note: If problems are noticed in the Meeting rooms, check port 8506; it is used for Meeting communication among the servers.

Third: Examine the Connect logs: Look first in the debug.log under the \logs\support directory and search on the string: cluster-  If replication is taking place, you will see this repeating cluster- entry logging the replication activity. Absence of these log entries will indicate that replication is not working:

[10-1 12:00:00,009] cluster-8507-630 (INFO) CLUSTER Sent file: \7\xxx-xxxx\fcs-meeting\public\all\224_XXX_4.fso 9978 bytes 12 ms 6371 kbps to: server1

cluster-debug.fw

Check for any error messages in these replication log entries. Search also for the word lucene. If you see a preponderance of lucene lock errors, contact Adobe Enterprise Support: entrsupp@adobe.com and provide a log snippet to expedite diagnosis.

Also check the error.log files for the entry  CLUSTER_CON_BROKEN

2014-10-02 15:28:48 “Server server1 unable to reach server2 on port 8507 to perform cluster operations.” CLUSTER  CLUSTER_CON_BROKEN

Fourth: Check the timing of active anti-virus scanning of the content directories \content\7\ on each server; compare the directory sizes on each server to see is there if a significant size delta. Antivirus software can impede replication in manner that is not uniform across servers; active scanning of the content directory during replication may lock the content files. Active scanning after hours or during a window when publishing is unlikely is prudent.

Fifth: Check the updater page. Make sure you are on the latest patches servers-side. http://helpx.adobe.com/adobe-connect/kb/connect-90-patches.html Keep in mind that 9.2 is a full installer and not a patch. For full installers, use LWShttps://licensing.adobe.com

These steps will solve most replication problems that you encounter. If problems persist, contact our  Enterprise Support Team.