Posts in Category "Clustering"

Connect 9.5 Bandwidth Utilization Estimates Illustrated

Introduction:

Adobe Connect bandwidth utilization will vary based on use case. Variables such as Meeting size, Connect features employed, disposition of clients and type of Connect server hosting, all have an effect on the network bandwidth utilized and as well as on where the bandwidth on a network will be most affected.

This article focuses on bandwidth and not on latency: Latency and bandwidth are different topics that are often confused or inappropriately lumped together. Although in some cases there may be a correlation between bandwidth and latency as when exceeding available bandwidth will drive up latency, nevertheless, the relationship between them is often inappropriately overstated. Bandwidth refers to the size of the network pipe while latency refers to the speed over the network or the time it takes for the data to travel. Connect clients with excess bandwidth may still experience latency with Connect for reasons unrelated to bandwidth such as network distance, transmission errors, resource performance (server, DB or client), maintenance schedules and imposed constraints such as QoS or blocking the RTMPS protocol and thereby forcing inefficient tunneling of RTMPS encapsulated in HTTPS.

Resources:

For information on latency see the following articles:

Tunneling with RTMP encapsulated in HTTP (RTMPT) should be avoided as it causes latency

Connect Meeting RTMP VS/VIPs on Load-Balancers

Adobe Connect Database Performance and Monitoring

 Connect on VMWare – some deployment tips

Below are some references on the topic of Connect bandwidth; these are often referred to as a general guide; the examples sited later in this article however are from baseline testing described herein:

Estimating Bandwidth Consumption in Connect Meetings

Best Practices for Adobe Connect

Adobe Connect best practices for large events and seminars

VoIP Bandwidth and Microphones

The Connect Cluster:

The context of the Connect bandwidth utilization is important and should be included among pre-deployment architectural considerations when rolling out Connect. There are basically two primary Adobe Connect deployment models for consideration: In the cloud and on-premise. The disposition of the clients attending the meeting has an effect where the on the network bandwidth will be utilized for any Connect Meetings or Webinars. To support these two broad deployment paradigms of cloud or local on-premise, there are three basic Adobe Connect licensing models.

  1. Adobe Connect Multi-tenancy Hosted
  2. Adobe Connect Managed Services (ACMS) single domain hosting
  3. Adobe Connect On-Premise licensed single domain

Note: An Adobe Connect on-premise license need not only be deployed within the customer’s LAN infrastructure. Adobe Connect Partners offer Managed ISP hosting options and an on-premise license deployment can be cloud-based as well.

Cloud or LAN:

For the purposes of considering the relationship of bandwidth to architecture, an Adobe Connect cloud-based deployment and an on-premise deployment within a customer’s LAN present different concerns with reference to the disposition of bandwidth rather than with the specific license type.

Primary differences with reference to bandwidth when comparing cloud-based Connect server clusters with on-premise based Connect server architecture is with the handling of external clients and the direction of the network traffic. There will be less internal network traffic on the LAN when external users (with possible lower-impact exceptions such an SSO server) remain external. Note that VPN users in remote offices are considered internal users.

See the following diagrams comparing a 2-server cloud-based cluster with a 2-server on-premise cluster hosted on a customer LAN. Both assume the need for internal and external users to participate in Meetings.

Cloud-based:

With a cloud deployment, the external users have little or  no impact on the internal LAN of the Connect account owner. The biggest bandwidth hit is at the cloud-based hosting facility. The primary concerns for the Connect owner’s IT will be routing RTMPS around third-party proxy servers and making sure client profiles and other internal restriction, limitations and constraints do not prevent or enervate Connect usage:

two-server-cloud

LAN-based:

With a LAN-based on premise deployment, internal Connect users will normally have great performance on the average corporate back-plane by avoiding any mitigating and often unpredictable WAN variables. External users may be hosted on a reverse proxy Connect Edge server in the local DMZ to meet security requirements. All external traffic is terminated in the DMZ where the remote Edge server does all the heavy lifting. The Edge connections to the internal cluster consolidate the external connections and reduce the overall internal impact of external users by limiting them to the external firewall and DMZ:

EdgeDMZAbbrev

Connect Meeting Features and Bandwidth:

Screen-sharing and the use of multiple live web cameras are the two most bandwidth intensive activities in any Connect Meeting or Webinar. The latter requires that a Meeting Host or Presenter project the client screen up to the Connect servers and then back down to each user in the Meeting. The more users with cameras, the more upload and download activity.

The features employed in any Connect Meeting determine the amount of bandwidth utilized.  Depending on the Connect use case, you could use between 5 to 10 Mbps per 100 participants. These general rules apply:

    • The typical average is 50 kbps per user
    • It depends on what Connect Meeting users are doing
    • For example, on the high end one client measured regularly uses 20 Mbps for 150 users in a Connect Meeting room.
    • More typically there is a max of 100 Mbps at the firewall (outgoing, incoming is never that high)
    • With just over 1000 users a safe estimate is < 100kbps per user totaling 12 MB
    • 5 Mbps per 100 is the low average (or 6MB per 1000 users using best practices for large Webinars).
    • Things like video and screen sharing tend to use more bandwidth
    • A one-to-many broadcast of an uploaded PowerPoint deck uses less bandwidth
      • PPTX should always be uploaded to the Meeting room
      • Do not present a PPTX using screen-sharing as it wastes bandwidth

Bandwidth Utilization Use Case Baseline Tests:

The following tests corroborate these guidelines listed above and offer a model to test bandwidth utilization for any Meeting or Webinar use case; it is prudent to do these type of tests with your specific use Webinar case as part of your rehearsals to get accurate bandwidth utilization baselines:

First bandwidth test: I uploaded a 15.5 MB PPTX to a Connect Meeting room and advanced thru all the slides while monitoring the download stream to each client; the average for each client was 50 kilobits; see below:

4clients.fw

Second bandwidth test: I then tested with a camera pod set to the highest quality and wide screen option: Meeting>Preferences>Video; see my host camera settings below:

videopref.fw

The average download stream at each client when I had the camera pod sized small in the upper corner of the meeting while flipping slides was under 200 kbps (kilobits); The host screen capture below corroborates; my host client in the Connect addin is pushing 121 kilobits up to the server; see below:

cambndwdthtrun

The average when I expanded the camera pod and moved around create some activity on screen (waving etc.) was 300kbps download streaming to each client; see below my upload from host client to server is 381 kbps; see below (wow- my hair is getting thin):

cambndwdthlrgtrun

Third bandwidth test: I then tested screen-sharing a slide deck from my desktop to simulate application sharing live demonstrations and found a significant variance based on the screen resolution of the host sharing the screen:

  • Host screen-sharing at 1920×1080 resulted in an average  download to each participant of 700 kbps
  • Host screen-sharing at 1288×768 resulted in an average download to each participant of 200 kbps
  • The participant experience was very similar. I see no reason to waste the bandwidth on the higher host screen-resolution: Lower screen resolution resulting in very economical bandwidth utilization while sharing the screen
  • See the bandwidth indicator while I share my screen flipping through multiple jpegs at 1288×768. I am pushing up 338 kbps to the Connect server while actively changing jpeg images on a second monitor that I am sharing.

hostss1280.fw

Fourth bandwidth test: Video – I created an mp4 using the following settings: Appropriate quality settings are a major variable determining the amount of bandwidth used and the playback experience of mp4 video in Connect:

  • I used: H.264, 720×404, Frame Rate 30, bitrate CBR, target rate 1Mbps, Key Frame Dist 90.
  • Best practice encoding for video in Connect imperative: Rehearsal and testing of any Video used is prudent prior to any Meeting or Webinar. It is easy to test playback experience and bandwidth consumption per client participant. Note also here (in the highlighted bandwidth indicator) how my remote office client experiences some latency albeit manageable  (595kbps down) that not affect the video playback experience – no choppiness at all watching the pups wrestle (the Jack-Russell ends up winning although this capture shows him as the underdog):

video

Bandwidth Utilization Use Cases:

Using the baselines from these tests, here is a list of potential use cases from a recent customer inquiry with the predicted overall account-wide bandwidth usage estimates running Connect 9.5.

Case 1: A 1500 user seminar room with a PowerPoint and a desktop share:

Estimate: 1500 users x 50 kilobits each for the PPTX use case =75000 kilobits or 9MB. Add to that a desktop share with conservative screen-resolution (1280×768) 1500×200=300000 kilobits or 37MB. Add them together (75000+300000) for an average of 375000 kilobits or 47MB. Realistically it will be less at any given time because it is rare to simultaneously do both activites at once in a meeting. It is more common to share either flip PPTX slides or share an application thereby lowering the actual bandwidth utilized at any given time.

 Note: There is an important caveat with reference to an initial spike caused by the PPTX SWF download of 1000 kbps per client participant for a few seconds each. This affects every use case with SWF content is a share pod. Assuming 1/3 at a time as participants enter the layout with the PPTX on stage in a share pod even if downsized in a corner or hidden behind another pod, and assuming that there will not be any screen-sharing initially during the downloading: 1500/3 = 500x1000kbps for a few seconds each = 50000 kilobits or 62.5MB. A worst case spike scenario of all entering at once due to host error or a late upload due to last minute editing (this should be avoided if possible): 1500×1000=1500000 kilobits or187MB Spike for up to 10 seconds depending on latency effects on the initial SWF download. This spike may be missed in testing if you rely on the Meeting room bandwidth indicator as it may happen too fast to register; it is better measured using Wireshark on a client in the Meeting.

Case 2: A 1500 user seminar room with a PowerPoint and a 10 minute MP4 Video played:

Estimate: 1500 users x 50 kilobits each for the PPTX=75000 kilobits or 9MB. Assuming a medium quality MP4 (as discussed above) uploaded to the meeting room and meted out at an average 600 kbps per participant with intermittent spikes to 1200 kbps: 1500×1000=1500000 kilobits or187MB + 9 = 196MB while the video is being played with the PPTX still onstage visible in another pod. Note: Until the Video is played, this room is at 9MB just meting out PPTX SWFs after the initial SWF downloads are complete.

 Note: PPTX download spike variables as described in case # 1 above applies.

Case 3:  A 1500 user seminar room with a PowerPoint and two, 3 minute videos played:

Estimate: Same as case 2 above assuming that the videos are played one at a time.

Case 4: Two 1500 user seminars with just PPTs for content (uploaded to the Meeting room and not screen-sharing) and jpeg pictures for Presenter head-shots in lieu of a camera:

Estimate: 1500 users x 50 kilobits for the PPTX=75000 kilobits or 9MB x 2 seminar rooms = 18MB

 Note: Assuming the seminars will not start both at the same time and will nevertheless overlap, the brief PPTX download spike variables as described in case # 1 above applies albeit multiplied by two: 126 MB for 1/3rd and worst case of both seminars doing a last minute deck upload at the start of the Meeting: 274MB until initial SWF downloads complete. Staggering slightly the start of each seminar will mitigate the overall spike as will allowing users to trickle into a prepared room with assets on stage.

Case 5:  Two 1500 user seminars with just PPTs for content (uploaded to the Meeting room and not screen-sharing) and PPTs for headshots shared, PLUS 100 separate Connect meetings of 20 users each, with 10 of those meeting rooms sharing one web camera:

Estimate: 1500 users x 50 kilobits each for the PPTX=75000 kilobits or 9MB x 2 = 18MB, plus 90 meeting rooms with 20 users each doing the same activity, 1800×50 kilobits=90000 kilobits or 11MB, plus the 10 meetings with cameras added 10 x 20 users each x 200 kilobits=40000 kbps or 5MB; also add the upload for each camera 10 cameras x 200 kilobits =2000 kbps or .25MB. Total =.28000 kilobits or 35MB.

Note: Assuming the various meetings will be starting at different times the spike estimates for #4 above apply.

Case 6:  Two 1500 user seminars with just PPTs for content and PPTs for headshots shared, PLUS 100 separate Connect meetings of 20 users each, with 10 of those meeting rooms sharing one web camera, PLUS one 500 user seminar with a PowerPoint shared(uploaded to the Meeting room and not screen-sharing).

Estimate: Add 500 users x 50 kilobits each = 25000 kilobits or 4MB. Add this to use case 5 depicted above: 35MB+4=39MB.

Note: Assuming the various meetings will be starting at different times the spike estimates for use cases #4 and #5 above apply here.

Case 7: Two 1500 user seminars with just PPTs for content and PPTs for headshots shared,  PLUS 50 separate Connect meetings of 20 users each with 10 of those meeting rooms sharing one web camera, and one 500 seminar with just a PowerPoint shared (uploaded to Meeting room and not screen-sharing):

Estimate: 1500 users x 50 kilobits each  for the PPTX=75000 kilobits or 9MB x 2 = 18MB, plus 40 meeting without cameras x 20 users each=800x50kbps= 40000kbps or 5MB, add the 10 meetings with cameras 10 meetings x 20 users x 200 kilobits=40000 kbps or 5MB; also add the upload for each camera 10×200=2000 kbps or .25MB. Total= 232000 kilobits or 29MB.

 Note: Assuming the various meetings will be starting at different times the spike estimates for use case #4, #5 and #6 above apply.

Conclusion:

It is easy to avoid both overestimating and underestimating the bandwidth needed account-wide for Connect deployments on-premise or in the cloud. For large Webinars, a quick rehearsal with a small sample audience of external and internal users as appropriate is prudent. Best practices in large Meetings make a substantial difference in the amount of bandwidth utilized: Proper camera use, screen-sharing and video-encoding are all very important considerations.  Although it may at this point seem like a mantra, a basic best practice that so many Connect users miss is to upload a PowerPoint to the Meeting room share pod rather than use screen-sharing to run through a slide-deck. Maybe some Presenters are just overly possessive and have an innate refusal to let their work upload – whatever the reason, I will surely be in a Meeting today and some Presenter will share a PowerPoint from their desktop via screen-sharing and I will need to avert my eyes to maintain my light grasp on sanity. Used properly, Adobe Connect is the best means to deliver rich collaboration experiences while minimizing the impact on bandwidth utilization.

Adobe Connect 9.5.3 On-Premise Servers on Windows 2003

Note: This article applies only to Connect on-premise customers running Connect on Windows 2003 servers.

For on-premise Adobe Connect customers running Windows 2003, it is vital to upgrade the servers to at least 2008 (or higher) prior to upgrading Connect to 9.5.3. The Connect command-line converter will not run on Windows 2003 and will fail with the following error in the Connect debug logs:

CreateProcess error=193, %1 is not a valid Win32 application

The release notes for Connect 9.5.3 are here: https://helpx.adobe.com/adobe-connect/release-note/adobe-connect-9-5-3-release-notes.html

The 9.5.3 updater is here: https://helpx.adobe.com/adobe-connect/kb/connect-90-patches.html

Connect Meeting RTMP VS/VIPs on Load-Balancers

This article applies to on-premise Adobe Connect servers running behind hardware-based load-balancing devices or SSL accelerators.

A common cause of performance problems in Adobe Connect Meetings stems from the improper configuration of the Virtual Server (VS) Virtual IP Address (VIP) handling Real Time Messaging Protocol (RTMP) traffic in on-premise Connect deployments.

An Adobe Connect Meeting Server is at least two servers in one (possibly more if AEM/Events and UV telephony are incorporated); it is at least always a Tomcat-based HTTP application server and an Adobe Media Server (AMS) using RTMP. The two servers are fully integrated to work together in tandem to support Adobe Connect Meetings.

The most popular load-balancing and SSL acceleration  option in the Adobe Connect on-premise enterprise is the F5 BIG-IP Local Traffic Manager (LTM). This tech-note will illustrate the proper configuration of an RTMP VIP supporting Adobe Connect Meeting on an F5 LTM. The concepts apply to any load-balancing device and SSL accelerator.

The first thing to note is that the general configuration of a Connect server or cluster running behind an SSL accelerator or load-balancing device always requires more then one VIP. There are no exceptions to this rule and any attempts at shortcuts will result in delayed deployments and support cases. Attempts to place all traffic on a single VS/VIP are as common as they are incapacitating. General Connect cluster architecture tech-notes are here:

Adobe® Connect™ server pools/clusters and hardware-based load-balancing devices with SSL acceleration

Adobe Connect Servers and Hardware-based Load-balancing Devices

A simple diagram of an Adobe Connect server behind an F5 LTM follows; see the two VS/VIPs and Fully Qualified Domain Names (FQDNs) for each on the LTM:

C9SSL

Below we add a server to show a basic Connect cluster VIP configuration; see how each Connect Meeting server has its own VS/VIP while one VS/VIP servers both HTTPS application servers.

C9SSLa

Note: Neither of these basic diagrams depicts advanced configurations such as the integration of the Adobe Experience Manager (AEM) Events module. This article focuses on the performance of the Adobe Connect Meeting RTMP VIP in its basic context.

There is usually not an option for RTMP in the VIP profile of a hardware-based load-balancing device. A basic TCP profile is the correct choice. Here it is depicted on an F5 BIP-IP LTM:

f5.fw

With detail:

f5a.fw

f5b.fw

f5c.fw

Note that the symptom for an improperly configured VS/VIP is either the inability to launch a Connect Meeting or excessive latency in the Meeting due to RTMP tunneling (RTMPT) encapsulated within HTTP when the RTMP VIP is blocked or inoperable.

The presence of a capitol “T” in the latency indicator of an Adobe Connect Meeting indicates tunneling as depicted in this tech-note:

Tunneling with RTMP encapsulated in HTTP (RTMPT) should be avoided as it causes latency

Further diagnosis is usually warranted by using the Connect Meeting Addin in logging mode as depicted here:

Enable Logging in the Meeting Addin

Also here:

Troubleshooting Verbose Meeting Addin Logging

When the RTMP VS/VIP profile is improperly configured, the Connect Meeting addin verbose log will show it clearly, particularly when it is compared with the server-side debug log.

Example snippet from a Connect Meeting addin verbose log:

18:51:55    16844    PLAYER_TRACE    SSL connection closed.
18:51:55    16844    PLAYER_TRACE    SSL DoSSLHandshake WaitHandshake not in ssl_active state. (State is 0.) Failing.
18:51:55    16844    PLAYER_TRACE    SSL DoSSLHandshake WaitForSocket not in ssl_active state, failing.
18:51:55    16844    PLAYER_TRACE    SSL Receive socket read error 0x0.
18:51:55    16844    ACTION_TRACE    5/10/2016 14:51:55.101 [DEBUG] breezeLive.main.FCSConnector [attempt 1 of 60] Trying fallback tunneling connection rtmps://onlinemeeting.connectexample.com:443/?rtmp://localhost:8506/meetingas3app/7/1234567/
18:51:55    17179    PLAYER_TRACE    NetConnectionIO::DoConnect rtmps protocol, HTTP(S) tunneling, tunnel open succeeded.

The corresponding snippet in the server debug log as well as the application logs will read: RTMPT and often reconnect=true:

               Line 23456: 2016-01-17  14:25:06              32260    (s)2641173          Asc-Room               IA_CONNECT      [dID:32, ticket:123456789xyz, phase:, uID:, name:]             New client connecting:  { ip=127.0.0.1, protocol=rtmpt, player=MAC 11,9,971,247, savedConnectionSpeed=undefined, reconnect=true }                        –

[11-05 15:08:05] FCSj_Worker:18 (INFO) params: {bytesdown=0, protocol=rtmpt, ticket=123456789xyz, status=C, reconnect=true, nickname=John Doe, action=register-client, role=v, bytesup=0, session-timeout=12}

Correct configuration of the RTMP VS/VIP is extremely important; a Connect Meeting VS/VIP must have a dedicated FQDN.  It must have its own SSL certificate if SSL is accelerated through the load-balancing device and the VS/VIP must not have an HTTP profile; a TCP profile is needed.

For some additional information about troubleshooting Connect architecture with reference to hardware-based load-balancing devises and SSL accelerators, see the following tech-notes:

The Adobe Connect Deployment Guide on the F5 Website needs Updating

Configuring application-level health monitors for Connect on BIG-IP Local Traffic Manager

9.5 Connect Edge Proxy Server Full Installer

The new 9.5 Connect Edge Proxy Server full installation procedure follows.

Note: This article applies to on-premise Connect customers who have purchased Edge Proxy servers and must install or upgrade to Connect version 9.5. The only possible exception to on-premise exclusivity may be for those who are hosted by a managed ISP that supports external Enterprise Proxy Edge Servers (this latter model is uncommon).

Step 1: Download the Connect Edge Server installer from the URL location provided by the Adobe Connect Support or Customer Service team;  extract and install it with local administrative permissions.

edgeintro.fw

edgeextract.fw

The first installation screen option allows language selection among the following:  English, French, German and Japanese. Click OK to proceed or x (in the upper right) to quit.

edge1a.fw

Step 2 displays the Welcome window in your selected language. Click ‘Next’ to proceed or ‘Quit’ to exit.

edge2a.fw

Note: If you have a previous version installed, this pop-up message will display:

edge2a.fw

Step 3 displays the License Agreement: The administrator conducting the installation must accept the agreement to proceed. Click ‘Previous’ to go to the previous panel, ‘Next’ to proceed or ‘Quit’ to exit.

edge3a.fw

Step 4 displays the option to Select Destination: This panel offers a browse button and facilitates choosing an installation directory. Choose the installation destination, click ‘Previous’ to go to the previous panel, ‘Next’ to proceed or ‘Quit’ to exit.

edge4a.fw

Note: If the destination directory for the installation selected in this panel already exists then the below warning will appear. Click ‘Yes’ to continue or ‘No’ to quit.

edge5a.fw

Note: If the target directory does not exist, this screen will display:

edge2b.fw

Step 5 presents Shortcut Creation options: This screen will facilitate creating the shortcuts in the Start menu. Click ‘Previous’ to go to the previous panel, ‘Next’ to proceed or ‘Quit’ to exit.

edge6a.fw

Step 6 presents a Summary: Click ‘Previous’ to go to the previous panel, ‘Next’ to proceed or ‘Quit’ to exit.

edge7a.fw

edge7a.fw

Step 7 presents a progress screen: This will occur when the installation starts. The installer will extract the files and will try to take a backup of any previous installation. During this panel a command prompt will occur if there were initially any edge services installed. It will create a backup in the same location where it originally existed, but will append “_backup” in the directory name. Wait for the processes to complete.

Note: A clean installation is highly recommended rather than any attempt at installing over older versions of the Edge.

edge8a.fw

After the processes are completed, click Next to proceed:

edge9a.fw

Step 8 offers a GUI, Edge Server Setup Configuration: This panel writes the Edge Server FQDN, the Connect server FQDN, the Cluster ID and the server ports into the Edge server configuration.

edge10b.fw

Example entries follow based on the sample deployment diagram below:

Edge Server Hostname: edge.company.com
Connect Server Hostname: connect.company.com
Edge cluster ID: edge.dmz-edge
Connect Server Normal Port: 80
Connect Server Secure Port: 443

 

95EdgeDMZstunnelOriginstunnel

Step 9 presents the Finish Panel: The installation has completed. Click ‘Done’ to finish the installation:

edge11a.fw

Post installation, the Edge config.ini file, based on our example will contain these relevant entries:

FCS_EDGE_HOST=edge.company.com
FCS_EDGE_REGISTER_HOST=connect.company.com
FCS_EDGE_CLUSTER_ID=edge.dmz-edge=1
FCS.HTTPCACHE_BREEZE_SERVER_NORMAL_PORT=connect.company.com:80
FCS.HTTPCACHE_BREEZE_SERVER_SECURE_PORT=connect.company.com:443

Note: Prior versions of the Connect Edge often employed (although never required) a custom.ini file in the Connect Edge server installation root directory for these entries. The custom.ini would then override the config.ini file in the \conf directory. Placing a custom.ini in the root installation directory is still an option as well as a hazard should one contain stale or wrong entries. The new Edge installer writes directly to config.in through the screen illustrated in step 8.

Once the Connect Edge Server is installed, it must be registered with the origin server or cluster for which it serves as proxy:

On the origin server, register the Edge server by adding the Edge server unique name into the host mappings section of the Connect Management Console; the settings will propagate throughout an origin server cluster from any one of the origin servers:

Start > Programs > Adobe Connect Server > Configure Adobe Connect Server

If the Edge is communicating with the origin server, then you will see a preregistration configuration under the server settings tab:

edgereg1.fw

Add the same unique Edge name into the host mapping fields as follows to register the Edge; this is a manual security mechanism to prevent unauthorized pirate Edge server registration:

edgereg2.fw

Note: The common identification variable in the custom.ini on the origin and the config.ini on the Edge is the cluster ID; following our example is dmz-edge=1 indication the first zone by name; add this to the custom.ini file on the origin(s).

Note: Even a single Edge server warrants its own cluster ID.

edge.dmz-edge=1

For additional information on Edge server deployments including maintenance and troubleshooting, see the articles on the Connect Users Community. Note that the custom.ini file is used in these articles to configure the Edge by overriding the config.ini. As aforementioned, while the new 9.5 installer writes directly to the config.ini, the custom.ini, when used as described will override the config.ini.

The first tutorial listed below discusses the reverse proxy use of the Edge and the second discusses the enterprise proxy use:

Adobe Connect Edge Server Deployment Options: part 1
Adobe Connect Edge Server Deployment Options: part 2

Generating Server-side Logs to Troubleshoot On-premise Connect Deployments

In order to diagnose unexpected behavior within Adobe Connect, it may be necessary for the Adobe Connect Support team to examine server-side logs from an on-premise Connect deployment. The logs directory is located in the Connect (or Breeze – it is not uncommon for Connect upgrades to reside in legacy Breeze directories) directory:

logsdir.fw

Within the logs directory there are sub-directories containing various logs:

logsdir1.fw

The most commonly requested log by the support team, is the debug.log. It can be found in the logs>support directory. With the services running, the current debug log will appear without a date at the top of the debug.log file list. The default rollover is 12 hours generating AM and PM logs each day:

logsdir2.fw

In order to make the debug.log file more useful for purposes of diagnosis, you can enable verbose logging by adding entries to the custom.ini file located in the Connect or Breeze version sub-directory. Here you see it located in a 9.3.1 directory under the Breeze root installation/upgrade directory:

logscustomini3.fw

Before editing the custom.ini file, be sure to create a backup copy of it. Add the following lines in order to enable verbose logging:

HTTP_TRACE=true
DB_LOG_ALL_QUERIES=true

Note that for versions of Connect 9.2 and prior, use yes instead of true:

HTTP_TRACE=yes
DB_LOG_ALL_QUERIES=yes

Save the custom.ini file (be careful not to accidentally change the file type to .txt) and during a scheduled maintenance window, cycle the Connect and AMS/FMS services in order to load the changes and begin verbose logging (note this will bring Connect down while the services cycle):

logssvcs4.fw

There are occasions when it may be prudent to provide more than one log for a more complete diagnosis. To provide a full sample of the various Connect logs without sending a massive historical sample of log files, you may simply stop the Connect services (during scheduled downtime as this will bring down Connect) and rename the entire log directory to log.old. Then upon starting the services back up, recreate the issue being diagnosed and then stop the services.

This activity will generate a new small log directory isolating the issue under scrutiny that you just reproduced in Connect: Zip/compress this new abbreviated log directory with all its fresh abbreviated sub-directories and provide it to the the Adobe Connect Support team to help expedite more exhaustive server-side log analysis. This option is particularly helpful when examining a cluster as each server will have a set of logs. When providing cluster logs, always label each compressed log folder to easily identify the server from which it came.

Note that often when diagnosing unexpected behavior in Adobe Connect Meetings, it may also be prudent to enable client-side Connect addin verbose logging as well.  The relevant client-side logging tech-notes are here:

Enable logging | Meeting Add-in

Troubleshooting Verbose Meeting Addin Logging

New Adobe Connect Support Blog Subscription Option

Now you can stay on top of the new articles and posts by subscribing to the Adobe Connect Support Blog. Simply go to the Adobe Connect Support Blog home page and enter your email address and check off the categories about which you would like to be notified. Click “Subscribe me” and you will begin receiving  regular updates:

subscribe.fw

 

 

Behind the Curtain: Making Multiple Connect Meetings or Seminars Appear as One

On those occasions when a Meeting invitation may attract more participants than expected or planned for at the last minute so that you are unable to increase Seminar capacity in a timely manner, a skilled host can use two or more Connect Meeting rooms and project them to participants as though it were one room as an emergency workaround. Here is a basic outline of how to split a large meeting onto multiple servers. It is prudent to not just have more than one Meeting in these cases, but also to make sure each Meeting is hosted on a separate server in a cluster to add robustness to the meeting. Load-balancing is a wonderful thing and you should always use it to its fullest.

Assume an example of a three-server cluster/pool of Connect servers and that you want to split a Connect Meeting onto all three servers; a simple 3-server cluster is depicted here to use as an example:

C9SSLCluster3Simple

For a working example, let’s place a Connect Meeting room hosted on each server; to do this you will need three separate URLs: One URL for each 1/3rd  of your attendees. Getting the attendees distributed among the three rooms can be tricky. One effective technique is to either send out three different invitations, with each targeting 1/3rd of your audience and each offering a different URL, or just point everyone to a page with  all three URLs and request/instruct the participants to alphabetically arrange themselves in subsets of users by URL selection. That way it is not random; I have seen this technique work fine; here are sample meeting URLs based on our picture above:

http://connect.domain.com/splitmeeting1
http://connect.domain.com/splitmeeting2
http://connect.domain.com/splitmeeting3

To make certain the each meeting is hosted on a separate server (rather than all three on one as load-balancing could easily prescribe), it will require some effort to keep entering and leaving the room until your meeting lands on the server you want. Using multiple browsers may be helpful as well. Working on this well in advance of the meeting is prudent as there is a session timeout factor to consider. The load balancing algorithm will eventually get the sessions distributed but it may take some effort.

The way to tell which server you are on is simple: In any meeting room click Help and while holding down the shift key click About Adobe Connect. This will pop up an RTMP string that will identify the server that Meeting is hosted on and also which server a client is coming through as each client can be using multiple servers (just to add not only to the complexity, but also the overall robustness).

Here is what the RTMP strings might look like for each of the three servers in our simple example above ( I am inserting some URL parameters from a hosted meeting as I write this in order to create our hypothetical example RTMP strings – rtmp://arfms3.adobeconnect.com:1935/?rtmp://pcparapp07:8506/meetingas3app/89676385/630888204/)

rtmps:// connectmtg01.domain.com/?rtmp://connapp01:8506/meetingas1app/847483075/1086833045/
rtmps:// connectmtg02.domain.com/?rtmp://connapp02:8506/meetingas2app/847483076/1086833046/
rtmps:// connectmtg03.domain.com/?rtmp://connapp03:8506/meetingas3app/847483077/1086833047/

The first name in the string (connectmtg0#) is the built-in Connect Edge server and the second name (connapp0#)  is the Connect origin server  hosting the meeting (each Connect servers runs both AMS/FMS and Tomcat together). The second name is the important one for our technique of splitting the attendees onto separate meeting servers.

In the hypothetical RTMP string samples above, I have made these artificially neat and tidy, the truth is that the first part of the string can be any of the three for any meeting participant regardless of the application server hosting the meeting. For example, you could come in to connapp01 through connectmtg03 – any combination is possible. Load balancing is done at more than one level as Connect leverages both a hardware-based load-balancing device and also its own internal clustering capabilities; combinations for various clients (including the hosts and presenters) in our example cluster depicted  above might include:

rtmps:// connectmtg01.domain.com/?rtmp://connapp02:8506/meetingas2app/847483076/1086833046/
rtmps:// connectmtg02.domain.com/?rtmp://connapp02:8506/meetingas2app/847483076/1086833046/
rtmps:// connectmtg01.domain.com/?rtmp://connapp03:8506/meetingas3app/847483077/1086833047/
rtmps:// connectmtg03.domain.com/?rtmp://connapp03:8506/meetingas3app/847483077/1086833047/
rtmps:// connectmtg02.domain.com/?rtmp://connapp01:8506/meetingas1app/847483075/1086833045/
rtmps:// connectmtg03.domain.com/?rtmp://connapp01:8506/meetingas1app/847483075/1086833045/

The key to remember is that the second name is the one that matters; a distribution of participants approximating 1/3rd on each server is the goal targeting: connapp01, connapp02 and connapp03. After this is set-up, the pre-meeting preparation part is complete (this should be done at least one hour prior to the meeting).

Next comes the creative hosting venture during the split meeting: As the host, you will need all three meetings open in front of you to manage them as one. From the perspective of the participants, there is only one meeting (ignore the host behind the curtain). Be sure to hide the Attendee List Pod in the Presenter-only area as it will only present those participants in that specific Connect Meeting thereby allowing a peek behind the curtain or misrepresenting the size of the entire three meeting combination.

And here is where the techniques are very much up to you:

  • Splitting video among the three rooms is possible using a third-party option, one we have used successfully is: Splitcam.com.
  • For audio, if using integrated audio, be sure to use the same integrated telephony number for all three rooms.
  • If using VoIP, then allow one speaker only at a time to send audio via VoIP.

Some ways in which you can limit the amount of data being processed in your room and to improve the overall performance of these sessions are:

  • Optimize room bandwidth. In a Connect Meeting, at the top of the screen click on MEETING > Preferences. Under the preferences menu you are able to adjust screen sharing, video and VoIP quality setting separately.
  • Turn off cameras whenever they are not in use.
  • When in use, multiple cameras should probably be set to SLOW images (depending on how many and other variables).
  • Turn off VoIP if not talking.
  • Participants should directly connect to the fastest internet connection available and be on a dedicated DSL connection, at a minimum.
  • No clients or hosts on wireless – allow no exceptions.
  • Shut down Email, instant messaging, and any programs NOT being used for the presentation.
  • Shut down any VPNs as a VPN will potentially destroy the possibility for success.

When large Connect Meetings or Seminars become commonplace in your enterprise, this cumbersome workaround quickly becomes impractical and you should increase your Seminar or Webinar licensed capacity as needed to avoid this complexity and manual work. With that said however, this technique will work in a bind and will provide a robust Connect Meeting experience for a very large audience even if it challenges a seasoned Connect Meeting host.

Changing the License Serial Key in Connect

This article applies to on-premise and managed ISP Connect users. It does not apply to multi-tenancy hosted or ACMS.

On rare occasions it may be necessary to change the serial key in Adobe Connect. Here are the steps:

  1. Navigate to: \Connect_installation_directory\appserv\conf\config.ini and change the value of  SERIAL_KEY=  to reflect the new serial number
  2. In \Connect_installation_directory\custom.ini,  if there’s a serial key value listed (SERIAL_KEY=), replace it there as well.
  3. Using MSSQL Studio Express (or your choice of SQL editing options), view the serial key currently being used by Connect by running this command: SELECT * from pps_accounts WHERE name=’Enterprise Account’
  4. To get Connect to accept the new license you must change the serial key that is currently in the database by running this SQL command: UPDATE pps_accounts SET serial_key = ‘NEW_SERIAL_NUMBER’ WHERE serial_key = ‘OLD_SERIAL_NUMBER’
  5. Restart the services: Application Server (Connect) and the Meeting Server (AMS or FMS depending on the version of Connect) services.fw
  6. Open the Administration Console (port 8510 locally on any Connect server)

connconfig.fw

7. Go to License Settings and upload the new license file.

connconfiglic.fw

8. Restart the AppServer (Connect) and the Meeting server (AMS or FMS depending on version) again and the  new license file will be applied

services.fw

Troubleshooting: If there are any problems, do the following to troubleshoot:

  • Shut down the Connect and AMS or FMS Services
  • Open and verify \Connect_installation_directory\appserv\conf\config.ini and update the entry for SERIAL_KEY
  • Open and verify  \Connect_installation_directory\custom.ini and update  the entry for SERIAL_KEY
  • Open SQL Server and choose the Connect database and run the following script (replacing the text as appropriate):

“Input New Serial Key Here” with the New Serial Key but leaving the quotes.
DECLARE @NEW_SERIAL VARCHAR(32)
SET @NEW_SERIAL=’Input New Serial Key Here’

UPDATE PPS_CONFIG
SET VALUE = @NEW_SERIAL
WHERE SECTION=’cps’ AND NAME=’serial_key’

UPDATE PPS_ACCOUNTS
SET SERIAL_KEY = @NEW_SERIAL
WHERE ACCOUNT_ID=7

UPDATE PPS_ENUM_DATA_HOSTS
SET LICENSE = @NEW_SERIAL
WHERE HOST_ID > 0

db.fw_

  • Start the Connect and FMS services

Problems will ensue when the license is reducing the allowed usage of Connect (if you are downsizing) and you leave an overage in place. For example, if you have 100 meeting hosts assigned, and you are changing to a license that only allows 50 named meeting hosts then when you  apply the license you will get an error unless you have reduced the number to accommodate the new licensed restriction.

Connect 9.5 Edge Server Installation Instructions

Note: The upgrade installer described in this article below is deprecated. For instruction on the latest 9.5 installer, please see the following article: 9.5 Connect Edge Proxy Server Full Installer

Connect 9.5 server installation instructions:

  1. Create a folder <Installation_Directory>/950/edgeserver
  2. Download the Edge 9.5 (based on AMS 5) installer
  3. Run the self-extracting .exe file downloaded in step#2 to <Installation_Directory>/950/edgeserver
  4. Refer the following articles for deployment options:
    1. http://www.connectusers.com/tutorials/2011/06/edge_server_deploy/
    2. http://www.connectusers.com/tutorials/2011/06/edge_server_deploy2/
  5. Run <Installation_Directory>/950/edgeserver/win32/vcredist_x64.exe
  6. Run the following commands as administrator:
    1. cd <Installation_Directory>/950/edgeserver/win32AMSAdmin.exe -install
    2. AMSMaster.exe -install
    3. sc start amsadmin
    4. sc start ams
  7. Confirm that services “Adobe Media Administration Server” and “Adobe Media Server (AMS)” are running
  8. If services need to run using specific user credentials, then be sure to set the credentials in service properties and restart the services

Troubleshooting Verbose Meeting Addin Logging

On occasion it can be difficult to get verbose addin logging to work. The tech-note describing how to set it up is here: Enable logging | Meeting Add-in

The tech-note correctly describes where to place the customized mms.cfg file for use with both 64 bit and 32 bit Windows clients as well as for the Mac OS.

If after following the instructions in the tech-note, you still do not see any verbose addin logs, one possible cause is that there may be an additional mms.cfg file in an alternate location on the client that is blocking the log creation process. To remedy this, add the customized debug mms.cfg to the following locations after renaming any existing mms.cfg files (to allow them to be restored after verbose logging or debugging is complete):

Here are the locations (more than in the tech-note):

  • Windows (32 bit) :

In: C:\Windows\System32\Macromed\Flash\mms.cfg
or C:\Windows\System32\mms.cfg

  • Windows 7 (64 bit):

In: c:\Windows\SysWOW64\Macromed\Flash\mms.cfg
or c:\Windows\SysWOW64\mms.cfg

After placing the mms.cfg in both folders, be sure to close all addin browsers and then to open the addin only in the one Meeting that you wish to troubleshoot.