Posts in Category "Edge Server"

Connect 9.5 Bandwidth Utilization Estimates Illustrated

Introduction:

Adobe Connect bandwidth utilization will vary based on use case. Variables such as Meeting size, Connect features employed, disposition of clients and type of Connect server hosting, all have an effect on the network bandwidth utilized and as well as on where the bandwidth on a network will be most affected.

This article focuses on bandwidth and not on latency: Latency and bandwidth are different topics that are often confused or inappropriately lumped together. Although in some cases there may be a correlation between bandwidth and latency as when exceeding available bandwidth will drive up latency, nevertheless, the relationship between them is often inappropriately overstated. Bandwidth refers to the size of the network pipe while latency refers to the speed over the network or the time it takes for the data to travel. Connect clients with excess bandwidth may still experience latency with Connect for reasons unrelated to bandwidth such as network distance, transmission errors, resource performance (server, DB or client), maintenance schedules and imposed constraints such as QoS or blocking the RTMPS protocol and thereby forcing inefficient tunneling of RTMPS encapsulated in HTTPS.

Resources:

For information on latency see the following articles:

Tunneling with RTMP encapsulated in HTTP (RTMPT) should be avoided as it causes latency

Connect Meeting RTMP VS/VIPs on Load-Balancers

Adobe Connect Database Performance and Monitoring

 Connect on VMWare – some deployment tips

Below are some references on the topic of Connect bandwidth; these are often referred to as a general guide; the examples sited later in this article however are from baseline testing described herein:

Estimating Bandwidth Consumption in Connect Meetings

Best Practices for Adobe Connect

Adobe Connect best practices for large events and seminars

VoIP Bandwidth and Microphones

The Connect Cluster:

The context of the Connect bandwidth utilization is important and should be included among pre-deployment architectural considerations when rolling out Connect. There are basically two primary Adobe Connect deployment models for consideration: In the cloud and on-premise. The disposition of the clients attending the meeting has an effect where the on the network bandwidth will be utilized for any Connect Meetings or Webinars. To support these two broad deployment paradigms of cloud or local on-premise, there are three basic Adobe Connect licensing models.

  1. Adobe Connect Multi-tenancy Hosted
  2. Adobe Connect Managed Services (ACMS) single domain hosting
  3. Adobe Connect On-Premise licensed single domain

Note: An Adobe Connect on-premise license need not only be deployed within the customer’s LAN infrastructure. Adobe Connect Partners offer Managed ISP hosting options and an on-premise license deployment can be cloud-based as well.

Cloud or LAN:

For the purposes of considering the relationship of bandwidth to architecture, an Adobe Connect cloud-based deployment and an on-premise deployment within a customer’s LAN present different concerns with reference to the disposition of bandwidth rather than with the specific license type.

Primary differences with reference to bandwidth when comparing cloud-based Connect server clusters with on-premise based Connect server architecture is with the handling of external clients and the direction of the network traffic. There will be less internal network traffic on the LAN when external users (with possible lower-impact exceptions such an SSO server) remain external. Note that VPN users in remote offices are considered internal users.

See the following diagrams comparing a 2-server cloud-based cluster with a 2-server on-premise cluster hosted on a customer LAN. Both assume the need for internal and external users to participate in Meetings.

Cloud-based:

With a cloud deployment, the external users have little or  no impact on the internal LAN of the Connect account owner. The biggest bandwidth hit is at the cloud-based hosting facility. The primary concerns for the Connect owner’s IT will be routing RTMPS around third-party proxy servers and making sure client profiles and other internal restriction, limitations and constraints do not prevent or enervate Connect usage:

two-server-cloud

LAN-based:

With a LAN-based on premise deployment, internal Connect users will normally have great performance on the average corporate back-plane by avoiding any mitigating and often unpredictable WAN variables. External users may be hosted on a reverse proxy Connect Edge server in the local DMZ to meet security requirements. All external traffic is terminated in the DMZ where the remote Edge server does all the heavy lifting. The Edge connections to the internal cluster consolidate the external connections and reduce the overall internal impact of external users by limiting them to the external firewall and DMZ:

EdgeDMZAbbrev

Connect Meeting Features and Bandwidth:

Screen-sharing and the use of multiple live web cameras are the two most bandwidth intensive activities in any Connect Meeting or Webinar. The latter requires that a Meeting Host or Presenter project the client screen up to the Connect servers and then back down to each user in the Meeting. The more users with cameras, the more upload and download activity.

The features employed in any Connect Meeting determine the amount of bandwidth utilized.  Depending on the Connect use case, you could use between 5 to 10 Mbps per 100 participants. These general rules apply:

    • The typical average is 50 kbps per user
    • It depends on what Connect Meeting users are doing
    • For example, on the high end one client measured regularly uses 20 Mbps for 150 users in a Connect Meeting room.
    • More typically there is a max of 100 Mbps at the firewall (outgoing, incoming is never that high)
    • With just over 1000 users a safe estimate is < 100kbps per user totaling 12 MB
    • 5 Mbps per 100 is the low average (or 6MB per 1000 users using best practices for large Webinars).
    • Things like video and screen sharing tend to use more bandwidth
    • A one-to-many broadcast of an uploaded PowerPoint deck uses less bandwidth
      • PPTX should always be uploaded to the Meeting room
      • Do not present a PPTX using screen-sharing as it wastes bandwidth

Bandwidth Utilization Use Case Baseline Tests:

The following tests corroborate these guidelines listed above and offer a model to test bandwidth utilization for any Meeting or Webinar use case; it is prudent to do these type of tests with your specific use Webinar case as part of your rehearsals to get accurate bandwidth utilization baselines:

First bandwidth test: I uploaded a 15.5 MB PPTX to a Connect Meeting room and advanced thru all the slides while monitoring the download stream to each client; the average for each client was 50 kilobits; see below:

4clients.fw

Second bandwidth test: I then tested with a camera pod set to the highest quality and wide screen option: Meeting>Preferences>Video; see my host camera settings below:

videopref.fw

The average download stream at each client when I had the camera pod sized small in the upper corner of the meeting while flipping slides was under 200 kbps (kilobits); The host screen capture below corroborates; my host client in the Connect addin is pushing 121 kilobits up to the server; see below:

cambndwdthtrun

The average when I expanded the camera pod and moved around create some activity on screen (waving etc.) was 300kbps download streaming to each client; see below my upload from host client to server is 381 kbps; see below (wow- my hair is getting thin):

cambndwdthlrgtrun

Third bandwidth test: I then tested screen-sharing a slide deck from my desktop to simulate application sharing live demonstrations and found a significant variance based on the screen resolution of the host sharing the screen:

  • Host screen-sharing at 1920×1080 resulted in an average  download to each participant of 700 kbps
  • Host screen-sharing at 1288×768 resulted in an average download to each participant of 200 kbps
  • The participant experience was very similar. I see no reason to waste the bandwidth on the higher host screen-resolution: Lower screen resolution resulting in very economical bandwidth utilization while sharing the screen
  • See the bandwidth indicator while I share my screen flipping through multiple jpegs at 1288×768. I am pushing up 338 kbps to the Connect server while actively changing jpeg images on a second monitor that I am sharing.

hostss1280.fw

Fourth bandwidth test: Video – I created an mp4 using the following settings: Appropriate quality settings are a major variable determining the amount of bandwidth used and the playback experience of mp4 video in Connect:

  • I used: H.264, 720×404, Frame Rate 30, bitrate CBR, target rate 1Mbps, Key Frame Dist 90.
  • Best practice encoding for video in Connect imperative: Rehearsal and testing of any Video used is prudent prior to any Meeting or Webinar. It is easy to test playback experience and bandwidth consumption per client participant. Note also here (in the highlighted bandwidth indicator) how my remote office client experiences some latency albeit manageable  (595kbps down) that not affect the video playback experience – no choppiness at all watching the pups wrestle (the Jack-Russell ends up winning although this capture shows him as the underdog):

video

Bandwidth Utilization Use Cases:

Using the baselines from these tests, here is a list of potential use cases from a recent customer inquiry with the predicted overall account-wide bandwidth usage estimates running Connect 9.5.

Case 1: A 1500 user seminar room with a PowerPoint and a desktop share:

Estimate: 1500 users x 50 kilobits each for the PPTX use case =75000 kilobits or 9MB. Add to that a desktop share with conservative screen-resolution (1280×768) 1500×200=300000 kilobits or 37MB. Add them together (75000+300000) for an average of 375000 kilobits or 47MB. Realistically it will be less at any given time because it is rare to simultaneously do both activites at once in a meeting. It is more common to share either flip PPTX slides or share an application thereby lowering the actual bandwidth utilized at any given time.

 Note: There is an important caveat with reference to an initial spike caused by the PPTX SWF download of 1000 kbps per client participant for a few seconds each. This affects every use case with SWF content is a share pod. Assuming 1/3 at a time as participants enter the layout with the PPTX on stage in a share pod even if downsized in a corner or hidden behind another pod, and assuming that there will not be any screen-sharing initially during the downloading: 1500/3 = 500x1000kbps for a few seconds each = 50000 kilobits or 62.5MB. A worst case spike scenario of all entering at once due to host error or a late upload due to last minute editing (this should be avoided if possible): 1500×1000=1500000 kilobits or187MB Spike for up to 10 seconds depending on latency effects on the initial SWF download. This spike may be missed in testing if you rely on the Meeting room bandwidth indicator as it may happen too fast to register; it is better measured using Wireshark on a client in the Meeting.

Case 2: A 1500 user seminar room with a PowerPoint and a 10 minute MP4 Video played:

Estimate: 1500 users x 50 kilobits each for the PPTX=75000 kilobits or 9MB. Assuming a medium quality MP4 (as discussed above) uploaded to the meeting room and meted out at an average 600 kbps per participant with intermittent spikes to 1200 kbps: 1500×1000=1500000 kilobits or187MB + 9 = 196MB while the video is being played with the PPTX still onstage visible in another pod. Note: Until the Video is played, this room is at 9MB just meting out PPTX SWFs after the initial SWF downloads are complete.

 Note: PPTX download spike variables as described in case # 1 above applies.

Case 3:  A 1500 user seminar room with a PowerPoint and two, 3 minute videos played:

Estimate: Same as case 2 above assuming that the videos are played one at a time.

Case 4: Two 1500 user seminars with just PPTs for content (uploaded to the Meeting room and not screen-sharing) and jpeg pictures for Presenter head-shots in lieu of a camera:

Estimate: 1500 users x 50 kilobits for the PPTX=75000 kilobits or 9MB x 2 seminar rooms = 18MB

 Note: Assuming the seminars will not start both at the same time and will nevertheless overlap, the brief PPTX download spike variables as described in case # 1 above applies albeit multiplied by two: 126 MB for 1/3rd and worst case of both seminars doing a last minute deck upload at the start of the Meeting: 274MB until initial SWF downloads complete. Staggering slightly the start of each seminar will mitigate the overall spike as will allowing users to trickle into a prepared room with assets on stage.

Case 5:  Two 1500 user seminars with just PPTs for content (uploaded to the Meeting room and not screen-sharing) and PPTs for headshots shared, PLUS 100 separate Connect meetings of 20 users each, with 10 of those meeting rooms sharing one web camera:

Estimate: 1500 users x 50 kilobits each for the PPTX=75000 kilobits or 9MB x 2 = 18MB, plus 90 meeting rooms with 20 users each doing the same activity, 1800×50 kilobits=90000 kilobits or 11MB, plus the 10 meetings with cameras added 10 x 20 users each x 200 kilobits=40000 kbps or 5MB; also add the upload for each camera 10 cameras x 200 kilobits =2000 kbps or .25MB. Total =.28000 kilobits or 35MB.

Note: Assuming the various meetings will be starting at different times the spike estimates for #4 above apply.

Case 6:  Two 1500 user seminars with just PPTs for content and PPTs for headshots shared, PLUS 100 separate Connect meetings of 20 users each, with 10 of those meeting rooms sharing one web camera, PLUS one 500 user seminar with a PowerPoint shared(uploaded to the Meeting room and not screen-sharing).

Estimate: Add 500 users x 50 kilobits each = 25000 kilobits or 4MB. Add this to use case 5 depicted above: 35MB+4=39MB.

Note: Assuming the various meetings will be starting at different times the spike estimates for use cases #4 and #5 above apply here.

Case 7: Two 1500 user seminars with just PPTs for content and PPTs for headshots shared,  PLUS 50 separate Connect meetings of 20 users each with 10 of those meeting rooms sharing one web camera, and one 500 seminar with just a PowerPoint shared (uploaded to Meeting room and not screen-sharing):

Estimate: 1500 users x 50 kilobits each  for the PPTX=75000 kilobits or 9MB x 2 = 18MB, plus 40 meeting without cameras x 20 users each=800x50kbps= 40000kbps or 5MB, add the 10 meetings with cameras 10 meetings x 20 users x 200 kilobits=40000 kbps or 5MB; also add the upload for each camera 10×200=2000 kbps or .25MB. Total= 232000 kilobits or 29MB.

 Note: Assuming the various meetings will be starting at different times the spike estimates for use case #4, #5 and #6 above apply.

Conclusion:

It is easy to avoid both overestimating and underestimating the bandwidth needed account-wide for Connect deployments on-premise or in the cloud. For large Webinars, a quick rehearsal with a small sample audience of external and internal users as appropriate is prudent. Best practices in large Meetings make a substantial difference in the amount of bandwidth utilized: Proper camera use, screen-sharing and video-encoding are all very important considerations.  Although it may at this point seem like a mantra, a basic best practice that so many Connect users miss is to upload a PowerPoint to the Meeting room share pod rather than use screen-sharing to run through a slide-deck. Maybe some Presenters are just overly possessive and have an innate refusal to let their work upload – whatever the reason, I will surely be in a Meeting today and some Presenter will share a PowerPoint from their desktop via screen-sharing and I will need to avert my eyes to maintain my light grasp on sanity. Used properly, Adobe Connect is the best means to deliver rich collaboration experiences while minimizing the impact on bandwidth utilization.

9.5 Connect Edge Proxy Server Full Installer

The new 9.5 Connect Edge Proxy Server full installation procedure follows.

Note: This article applies to on-premise Connect customers who have purchased Edge Proxy servers and must install or upgrade to Connect version 9.5. The only possible exception to on-premise exclusivity may be for those who are hosted by a managed ISP that supports external Enterprise Proxy Edge Servers (this latter model is uncommon).

Step 1: Download the Connect Edge Server installer from the URL location provided by the Adobe Connect Support or Customer Service team;  extract and install it with local administrative permissions.

edgeintro.fw

edgeextract.fw

The first installation screen option allows language selection among the following:  English, French, German and Japanese. Click OK to proceed or x (in the upper right) to quit.

edge1a.fw

Step 2 displays the Welcome window in your selected language. Click ‘Next’ to proceed or ‘Quit’ to exit.

edge2a.fw

Note: If you have a previous version installed, this pop-up message will display:

edge2a.fw

Step 3 displays the License Agreement: The administrator conducting the installation must accept the agreement to proceed. Click ‘Previous’ to go to the previous panel, ‘Next’ to proceed or ‘Quit’ to exit.

edge3a.fw

Step 4 displays the option to Select Destination: This panel offers a browse button and facilitates choosing an installation directory. Choose the installation destination, click ‘Previous’ to go to the previous panel, ‘Next’ to proceed or ‘Quit’ to exit.

edge4a.fw

Note: If the destination directory for the installation selected in this panel already exists then the below warning will appear. Click ‘Yes’ to continue or ‘No’ to quit.

edge5a.fw

Note: If the target directory does not exist, this screen will display:

edge2b.fw

Step 5 presents Shortcut Creation options: This screen will facilitate creating the shortcuts in the Start menu. Click ‘Previous’ to go to the previous panel, ‘Next’ to proceed or ‘Quit’ to exit.

edge6a.fw

Step 6 presents a Summary: Click ‘Previous’ to go to the previous panel, ‘Next’ to proceed or ‘Quit’ to exit.

edge7a.fw

edge7a.fw

Step 7 presents a progress screen: This will occur when the installation starts. The installer will extract the files and will try to take a backup of any previous installation. During this panel a command prompt will occur if there were initially any edge services installed. It will create a backup in the same location where it originally existed, but will append “_backup” in the directory name. Wait for the processes to complete.

Note: A clean installation is highly recommended rather than any attempt at installing over older versions of the Edge.

edge8a.fw

After the processes are completed, click Next to proceed:

edge9a.fw

Step 8 offers a GUI, Edge Server Setup Configuration: This panel writes the Edge Server FQDN, the Connect server FQDN, the Cluster ID and the server ports into the Edge server configuration.

edge10b.fw

Example entries follow based on the sample deployment diagram below:

Edge Server Hostname: edge.company.com
Connect Server Hostname: connect.company.com
Edge cluster ID: edge.dmz-edge
Connect Server Normal Port: 80
Connect Server Secure Port: 443

 

95EdgeDMZstunnelOriginstunnel

Step 9 presents the Finish Panel: The installation has completed. Click ‘Done’ to finish the installation:

edge11a.fw

Post installation, the Edge config.ini file, based on our example will contain these relevant entries:

FCS_EDGE_HOST=edge.company.com
FCS_EDGE_REGISTER_HOST=connect.company.com
FCS_EDGE_CLUSTER_ID=edge.dmz-edge=1
FCS.HTTPCACHE_BREEZE_SERVER_NORMAL_PORT=connect.company.com:80
FCS.HTTPCACHE_BREEZE_SERVER_SECURE_PORT=connect.company.com:443

Note: Prior versions of the Connect Edge often employed (although never required) a custom.ini file in the Connect Edge server installation root directory for these entries. The custom.ini would then override the config.ini file in the \conf directory. Placing a custom.ini in the root installation directory is still an option as well as a hazard should one contain stale or wrong entries. The new Edge installer writes directly to config.in through the screen illustrated in step 8.

Once the Connect Edge Server is installed, it must be registered with the origin server or cluster for which it serves as proxy:

On the origin server, register the Edge server by adding the Edge server unique name into the host mappings section of the Connect Management Console; the settings will propagate throughout an origin server cluster from any one of the origin servers:

Start > Programs > Adobe Connect Server > Configure Adobe Connect Server

If the Edge is communicating with the origin server, then you will see a preregistration configuration under the server settings tab:

edgereg1.fw

Add the same unique Edge name into the host mapping fields as follows to register the Edge; this is a manual security mechanism to prevent unauthorized pirate Edge server registration:

edgereg2.fw

Note: The common identification variable in the custom.ini on the origin and the config.ini on the Edge is the cluster ID; following our example is dmz-edge=1 indication the first zone by name; add this to the custom.ini file on the origin(s).

Note: Even a single Edge server warrants its own cluster ID.

edge.dmz-edge=1

For additional information on Edge server deployments including maintenance and troubleshooting, see the articles on the Connect Users Community. Note that the custom.ini file is used in these articles to configure the Edge by overriding the config.ini. As aforementioned, while the new 9.5 installer writes directly to the config.ini, the custom.ini, when used as described will override the config.ini.

The first tutorial listed below discusses the reverse proxy use of the Edge and the second discusses the enterprise proxy use:

Adobe Connect Edge Server Deployment Options: part 1
Adobe Connect Edge Server Deployment Options: part 2

New Adobe Connect Support Blog Subscription Option

Now you can stay on top of the new articles and posts by subscribing to the Adobe Connect Support Blog. Simply go to the Adobe Connect Support Blog home page and enter your email address and check off the categories about which you would like to be notified. Click “Subscribe me” and you will begin receiving  regular updates:

subscribe.fw

 

 

Connect 9.5 Edge Server Installation Instructions

Note: The upgrade installer described in this article below is deprecated. For instruction on the latest 9.5 installer, please see the following article: 9.5 Connect Edge Proxy Server Full Installer

Connect 9.5 server installation instructions:

  1. Create a folder <Installation_Directory>/950/edgeserver
  2. Download the Edge 9.5 (based on AMS 5) installer
  3. Run the self-extracting .exe file downloaded in step#2 to <Installation_Directory>/950/edgeserver
  4. Refer the following articles for deployment options:
    1. http://www.connectusers.com/tutorials/2011/06/edge_server_deploy/
    2. http://www.connectusers.com/tutorials/2011/06/edge_server_deploy2/
  5. Run <Installation_Directory>/950/edgeserver/win32/vcredist_x64.exe
  6. Run the following commands as administrator:
    1. cd <Installation_Directory>/950/edgeserver/win32AMSAdmin.exe -install
    2. AMSMaster.exe -install
    3. sc start amsadmin
    4. sc start ams
  7. Confirm that services “Adobe Media Administration Server” and “Adobe Media Server (AMS)” are running
  8. If services need to run using specific user credentials, then be sure to set the credentials in service properties and restart the services

Troubleshooting Verbose Meeting Addin Logging

On occasion it can be difficult to get verbose addin logging to work. The tech-note describing how to set it up is here: Enable logging | Meeting Add-in

The tech-note correctly describes where to place the customized mms.cfg file for use with both 64 bit and 32 bit Windows clients as well as for the Mac OS.

If after following the instructions in the tech-note, you still do not see any verbose addin logs, one possible cause is that there may be an additional mms.cfg file in an alternate location on the client that is blocking the log creation process. To remedy this, add the customized debug mms.cfg to the following locations after renaming any existing mms.cfg files (to allow them to be restored after verbose logging or debugging is complete):

Here are the locations (more than in the tech-note):

  • Windows (32 bit) :

In: C:\Windows\System32\Macromed\Flash\mms.cfg
or C:\Windows\System32\mms.cfg

  • Windows 7 (64 bit):

In: c:\Windows\SysWOW64\Macromed\Flash\mms.cfg
or c:\Windows\SysWOW64\mms.cfg

After placing the mms.cfg in both folders, be sure to close all addin browsers and then to open the addin only in the one Meeting that you wish to troubleshoot.

Tunneling with RTMP encapsulated in HTTP (RTMPT) should be avoided as it causes latency

Tunneling with RTMP encapsulated in HTTP or RTMPT should be avoided as it causes latency that can have a negative impact on user experience in a Connect meeting. In rare circumstances,the latency commensurate with tunneling RTMP encapsulated in HTTP, can become so acute that it renders Connect unusable for affected clients. The performance hit commensurate with tunneling is one of the primary reasons we continue to deploy Connect Edge servers as they often can replace third-party proxy servers that are often the cause of tunneling latency..

While the amount of acceptable latency depends on what one is doing in the room; RTMPT tunneling affects most activities. Some activities, such as screen-sharing are more bandwidth intensive than other activities such as presenting an uploaded PowerPoint from within a meeting room; The high latency commensurate with RTMPT tunneling would affect the former more than the latter. VoIP is often the first thing to make the effects of high latency felt. Here is some feedback from a test I did while on site with a client dealing with tunneling because of their refusal to pipe RTMP around a third-party proxy:.

External user tunneling during test:
Spike at 3.10/3.02 sec
Latency 403/405 ms up to 3.53/3.52 sec up .064 down 118
When latency peaks to 2.6/2.4 sec I get a mild interruption to the audio V
Video pauses momentarily when the latency spikes

Internal user with direct connection:
2 msec / 1 msec Up 0.08 kbits down 127 kbits
No pauses, delays or spikes

Tunneling should only be considered as a fallback mechanism or safety net to allow connections when RTMP is blocked due to something unplanned or for a few remote clients who must negotiate specific network obstacles. When RTMPT is the default by network design, you will need to limit your activities within Connect to those feature that use the least bandwidth.

The picture below shows a direct connection over RTMPS on 443 is being blocked somewhere on the client’s network and the fallback mechanism built into Connect of tunneling RTMP encapsulated in HTTPS is the fallback path. This is usually caused by either proxy servers or firewalls or both – any application-aware appliance on one’s network that sees the RTMPS traffic on 443 and recognizes that it is not HTTPS is a potential obstacle; RTMPS traffic is on port 443 and while it is disguised as HTTPS, it still may be blocked. The result is tunneling, indicated by the “T” in the output:

.tunneljpgOctagon

Compare with a connection without tunneling:

no-tunnel

The recommended steps for anyone experiencing persistent tunneling, is for their network engineers to trust the source IP addresses of the Adobe hosted, ACMS, managed ISP or on-premise Connect/FMS server VIPs in order to prevent the blocking of RTMPS traffic. RTMPS is not supported by any third-party proxy server. Static bypass works well to solve this issue. The problem stems from network policies that require all traffic to go through a proxy. The result is tunneling with commensurate high latency and drops. RTMPS must be allowed to stream around a proxy to avoid the overhead and latency of tunneling encapsulated within HTTPS. Attempts to cache the stream add no value.

Other options will depend on the capabilities of the third-party proxy servers in the affected client infrastructure. Blue Coat ProxySG is one of the popular proxy server options in our niche. In cases of latency invoked by tunneling RTMP encapsulated in HTTP on a network that employs Blue Coat ProxySG servers, sniff tests done by support representatives have indicated that when an affected client attempts to connect to an Adobe Connect meeting, those clients would establish both explicit HTTP connections based on PAC file settings in the system registry to the Blue Coat ProxySG pool through a hardware-based load balancing device (HLD) and transparent HTTP and SSL connections through Blue Coat ProxySG via WCCP GRE redirect to several Adobe Connect servers. The problem manifests with RTMPS when the clients attempt to establish an SSL connection directly to the destination host without going through PAC file proxy settings. Since a Blue Coat ProxySG is commonly configured to perform an SSL intercept on both explicit and transparent HTTPS traffic, upon examining the content after decrypting the SSL payload from the clients, the Blue Coat ProxySG will return an exception and close the connection because the request doesn’t contain an HTTP component and cannot be parsed for policy evaluation. As a workaround, other than using static bypass, it is possible to create a proxy service with the destination set to the Adobe Connect server IP range on port 443 and to set the proxy setting to TCP-Tunnel with Early Intercept enabled. This will allows Blue Coat ProxySG to intercept and tunnel the traffic without considering whether it is RTMPS or HTTPS.

Watch for a more comprehensive article on this topic forthcoming.

Connect Meeting Bandwidth Utilization using Multiple Interactive Collaborating Video Feeds

Usage question: How many Video feeds can I have running in my meeting room at once?

Answer: Let’s consider a working example around the bandwidth utilization of six Video cameras in a single meeting room consisting of one host and five participants. This working example may be that a of an interactive management meeting or of a college classroom where multiple students interact in a small group session using their webcams. From an examination of this example, you will be able to calculate video camera utilization parameters for other meetings whether they be larger or smaller ones.

To help illustrate what I mean, see this picture from our Connect 9.1 Release Notes

six-cams.fw

Each of our six webcam-wielding clients is connected to the server and will receive five video streams from the server (N-1).

Lets calculate first the number of streams outbound: 6 x 5 = 30

Lets also consider the 6 publishing streams from each client to the Connect server for a total of 36  total streams to support the Connect Video pods.

Now lets calculate the amount of bandwidth used by each stream; here you have power to decide how much bandwidth is to be used by each stream as there are many variables that Adobe Connect puts in your control:

In your meeting room, as a host, click Meeting > Preferences:

meeting-preferences.fw

Under Meeting > Preferences, there are two important options that you are going to adjust – Bandwidth and Video:

meeting-preferences-room-bandwidth.fw

meeting-preferences-video.fw

 

The size of the video streams commensurate with each webcam instance will depend on how you configure these settings.

webcam.fw

Following our example, if you go with the settings that I have depicted in the screen captures above to support the 6 Video feeds in a single meeting: DSL Bandwith and Standard Video quality = 213 kbps per stream:

36 streams x 213 = 7668 kbps or 8 Mbps for the 6 separate cameras.

There are other variables to consider as well. Building on our example, let’s say you also want to use VoIP:

VoIP.fw

DSL VoIP = 22 kbs x 36 = 792 kbs or 1Mbps (rounded up) additional bandwidth needed.

There are other ways to optimize: the video streams are always larger when clients use the Flashplayer in a browser rather than using the Connect Meeting addin. The Connect addin uses the ON2 codec and is far more economical when it comes to bandwidth utilization. For each client running without the Connect addin it is prudent to plan for an additional 50% for each of their Video streams. To avoid this additional bandwidth consumption, send out a link with the Adobe Connect Addin prior to your meeting and encourage clients to install it. It is a small modified version of the Flashplayer:

Another variable to consider that when the Video instance sizes are smaller, Connect adjusts to a lower publishing resolution to save some bandwidth. Unless you are sure the clients have the addin, the final planning number for our 12 webcam meeting is:

300 kbps for each stream (assuming that the addin will not be ubiquitous) x 36 stream = 11 Mbps + 1 Mbps for VoIP = 12Mbps.

Presenting a PowerPoint or a PDF that is uploaded to the Meeting room does not add to the overhead. Chat, Notes and Whiteboards are also insignificant with reference to bandwidth impact.

To drill home the point and procedure, let’s try the same exercise with 12 concurrent interactive collaborating Video feeds:

  • Each of our 12 clients is connected to the server and will receive 11 video streams from the server (N-1).
  • Lets calculate first the number of streams outbound: 12 x 11 = 132
  • Lets also consider the 12 publishing streams from each client to the Connect server for a total of 144  total streams to support the Connect Video feeds.
  • Following this larger example, if you go with the settings that I depicted previously in the screen captures above  to support the 12 Video feeds in a single meeting: DSL Bandwith and Standard Video quality = 213 kbps per stream:
  • 144 streams x 213 = 30672 kbps or 31 Mbps (rounded up) for the 12 separate cameras.
  • DSL VoIP = 22 kbps x 144 = 3168 kbs or 3Mbps additional bandwidth needed.
  • 300 kbps for each stream (assuming that the addin will not be ubiquitous) x 144 stream = 43Mbps + 3 Mbps for VoIP = 46Mbps.

Hopefully these exercises help with your planning for large successful meetings. There are other variables to consider such as Screen-Sharing and we will touch on those in a subsequent blog article. Consider, for example that when you are pushing the limits of your network, audio is usually given QoS priority over video.

Note: These examples assume that each client has a separate connection with the server and the Connect Edge servers are not remote to consolidate streams; they are not geographically distributed; they are collocated with the origin servers as is commonly the case so that each of the 12 attendees are receiving 11 subscribed streams from the data center (N-1).

Note also: Regardless of how many cameras you have active, the Connect addin will use one TCP connection over port 1935 (and encrypted on 443) that carries all the streams.

 

Meeting Connection Test Page Fails at Step Two

Issue: Meeting test page fails at step 2; there are many possible causes for this:

1. If it is failing when you attempt to connect through a remote edge but does not fail when you connect directly to an origin, then the most likely culprit is name resolution. Audit your remote Edge name-resolution against these tutorials:

Adobe Connect Edge Server Deployment Options: part 1

Adobe Connect Edge Server Deployment Options: part 2

2. MS Patch MS12-006 (installed on client workstation) has been problematic:

MS Update MS12-006 (http://technet.microsoft.com/en-us/security/bulletin/ms12-006) causes step two of the meeting test page to fail; more details on the Patch regarding TLS/SSL can be found here:  http://support.microsoft.com/kb/2643584

3. Other network and client-side constraints and configuration issues can cause this test to fail. In order to troubleshoot, try the following tests and gather the data from them:

For on-premise customers who are experiencing a failure at step two, try hitting the test page on the Adobe hosted service:

https://my.adobeconnect.com/common/help/en/support/meeting_test.htm

Firefox and IE should look like this:

FFMT.fw

Chrome looks like this; notice the absence of an addin:

ChromeMT.fw

To drill deeper, click on the Send Results link:

FFMTSendResults.fw

And then click on the Details link

FFMTSendResultsDetails.fw

FFMTSendResultsDetailsPV.fw

Note: Currently, as of the writing of this blog entry, in Connect 9.1, there is a known bug that prevents the addin version from appearing in the the meeting test-link output. In the screen capture above, you only see the Flash-player version. This is scheduled, tentatively to be fixed in Connect 9.2.

You may copy and paste the output from the details link to assist the Adobe Customer Care team with diagnosing meeting connection issues. Do not use the Send Results link on the meeting connection test page, but manually copy and paste the results into an email.

On the meeting test page, if you find that you need the addin, (excluding Chrome) simply click on the Downloads link:

FFMTDownloads.fw

Note that the Flash-player download link is also available:

FFMTDownloadsAddinFP.fw

The meeting test link has some limitations, for example, it will not diagnose tunneling RTMP(S) over HTTP(S). To figure out if you are tunneling and thereby experiencing additional latency and connection drops caused by something on the network blocking RTMP, (usually a proxy or firewall), look in the upper left corner of a live meeting room and see if there is a “T” in the output when you click on the green connection indicator:

.tunneljpgOctagon

All of this information is vital when trying to diagnose meeting connection issues.

Look for an upcoming blog article on diagnosing and traversing sources of network blockages of RTMP(S) resulting in tunneling RTMP(S) over HTTP(S) and causing increased latency in a meeting. You should have a lightening fast connection and it is very doable with the right configuration:

no-tunnel-star.fw

Stunnel does not Startup with Connect

Problem: stunnel does not start up with Connect

Although stunnel can be installed as a service, it doesn’t load the stunnel.conf file(!) one workaround is to not setup the services to run automatically but to auto-run these batch files at startup:

Note: This tech-note assumes stunnel is installed in c:\Connect\9.0.0.1\; be sure to adapt the scripts accordingly.

Origin server startup.bat:

@ECHO ON
net start FMS
net start FMSAdmin
net start ConnectPro
net start CPTelephonyService
c:\Connect\9.0.0.1\stunnel\stunnel.exe stunnel.conf
@ECHO OFF Origins stop.bat:

@ECHO ON
net stop ConnectPro
net stop CPTelephonyService
net stop FMSAdmin
net stop FMS /y
@ECHO OFF

If you have remote Edge servers, use these; they includes cache clearing maintenance.

Edges start.bat:

@ECHO ON
net start fms
ping 1.1.1.1 -n 1 -w 10000>nul
net start fmsadmin
c:\breeze\edgeserver\stunnel\stunnel.exe stunnel.conf
@ECHO OFF

Edges stop.bat:

@ECHO ON
net stop fmsadmin
ping 1.1.1.1 -n 1 -w 10000>nul
net stop fms
ping 1.1.1.1 -n 1 -w 20000>nul
del /Q /S c:\breeze\edgeserver\win32\cache\http\*.*
ping 1.1.1.1 -n 1 -w 10000>nul
@ECHO OFF

Run > gpedit.msc
Local Computer Policy > Computer Configuration > Windows Settings > Scripts (Startup/Shutdown)
Batch files are assigned as startup & shutdown scripts. This is in addition to being available to be run manually.

Providing Diagnostic Data to Expedite Solutions for Connect Meeting Issues

Issue: Anything that may happen during a meeting which has a pejorative effect on end-user experience.

Solution: In Connect 9.1 we have a great diagnostic option in the meeting room. You can immediately pull logs from any meeting to diagnose:

If you click Help>About Adobe Connect, while holding down the Ctrl key, the debug logs will appear int he meeting room and you will have the option to copy them to your clipboard.

log-mtg.fw

 

log-mtg1.fw

Sending me these, along with the RTMP string  Help> About Adobe Connect, while holding down the Shift key – this will be most helpful from the client experiencing the extreme latency.

rtmp-mtg.fw

Now if you want to take it even one step further and provide a client-side view of the meeting:

The instructions for enabling client-side logging are here: http://helpx.adobe.com/adobe-connect/kb/enable-logging-acrobat-connect-professional.html

Providing all this data along with the date and time (including timezone) and Meeting URL of any issue, will greatly expedite analysis and solution.