Posts in Category "Edge Server"

Tunneling with RTMP encapsulated in HTTP (RTMPT) should be avoided as it causes latency

Tunneling with RTMP encapsulated in HTTP or RTMPT should be avoided as it causes latency that can have a negative impact on user experience in a Connect meeting. In rare circumstances,the latency commensurate with tunneling RTMP encapsulated in HTTP, can become so acute that it renders Connect unusable for affected clients. The performance hit commensurate with tunneling is one of the primary reasons we continue to deploy Connect Edge servers as they often can replace third-party proxy servers that are often the cause of tunneling latency..

While the amount of acceptable latency depends on what one is doing in the room; RTMPT tunneling affects most activities. Some activities, such as screen-sharing are more bandwidth intensive than other activities such as presenting an uploaded PowerPoint from within a meeting room; The high latency commensurate with RTMPT tunneling would affect the former more than the latter. VoIP is often the first thing to make the effects of high latency felt. Here is some feedback from a test I did while on site with a client dealing with tunneling because of their refusal to pipe RTMP around a third-party proxy:.

External user tunneling during test:
Spike at 3.10/3.02 sec
Latency 403/405 ms up to 3.53/3.52 sec up .064 down 118
When latency peaks to 2.6/2.4 sec I get a mild interruption to the audio V
Video pauses momentarily when the latency spikes

Internal user with direct connection:
2 msec / 1 msec Up 0.08 kbits down 127 kbits
No pauses, delays or spikes

Tunneling should only be considered as a fallback mechanism or safety net to allow connections when RTMP is blocked due to something unplanned or for a few remote clients who must negotiate specific network obstacles. When RTMPT is the default by network design, you will need to limit your activities within Connect to those feature that use the least bandwidth.

The picture below shows a direct connection over RTMPS on 443 is being blocked somewhere on the client’s network and the fallback mechanism built into Connect of tunneling RTMP encapsulated in HTTPS is the fallback path. This is usually caused by either proxy servers or firewalls or both – any application-aware appliance on one’s network that sees the RTMPS traffic on 443 and recognizes that it is not HTTPS is a potential obstacle; RTMPS traffic is on port 443 and while it is disguised as HTTPS, it still may be blocked. The result is tunneling, indicated by the “T” in the output:

.tunneljpgOctagon

Compare with a connection without tunneling:

no-tunnel

The recommended steps for anyone experiencing persistent tunneling, is for their network engineers to trust the source IP addresses of the Adobe hosted, ACMS, managed ISP or on-premise Connect/FMS server VIPs in order to prevent the blocking of RTMPS traffic. RTMPS is not supported by any third-party proxy server. Static bypass works well to solve this issue. The problem stems from network policies that require all traffic to go through a proxy. The result is tunneling with commensurate high latency and drops. RTMPS must be allowed to stream around a proxy to avoid the overhead and latency of tunneling encapsulated within HTTPS. Attempts to cache the stream add no value.

Other options will depend on the capabilities of the third-party proxy servers in the affected client infrastructure. Blue Coat ProxySG is one of the popular proxy server options in our niche. In cases of latency invoked by tunneling RTMP encapsulated in HTTP on a network that employs Blue Coat ProxySG servers, sniff tests done by support representatives have indicated that when an affected client attempts to connect to an Adobe Connect meeting, those clients would establish both explicit HTTP connections based on PAC file settings in the system registry to the Blue Coat ProxySG pool through a hardware-based load balancing device (HLD) and transparent HTTP and SSL connections through Blue Coat ProxySG via WCCP GRE redirect to several Adobe Connect servers. The problem manifests with RTMPS when the clients attempt to establish an SSL connection directly to the destination host without going through PAC file proxy settings. Since a Blue Coat ProxySG is commonly configured to perform an SSL intercept on both explicit and transparent HTTPS traffic, upon examining the content after decrypting the SSL payload from the clients, the Blue Coat ProxySG will return an exception and close the connection because the request doesn’t contain an HTTP component and cannot be parsed for policy evaluation. As a workaround, other than using static bypass, it is possible to create a proxy service with the destination set to the Adobe Connect server IP range on port 443 and to set the proxy setting to TCP-Tunnel with Early Intercept enabled. This will allows Blue Coat ProxySG to intercept and tunnel the traffic without considering whether it is RTMPS or HTTPS.

Watch for a more comprehensive article on this topic forthcoming.

Connect Meeting Bandwidth Utilization using Multiple Interactive Collaborating Video Feeds

Usage question: How many Video feeds can I have running in my meeting room at once?

Answer: Let’s consider a working example around the bandwidth utilization of six Video cameras in a single meeting room consisting of one host and five participants. This working example may be that a of an interactive management meeting or of a college classroom where multiple students interact in a small group session using their webcams. From an examination of this example, you will be able to calculate video camera utilization parameters for other meetings whether they be larger or smaller ones.

To help illustrate what I mean, see this picture from our Connect 9.1 Release Notes

six-cams.fw

Each of our six webcam-wielding clients is connected to the server and will receive five video streams from the server (N-1).

Lets calculate first the number of streams outbound: 6 x 5 = 30

Lets also consider the 6 publishing streams from each client to the Connect server for a total of 36  total streams to support the Connect Video pods.

Now lets calculate the amount of bandwidth used by each stream; here you have power to decide how much bandwidth is to be used by each stream as there are many variables that Adobe Connect puts in your control:

In your meeting room, as a host, click Meeting > Preferences:

meeting-preferences.fw

Under Meeting > Preferences, there are two important options that you are going to adjust – Bandwidth and Video:

meeting-preferences-room-bandwidth.fw

meeting-preferences-video.fw

 

The size of the video streams commensurate with each webcam instance will depend on how you configure these settings.

webcam.fw

Following our example, if you go with the settings that I have depicted in the screen captures above to support the 6 Video feeds in a single meeting: DSL Bandwith and Standard Video quality = 213 kbps per stream:

36 streams x 213 = 7668 kbps or 8 Mbps for the 6 separate cameras.

There are other variables to consider as well. Building on our example, let’s say you also want to use VoIP:

VoIP.fw

DSL VoIP = 22 kbs x 36 = 792 kbs or 1Mbps (rounded up) additional bandwidth needed.

There are other ways to optimize: the video streams are always larger when clients use the Flashplayer in a browser rather than using the Connect Meeting addin. The Connect addin uses the ON2 codec and is far more economical when it comes to bandwidth utilization. For each client running without the Connect addin it is prudent to plan for an additional 50% for each of their Video streams. To avoid this additional bandwidth consumption, send out a link with the Adobe Connect Addin prior to your meeting and encourage clients to install it. It is a small modified version of the Flashplayer:

Another variable to consider that when the Video instance sizes are smaller, Connect adjusts to a lower publishing resolution to save some bandwidth. Unless you are sure the clients have the addin, the final planning number for our 12 webcam meeting is:

300 kbps for each stream (assuming that the addin will not be ubiquitous) x 36 stream = 11 Mbps + 1 Mbps for VoIP = 12Mbps.

Presenting a PowerPoint or a PDF that is uploaded to the Meeting room does not add to the overhead. Chat, Notes and Whiteboards are also insignificant with reference to bandwidth impact.

To drill home the point and procedure, let’s try the same exercise with 12 concurrent interactive collaborating Video feeds:

  • Each of our 12 clients is connected to the server and will receive 11 video streams from the server (N-1).
  • Lets calculate first the number of streams outbound: 12 x 11 = 132
  • Lets also consider the 12 publishing streams from each client to the Connect server for a total of 144  total streams to support the Connect Video feeds.
  • Following this larger example, if you go with the settings that I depicted previously in the screen captures above  to support the 12 Video feeds in a single meeting: DSL Bandwith and Standard Video quality = 213 kbps per stream:
  • 144 streams x 213 = 30672 kbps or 31 Mbps (rounded up) for the 12 separate cameras.
  • DSL VoIP = 22 kbps x 144 = 3168 kbs or 3Mbps additional bandwidth needed.
  • 300 kbps for each stream (assuming that the addin will not be ubiquitous) x 144 stream = 43Mbps + 3 Mbps for VoIP = 46Mbps.

Hopefully these exercises help with your planning for large successful meetings. There are other variables to consider such as Screen-Sharing and we will touch on those in a subsequent blog article. Consider, for example that when you are pushing the limits of your network, audio is usually given QoS priority over video.

Note: These examples assume that each client has a separate connection with the server and the Connect Edge servers are not remote to consolidate streams; they are not geographically distributed; they are collocated with the origin servers as is commonly the case so that each of the 12 attendees are receiving 11 subscribed streams from the data center (N-1).

Meeting Connection Test Page Fails at Step Two

Issue: Meeting test page fails at step 2; there are many possible causes for this:

1. If it is failing when you attempt to connect through a remote edge but does not fail when you connect directly to an origin, then the most likely culprit is name resolution. Audit your remote Edge name-resolution against these tutorials:

Adobe Connect Edge Server Deployment Options: part 1

Adobe Connect Edge Server Deployment Options: part 2

2. MS Patch MS12-006 (installed on client workstation) has been problematic:

MS Update MS12-006 (http://technet.microsoft.com/en-us/security/bulletin/ms12-006) causes step two of the meeting test page to fail; more details on the Patch regarding TLS/SSL can be found here:  http://support.microsoft.com/kb/2643584

3. Other network and client-side constraints and configuration issues can cause this test to fail. In order to troubleshoot, try the following tests and gather the data from them:

For on-premise customers who are experiencing a failure at step two, try hitting the test page on the Adobe hosted service:

https://my.adobeconnect.com/common/help/en/support/meeting_test.htm

Firefox and IE should look like this:

FFMT.fw

Chrome looks like this; notice the absence of an addin:

ChromeMT.fw

To drill deeper, click on the Send Results link:

FFMTSendResults.fw

And then click on the Details link

FFMTSendResultsDetails.fw

FFMTSendResultsDetailsPV.fw

Note: Currently, as of the writing of this blog entry, in Connect 9.1, there is a known bug that prevents the addin version from appearing in the the meeting test-link output. In the screen capture above, you only see the Flash-player version. This is scheduled, tentatively to be fixed in Connect 9.2.

You may copy and paste the output from the details link to assist the Adobe Customer Care team with diagnosing meeting connection issues. Do not use the Send Results link on the meeting connection test page, but manually copy and paste the results into an email.

On the meeting test page, if you find that you need the addin, (excluding Chrome) simply click on the Downloads link:

FFMTDownloads.fw

Note that the Flash-player download link is also available:

FFMTDownloadsAddinFP.fw

The meeting test link has some limitations, for example, it will not diagnose tunneling RTMP(S) over HTTP(S). To figure out if you are tunneling and thereby experiencing additional latency and connection drops caused by something on the network blocking RTMP, (usually a proxy or firewall), look in the upper left corner of a live meeting room and see if there is a “T” in the output when you click on the green connection indicator:

.tunneljpgOctagon

All of this information is vital when trying to diagnose meeting connection issues.

Look for an upcoming blog article on diagnosing and traversing sources of network blockages of RTMP(S) resulting in tunneling RTMP(S) over HTTP(S) and causing increased latency in a meeting. You should have a lightening fast connection and it is very doable with the right configuration:

no-tunnel-star.fw

Stunnel does not Startup with Connect

Problem: stunnel does not start up with Connect

Although stunnel can be installed as a service, it doesn’t load the stunnel.conf file(!) one workaround is to not setup the services to run automatically but to auto-run these batch files at startup:

Note: This tech-note assumes stunnel is installed in c:\Connect\9.0.0.1\; be sure to adapt the scripts accordingly.

Origin server startup.bat:

@ECHO ON
net start FMS
net start FMSAdmin
net start ConnectPro
net start CPTelephonyService
c:\Connect\9.0.0.1\stunnel\stunnel.exe stunnel.conf
@ECHO OFF Origins stop.bat:

@ECHO ON
net stop ConnectPro
net stop CPTelephonyService
net stop FMSAdmin
net stop FMS /y
@ECHO OFF

If you have remote Edge servers, use these; they includes cache clearing maintenance.

Edges start.bat:

@ECHO ON
net start fms
ping 1.1.1.1 -n 1 -w 10000>nul
net start fmsadmin
c:\breeze\edgeserver\stunnel\stunnel.exe stunnel.conf
@ECHO OFF

Edges stop.bat:

@ECHO ON
net stop fmsadmin
ping 1.1.1.1 -n 1 -w 10000>nul
net stop fms
ping 1.1.1.1 -n 1 -w 20000>nul
del /Q /S c:\breeze\edgeserver\win32\cache\http\*.*
ping 1.1.1.1 -n 1 -w 10000>nul
@ECHO OFF

Run > gpedit.msc
Local Computer Policy > Computer Configuration > Windows Settings > Scripts (Startup/Shutdown)
Batch files are assigned as startup & shutdown scripts. This is in addition to being available to be run manually.

Providing Diagnostic Data to Expedite Solutions for Connect Meeting Issues

Issue: Anything that may happen during a meeting which has a pejorative effect on end-user experience.

Solution: In Connect 9.1 we have a great diagnostic option in the meeting room. You can immediately pull logs from any meeting to diagnose:

If you click Help>About Adobe Connect, while holding down the Ctrl key, the debug logs will appear int he meeting room and you will have the option to copy them to your clipboard.

log-mtg.fw

 

log-mtg1.fw

Sending me these, along with the RTMP string  Help> About Adobe Connect, while holding down the Shift key – this will be most helpful from the client experiencing the extreme latency.

rtmp-mtg.fw

Now if you want to take it even one step further and provide a client-side view of the meeting:

The instructions for enabling client-side logging are here: http://helpx.adobe.com/adobe-connect/kb/enable-logging-acrobat-connect-professional.html

Providing all this data along with the date and time (including timezone) and Meeting URL of any issue, will greatly expedite analysis and solution.

The Adobe Connect Deployment Guide on the F5 Website needs Updating

Issue: Be careful when following the Adobe Connect Deployment Guide posted on the F5 Website. While the article is be helpful, there are some ambiguities that can lead to trouble. I have tried to update their deployment guide but have not succeeded; the LTM is the most popular load-balancing device and SSL accelerator in the Connect niche and when it is set up properly it works splendidly. Here are corrections, updates and things to watch out for when deploying Connect behind an LTM:

1. Do not use an HTTP profile for an RTMP VIP. An HTTP profile for RTMP VIPs may affect playback of video as well as break remote Edge connectivity. Remember that you have two servers running on each box, a Tomcat application server and an FMS server. Do not treat the FMS server as though it were an application server; RTMP is a streaming protocol that requires a TCP profile at the HLD VIP.

2. Use the health monitor documented here for LTM.

3. Do not use session-awareness or stick-sessions even if you use SSL. The Round Robin algorithm should float freely to the Tomcat application pool.

4. Do not use Nagle’s Algorithm with SSL; it will have a negative effect on performance.

Review this general Connect pool/cluster configuration tutorial before configuring BIG-IP LTM with Connect: Adobe® Connect™ server pools/clusters and hardware-based load-balancing devices with SSL acceleration

Adobe Connect Edge Server Deployment Options: part 2

This article focuses on Enterprise Proxy Connect Edge deployments and troubleshooting: http://www.connectusers.com/tutorials/2011/06/edge_server_deploy2/index.php

Adobe Connect Edge Server Deployment Options: part 1

This article focuses on reverse proxy Connect Edge deployments: http://www.connectusers.com/tutorials/2011/06/edge_server_deploy/index.php

Enable Client-side Connect Meeting Addin Debug Logging to Diagnose Meeting Connection Problems

To facilitate troubleshooting of Connect Meeting connection issues, enable client-side Connect Meeting Addin Debug Logging according to the following tech-note: http://helpx.adobe.com/adobe-connect/kb/enable-logging-acrobat-connect-professional.html

Clustering Connect Servers with Multiple Availability Zones

Challenge: When a Connect cluster/pool is split into two data centers it is prudent for backup meeting rooms to be created in an alternate datacenter; you do not want the backup and the primary meeting rooms hosted in the same datacenter.

Connect Edge servers fill the gap of extending resources out to remote offices in enterprise proxy mode or of splitting external client traffic from internal traffic in reverse proxy mode; there is not a federated option to allow an origin server cluster to be spread about geographically. Very little latency is tolerable between Connect origin servers. With that said, there is the possibility of splitting an origin cluster across two datacenters for additional redundancy if the latency between the two datacenters is consistently 3 milliseconds or less. Intra-cluster communication on ports 8507 and 8506 must be unhampered for a cluster to work properly.

Spreading a cluster across two local datacenters with less than 3ms of latency requires that all servers point to the same database. Basically I am describing a regular cluster except with half of the servers in one building and the other half in another building nearby. When you do this, it is prudent to make sure that the backup meeting rooms that are spawned in support of every primary active meeting room are always created on servers in the separate datacenter from the active primary meeting. You can do this in the Connect database in PPS_ENUM_DATA_HOSTS. Configure the hosts in one data center with one pool_id, and give the other servers in the second datacenter another pool_id. Then add this setting to the custom.ini on each origin server:

APP_SERVER_POOLING=true

This setting is necessary to configure failover to take advantage of multiple availability zones.