Posts in Category "Edge Server"

9.5 Connect Edge Proxy Server Full Installer

The new 9.5 Connect Edge Proxy Server full installation procedure follows.

Note: This article applies to on-premise Connect customers who have purchased Edge Proxy servers and must install or upgrade to Connect version 9.5. The only possible exception to on-premise exclusivity may be for those who are hosted by a managed ISP that supports external Enterprise Proxy Edge Servers (this latter model is uncommon).

Step 1: Download the Connect Edge Server installer from the URL location provided by the Adobe Connect Support or Customer Service team;  extract and install it with local administrative permissions.

edgeintro.fw

edgeextract.fw

The first installation screen option allows language selection among the following:  English, French, German and Japanese. Click OK to proceed or x (in the upper right) to quit.

edge1a.fw

Step 2 displays the Welcome window in your selected language. Click ‘Next’ to proceed or ‘Quit’ to exit.

edge2a.fw

Note: If you have a previous version installed, this pop-up message will display:

edge2a.fw

Step 3 displays the License Agreement: The administrator conducting the installation must accept the agreement to proceed. Click ‘Previous’ to go to the previous panel, ‘Next’ to proceed or ‘Quit’ to exit.

edge3a.fw

Step 4 displays the option to Select Destination: This panel offers a browse button and facilitates choosing an installation directory. Choose the installation destination, click ‘Previous’ to go to the previous panel, ‘Next’ to proceed or ‘Quit’ to exit.

edge4a.fw

Note: If the destination directory for the installation selected in this panel already exists then the below warning will appear. Click ‘Yes’ to continue or ‘No’ to quit.

edge5a.fw

Note: If the target directory does not exist, this screen will display:

edge2b.fw

Step 5 presents Shortcut Creation options: This screen will facilitate creating the shortcuts in the Start menu. Click ‘Previous’ to go to the previous panel, ‘Next’ to proceed or ‘Quit’ to exit.

edge6a.fw

Step 6 presents a Summary: Click ‘Previous’ to go to the previous panel, ‘Next’ to proceed or ‘Quit’ to exit.

edge7a.fw

edge7a.fw

Step 7 presents a progress screen: This will occur when the installation starts. The installer will extract the files and will try to take a backup of any previous installation. During this panel a command prompt will occur if there were initially any edge services installed. It will create a backup in the same location where it originally existed, but will append “_backup” in the directory name. Wait for the processes to complete.

Note: A clean installation is highly recommended rather than any attempt at installing over older versions of the Edge.

edge8a.fw

After the processes are completed, click Next to proceed:

edge9a.fw

Step 8 offers a GUI, Edge Server Setup Configuration: This panel writes the Edge Server FQDN, the Connect server FQDN, the Cluster ID and the server ports into the Edge server configuration.

edge10b.fw

Example entries follow based on the sample deployment diagram below:

Edge Server Hostname: edge.company.com
Connect Server Hostname: connect.company.com
Edge cluster ID: edge.dmz-edge
Connect Server Normal Port: 80
Connect Server Secure Port: 443

 

95EdgeDMZstunnelOriginstunnel

Step 9 presents the Finish Panel: The installation has completed. Click ‘Done’ to finish the installation:

edge11a.fw

Post installation, the Edge config.ini file, based on our example will contain these relevant entries:

FCS_EDGE_HOST=edge.company.com
FCS_EDGE_REGISTER_HOST=connect.company.com
FCS_EDGE_CLUSTER_ID=edge.dmz-edge=1
FCS.HTTPCACHE_BREEZE_SERVER_NORMAL_PORT=connect.company.com:80
FCS.HTTPCACHE_BREEZE_SERVER_SECURE_PORT=connect.company.com:443

Note: Prior versions of the Connect Edge often employed (although never required) a custom.ini file in the Connect Edge server installation root directory for these entries. The custom.ini would then override the config.ini file in the \conf directory. Placing a custom.ini in the root installation directory is still an option as well as a hazard should one contain stale or wrong entries. The new Edge installer writes directly to config.in through the screen illustrated in step 8.

Once the Connect Edge Server is installed, it must be registered with the origin server or cluster for which it serves as proxy:

On the origin server, register the Edge server by adding the Edge server unique name into the host mappings section of the Connect Management Console; the settings will propagate throughout an origin server cluster from any one of the origin servers:

Start > Programs > Adobe Connect Server > Configure Adobe Connect Server

If the Edge is communicating with the origin server, then you will see a preregistration configuration under the server settings tab:

edgereg1.fw

Add the same unique Edge name into the host mapping fields as follows to register the Edge; this is a manual security mechanism to prevent unauthorized pirate Edge server registration:

edgereg2.fw

Note: The common identification variable in the custom.ini on the origin and the config.ini on the Edge is the cluster ID; following our example is dmz-edge=1 indication the first zone by name; add this to the custom.ini file on the origin(s).

Note: Even a single Edge server warrants its own cluster ID.

edge.dmz-edge=1

For additional information on Edge server deployments including maintenance and troubleshooting, see the articles on the Connect Users Community. Note that the custom.ini file is used in these articles to configure the Edge by overriding the config.ini. As aforementioned, while the new 9.5 installer writes directly to the config.ini, the custom.ini, when used as described will override the config.ini.

The first tutorial listed below discusses the reverse proxy use of the Edge and the second discusses the enterprise proxy use:

Adobe Connect Edge Server Deployment Options: part 1
Adobe Connect Edge Server Deployment Options: part 2

New Adobe Connect Support Blog Subscription Option

Now you can stay on top of the new articles and posts by subscribing to the Adobe Connect Support Blog. Simply go to the Adobe Connect Support Blog home page and enter your email address and check off the categories about which you would like to be notified. Click “Subscribe me” and you will begin receiving  regular updates:

subscribe.fw

 

 

Connect 9.5 Edge Server Installation Instructions

Note: The upgrade installer described in this article below is deprecated. For instruction on the latest 9.5 installer, please see the following article: 9.5 Connect Edge Proxy Server Full Installer

Connect 9.5 server installation instructions:

  1. Create a folder <Installation_Directory>/950/edgeserver
  2. Download the Edge 9.5 (based on AMS 5) installer
  3. Run the self-extracting .exe file downloaded in step#2 to <Installation_Directory>/950/edgeserver
  4. Refer the following articles for deployment options:
    1. http://www.connectusers.com/tutorials/2011/06/edge_server_deploy/
    2. http://www.connectusers.com/tutorials/2011/06/edge_server_deploy2/
  5. Run <Installation_Directory>/950/edgeserver/win32/vcredist_x64.exe
  6. Run the following commands as administrator:
    1. cd <Installation_Directory>/950/edgeserver/win32AMSAdmin.exe -install
    2. AMSMaster.exe -install
    3. sc start amsadmin
    4. sc start ams
  7. Confirm that services “Adobe Media Administration Server” and “Adobe Media Server (AMS)” are running
  8. If services need to run using specific user credentials, then be sure to set the credentials in service properties and restart the services

Troubleshooting Verbose Meeting Addin Logging

On occasion it can be difficult to get verbose addin logging to work. The tech-note describing how to set it up is here: Enable logging | Meeting Add-in

The tech-note correctly describes where to place the customized mms.cfg file for use with both 64 bit and 32 bit Windows clients as well as for the Mac OS.

If after following the instructions in the tech-note, you still do not see any verbose addin logs, one possible cause is that there may be an additional mms.cfg file in an alternate location on the client that is blocking the log creation process. To remedy this, add the customized debug mms.cfg to the following locations after renaming any existing mms.cfg files (to allow them to be restored after verbose logging or debugging is complete):

Here are the locations (more than in the tech-note):

  • Windows (32 bit) :

In: C:\Windows\System32\Macromed\Flash\mms.cfg
or C:\Windows\System32\mms.cfg

  • Windows 7 (64 bit):

In: c:\Windows\SysWOW64\Macromed\Flash\mms.cfg
or c:\Windows\SysWOW64\mms.cfg

After placing the mms.cfg in both folders, be sure to close all addin browsers and then to open the addin only in the one Meeting that you wish to troubleshoot.

Tunneling with RTMP encapsulated in HTTP (RTMPT) should be avoided as it causes latency

Tunneling with RTMP encapsulated in HTTP or RTMPT should be avoided as it causes latency that can have a negative impact on user experience in a Connect meeting. In rare circumstances,the latency commensurate with tunneling RTMP encapsulated in HTTP, can become so acute that it renders Connect unusable for affected clients. The performance hit commensurate with tunneling is one of the primary reasons we continue to deploy Connect Edge servers as they often can replace third-party proxy servers that are often the cause of tunneling latency..

While the amount of acceptable latency depends on what one is doing in the room; RTMPT tunneling affects most activities. Some activities, such as screen-sharing are more bandwidth intensive than other activities such as presenting an uploaded PowerPoint from within a meeting room; The high latency commensurate with RTMPT tunneling would affect the former more than the latter. VoIP is often the first thing to make the effects of high latency felt. Here is some feedback from a test I did while on site with a client dealing with tunneling because of their refusal to pipe RTMP around a third-party proxy:.

External user tunneling during test:
Spike at 3.10/3.02 sec
Latency 403/405 ms up to 3.53/3.52 sec up .064 down 118
When latency peaks to 2.6/2.4 sec I get a mild interruption to the audio V
Video pauses momentarily when the latency spikes

Internal user with direct connection:
2 msec / 1 msec Up 0.08 kbits down 127 kbits
No pauses, delays or spikes

Tunneling should only be considered as a fallback mechanism or safety net to allow connections when RTMP is blocked due to something unplanned or for a few remote clients who must negotiate specific network obstacles. When RTMPT is the default by network design, you will need to limit your activities within Connect to those feature that use the least bandwidth.

The picture below shows a direct connection over RTMPS on 443 is being blocked somewhere on the client’s network and the fallback mechanism built into Connect of tunneling RTMP encapsulated in HTTPS is the fallback path. This is usually caused by either proxy servers or firewalls or both – any application-aware appliance on one’s network that sees the RTMPS traffic on 443 and recognizes that it is not HTTPS is a potential obstacle; RTMPS traffic is on port 443 and while it is disguised as HTTPS, it still may be blocked. The result is tunneling, indicated by the “T” in the output:

.tunneljpgOctagon

Compare with a connection without tunneling:

no-tunnel

The recommended steps for anyone experiencing persistent tunneling, is for their network engineers to trust the source IP addresses of the Adobe hosted, ACMS, managed ISP or on-premise Connect/FMS server VIPs in order to prevent the blocking of RTMPS traffic. RTMPS is not supported by any third-party proxy server. Static bypass works well to solve this issue. The problem stems from network policies that require all traffic to go through a proxy. The result is tunneling with commensurate high latency and drops. RTMPS must be allowed to stream around a proxy to avoid the overhead and latency of tunneling encapsulated within HTTPS. Attempts to cache the stream add no value.

Other options will depend on the capabilities of the third-party proxy servers in the affected client infrastructure. Blue Coat ProxySG is one of the popular proxy server options in our niche. In cases of latency invoked by tunneling RTMP encapsulated in HTTP on a network that employs Blue Coat ProxySG servers, sniff tests done by support representatives have indicated that when an affected client attempts to connect to an Adobe Connect meeting, those clients would establish both explicit HTTP connections based on PAC file settings in the system registry to the Blue Coat ProxySG pool through a hardware-based load balancing device (HLD) and transparent HTTP and SSL connections through Blue Coat ProxySG via WCCP GRE redirect to several Adobe Connect servers. The problem manifests with RTMPS when the clients attempt to establish an SSL connection directly to the destination host without going through PAC file proxy settings. Since a Blue Coat ProxySG is commonly configured to perform an SSL intercept on both explicit and transparent HTTPS traffic, upon examining the content after decrypting the SSL payload from the clients, the Blue Coat ProxySG will return an exception and close the connection because the request doesn’t contain an HTTP component and cannot be parsed for policy evaluation. As a workaround, other than using static bypass, it is possible to create a proxy service with the destination set to the Adobe Connect server IP range on port 443 and to set the proxy setting to TCP-Tunnel with Early Intercept enabled. This will allows Blue Coat ProxySG to intercept and tunnel the traffic without considering whether it is RTMPS or HTTPS.

Watch for a more comprehensive article on this topic forthcoming.

Connect Meeting Bandwidth Utilization using Multiple Interactive Collaborating Video Feeds

Usage question: How many Video feeds can I have running in my meeting room at once?

Answer: Let’s consider a working example around the bandwidth utilization of six Video cameras in a single meeting room consisting of one host and five participants. This working example may be that a of an interactive management meeting or of a college classroom where multiple students interact in a small group session using their webcams. From an examination of this example, you will be able to calculate video camera utilization parameters for other meetings whether they be larger or smaller ones.

To help illustrate what I mean, see this picture from our Connect 9.1 Release Notes

six-cams.fw

Each of our six webcam-wielding clients is connected to the server and will receive five video streams from the server (N-1).

Lets calculate first the number of streams outbound: 6 x 5 = 30

Lets also consider the 6 publishing streams from each client to the Connect server for a total of 36  total streams to support the Connect Video pods.

Now lets calculate the amount of bandwidth used by each stream; here you have power to decide how much bandwidth is to be used by each stream as there are many variables that Adobe Connect puts in your control:

In your meeting room, as a host, click Meeting > Preferences:

meeting-preferences.fw

Under Meeting > Preferences, there are two important options that you are going to adjust – Bandwidth and Video:

meeting-preferences-room-bandwidth.fw

meeting-preferences-video.fw

 

The size of the video streams commensurate with each webcam instance will depend on how you configure these settings.

webcam.fw

Following our example, if you go with the settings that I have depicted in the screen captures above to support the 6 Video feeds in a single meeting: DSL Bandwith and Standard Video quality = 213 kbps per stream:

36 streams x 213 = 7668 kbps or 8 Mbps for the 6 separate cameras.

There are other variables to consider as well. Building on our example, let’s say you also want to use VoIP:

VoIP.fw

DSL VoIP = 22 kbs x 36 = 792 kbs or 1Mbps (rounded up) additional bandwidth needed.

There are other ways to optimize: the video streams are always larger when clients use the Flashplayer in a browser rather than using the Connect Meeting addin. The Connect addin uses the ON2 codec and is far more economical when it comes to bandwidth utilization. For each client running without the Connect addin it is prudent to plan for an additional 50% for each of their Video streams. To avoid this additional bandwidth consumption, send out a link with the Adobe Connect Addin prior to your meeting and encourage clients to install it. It is a small modified version of the Flashplayer:

Another variable to consider that when the Video instance sizes are smaller, Connect adjusts to a lower publishing resolution to save some bandwidth. Unless you are sure the clients have the addin, the final planning number for our 12 webcam meeting is:

300 kbps for each stream (assuming that the addin will not be ubiquitous) x 36 stream = 11 Mbps + 1 Mbps for VoIP = 12Mbps.

Presenting a PowerPoint or a PDF that is uploaded to the Meeting room does not add to the overhead. Chat, Notes and Whiteboards are also insignificant with reference to bandwidth impact.

To drill home the point and procedure, let’s try the same exercise with 12 concurrent interactive collaborating Video feeds:

  • Each of our 12 clients is connected to the server and will receive 11 video streams from the server (N-1).
  • Lets calculate first the number of streams outbound: 12 x 11 = 132
  • Lets also consider the 12 publishing streams from each client to the Connect server for a total of 144  total streams to support the Connect Video feeds.
  • Following this larger example, if you go with the settings that I depicted previously in the screen captures above  to support the 12 Video feeds in a single meeting: DSL Bandwith and Standard Video quality = 213 kbps per stream:
  • 144 streams x 213 = 30672 kbps or 31 Mbps (rounded up) for the 12 separate cameras.
  • DSL VoIP = 22 kbps x 144 = 3168 kbs or 3Mbps additional bandwidth needed.
  • 300 kbps for each stream (assuming that the addin will not be ubiquitous) x 144 stream = 43Mbps + 3 Mbps for VoIP = 46Mbps.

Hopefully these exercises help with your planning for large successful meetings. There are other variables to consider such as Screen-Sharing and we will touch on those in a subsequent blog article. Consider, for example that when you are pushing the limits of your network, audio is usually given QoS priority over video.

Note: These examples assume that each client has a separate connection with the server and the Connect Edge servers are not remote to consolidate streams; they are not geographically distributed; they are collocated with the origin servers as is commonly the case so that each of the 12 attendees are receiving 11 subscribed streams from the data center (N-1).

Note also: Regardless of how many cameras you have active, the Connect addin will use one TCP connection over port 1935 (and encrypted on 443) that carries all the streams.

 

Meeting Connection Test Page Fails at Step Two

Issue: Meeting test page fails at step 2; there are many possible causes for this:

1. If it is failing when you attempt to connect through a remote edge but does not fail when you connect directly to an origin, then the most likely culprit is name resolution. Audit your remote Edge name-resolution against these tutorials:

Adobe Connect Edge Server Deployment Options: part 1

Adobe Connect Edge Server Deployment Options: part 2

2. MS Patch MS12-006 (installed on client workstation) has been problematic:

MS Update MS12-006 (http://technet.microsoft.com/en-us/security/bulletin/ms12-006) causes step two of the meeting test page to fail; more details on the Patch regarding TLS/SSL can be found here:  http://support.microsoft.com/kb/2643584

3. Other network and client-side constraints and configuration issues can cause this test to fail. In order to troubleshoot, try the following tests and gather the data from them:

For on-premise customers who are experiencing a failure at step two, try hitting the test page on the Adobe hosted service:

https://my.adobeconnect.com/common/help/en/support/meeting_test.htm

Firefox and IE should look like this:

FFMT.fw

Chrome looks like this; notice the absence of an addin:

ChromeMT.fw

To drill deeper, click on the Send Results link:

FFMTSendResults.fw

And then click on the Details link

FFMTSendResultsDetails.fw

FFMTSendResultsDetailsPV.fw

Note: Currently, as of the writing of this blog entry, in Connect 9.1, there is a known bug that prevents the addin version from appearing in the the meeting test-link output. In the screen capture above, you only see the Flash-player version. This is scheduled, tentatively to be fixed in Connect 9.2.

You may copy and paste the output from the details link to assist the Adobe Customer Care team with diagnosing meeting connection issues. Do not use the Send Results link on the meeting connection test page, but manually copy and paste the results into an email.

On the meeting test page, if you find that you need the addin, (excluding Chrome) simply click on the Downloads link:

FFMTDownloads.fw

Note that the Flash-player download link is also available:

FFMTDownloadsAddinFP.fw

The meeting test link has some limitations, for example, it will not diagnose tunneling RTMP(S) over HTTP(S). To figure out if you are tunneling and thereby experiencing additional latency and connection drops caused by something on the network blocking RTMP, (usually a proxy or firewall), look in the upper left corner of a live meeting room and see if there is a “T” in the output when you click on the green connection indicator:

.tunneljpgOctagon

All of this information is vital when trying to diagnose meeting connection issues.

Look for an upcoming blog article on diagnosing and traversing sources of network blockages of RTMP(S) resulting in tunneling RTMP(S) over HTTP(S) and causing increased latency in a meeting. You should have a lightening fast connection and it is very doable with the right configuration:

no-tunnel-star.fw

Stunnel does not Startup with Connect

Problem: stunnel does not start up with Connect

Although stunnel can be installed as a service, it doesn’t load the stunnel.conf file(!) one workaround is to not setup the services to run automatically but to auto-run these batch files at startup:

Note: This tech-note assumes stunnel is installed in c:\Connect\9.0.0.1\; be sure to adapt the scripts accordingly.

Origin server startup.bat:

@ECHO ON
net start FMS
net start FMSAdmin
net start ConnectPro
net start CPTelephonyService
c:\Connect\9.0.0.1\stunnel\stunnel.exe stunnel.conf
@ECHO OFF Origins stop.bat:

@ECHO ON
net stop ConnectPro
net stop CPTelephonyService
net stop FMSAdmin
net stop FMS /y
@ECHO OFF

If you have remote Edge servers, use these; they includes cache clearing maintenance.

Edges start.bat:

@ECHO ON
net start fms
ping 1.1.1.1 -n 1 -w 10000>nul
net start fmsadmin
c:\breeze\edgeserver\stunnel\stunnel.exe stunnel.conf
@ECHO OFF

Edges stop.bat:

@ECHO ON
net stop fmsadmin
ping 1.1.1.1 -n 1 -w 10000>nul
net stop fms
ping 1.1.1.1 -n 1 -w 20000>nul
del /Q /S c:\breeze\edgeserver\win32\cache\http\*.*
ping 1.1.1.1 -n 1 -w 10000>nul
@ECHO OFF

Run > gpedit.msc
Local Computer Policy > Computer Configuration > Windows Settings > Scripts (Startup/Shutdown)
Batch files are assigned as startup & shutdown scripts. This is in addition to being available to be run manually.

Providing Diagnostic Data to Expedite Solutions for Connect Meeting Issues

Issue: Anything that may happen during a meeting which has a pejorative effect on end-user experience.

Solution: In Connect 9.1 we have a great diagnostic option in the meeting room. You can immediately pull logs from any meeting to diagnose:

If you click Help>About Adobe Connect, while holding down the Ctrl key, the debug logs will appear int he meeting room and you will have the option to copy them to your clipboard.

log-mtg.fw

 

log-mtg1.fw

Sending me these, along with the RTMP string  Help> About Adobe Connect, while holding down the Shift key – this will be most helpful from the client experiencing the extreme latency.

rtmp-mtg.fw

Now if you want to take it even one step further and provide a client-side view of the meeting:

The instructions for enabling client-side logging are here: http://helpx.adobe.com/adobe-connect/kb/enable-logging-acrobat-connect-professional.html

Providing all this data along with the date and time (including timezone) and Meeting URL of any issue, will greatly expedite analysis and solution.

The Adobe Connect Deployment Guide on the F5 Website needs Updating

Issue: Be careful when following the Adobe Connect Deployment Guide posted on the F5 Website. While the article is be helpful, there are some ambiguities that can lead to trouble. I have tried to update their deployment guide but have not succeeded; the LTM is the most popular load-balancing device and SSL accelerator in the Connect niche and when it is set up properly it works splendidly. Here are corrections, updates and things to watch out for when deploying Connect behind an LTM:

1. Do not use an HTTP profile for an RTMP VIP. An HTTP profile for RTMP VIPs may affect playback of video as well as break remote Edge connectivity. Remember that you have two servers running on each box, a Tomcat application server and an FMS server. Do not treat the FMS server as though it were an application server; RTMP is a streaming protocol that requires a TCP profile at the HLD VIP:

f5.fw

2. Use the health monitor documented here for LTM.

3. Do not use session-awareness or stick-sessions even if you use SSL. The Round Robin algorithm should float freely to the Tomcat application pool.

4. Do not use Nagle’s Algorithm with SSL; it will have a negative effect on performance.

Review this general Connect pool/cluster configuration tutorial before configuring BIG-IP LTM with Connect: Adobe® Connect™ server pools/clusters and hardware-based load-balancing devices with SSL acceleration