CSO : DATE (20 NOV 2015) – Adobe Connect Access Intermittent for DA2 Datacenter – RESOLVED

UPDATE #4:

As of 1:05 PM ET,  while we are still waiting on Level 3 to confirm on their end that the issue is resolved, further testing by our Technical Operations team was able to confirm that network traffic has returned to normal levels and Adobe Connect services are no longer appearing to be impacted.
We are marking this as resolved and a full incident review will be conducted.  Out of this, a statement will be prepared and delivered.

 

UPDATE #3:

As of 12:50 PM ET, we are still waiting on the service provider.  Level 3 is experiencing major issues across possibly multiple locations although we still do not have confirmation outside of Chicago.

The issue is still intermittent depending on the network route that customers are taking to our Dallas datacenter.  Customers may have packet loss but still get to the Connect account.  They may experience slower load times and performance issues.  Other customers may experience timeouts.   Customers not going through affected areas will not see an issue.
Update to follow.

 

UPDATE #2:

As of 12:00 PM ET, we have identified this being a carrier issue currently in the Chicago area. It appears to be an outage on the Level 3 network where it is affecting customers routing to our Dallas datacenter (DA2) through Chicago.  It may possibly be impacting other routes as well, but currently the teams are waiting on official word from Level 3 on this issue.

 

UPDATE #1:

As of 11:15 AM ET, there is intermittent connectivity to Adobe Connect meetings and the Adobe Connect web application for all clusters in the DA2 datacenter.  This affects accounts on the following clusters:

NA3, NA4, NA5, NA7, NA10, NA11

The teams are currently engaged and investigating this issue.

Update to follow.

 

Connect 9.5 Edge Server Installation Instructions

Note: The upgrade installer described in this article below is deprecated. For instruction on the latest 9.5 installer, please see the following article: 9.5 Connect Edge Proxy Server Full Installer

Connect 9.5 server installation instructions:

  1. Create a folder <Installation_Directory>/950/edgeserver
  2. Download the Edge 9.5 (based on AMS 5) installer
  3. Run the self-extracting .exe file downloaded in step#2 to <Installation_Directory>/950/edgeserver
  4. Refer the following articles for deployment options:
    1. http://www.connectusers.com/tutorials/2011/06/edge_server_deploy/
    2. http://www.connectusers.com/tutorials/2011/06/edge_server_deploy2/
  5. Run <Installation_Directory>/950/edgeserver/win32/vcredist_x64.exe
  6. Run the following commands as administrator:
    1. cd <Installation_Directory>/950/edgeserver/win32AMSAdmin.exe -install
    2. AMSMaster.exe -install
    3. sc start amsadmin
    4. sc start ams
  7. Confirm that services “Adobe Media Administration Server” and “Adobe Media Server (AMS)” are running
  8. If services need to run using specific user credentials, then be sure to set the credentials in service properties and restart the services

Estimating Bandwidth Consumption in Connect Meetings:

Estimating bandwidth consumption in Connect 9.x Meetings:

The following tables may be used for estimating the amount of bandwidth that will be required for a Connect Meeting based on the features used.

Screen Share:

Screen Share Frame Rate Quality Bandwidth
(kbps)
Low 2 65 500
Medium 4 80 800
Standard 6 90 1200
High 8 100 2000

VoIP:

With reference to VoIP, it is independent of any settings. There is no metric comparable to frame rate or quality or bandwidth with VoIP;  a single metric of micRate is used. Currently we use following values:

Number of VoIP users Without UV With UV
<4 22 kHz 22 kHz
>=4 11 kHz 8 kHz

Note: In a simple test during which a song was played via VoIP for a single user, up bandwidth consumed was 44 kbps when the number of VoIP users was < 4.

Webcam video:

Webcam Video Frame Rate Resolution Quality Bandwidth
(kbps)
Normal Widescreen
Low 4 160×120 214×120 70 200
Medium 4 320×240 427×240 70 300
Standard 8 320×240 427×240 70 300
High 20 640×480 854×480 90 600

Login page doesn’t load post 9.5 upgrade

Issue: After upgrading the existing version of Adobe Connect to 9.5, login page doesn’t load

Environment: Adobe Connect License

Description: When we try to open the login page it just gets refreshed or throws an error below in the screenshot:

Login

In the debug.log file you will see the below error:

[10-28 00:26:41,151] http-80-21 (INFO) [*********************nu6qn:142.150.192.244] GET http://[Domain_URL]/admin?domain=Domain_URL
[10-28 00:26:41,157] http-80-21 (INFO) BaseServlet.keepSessionAlive(), extending session cookie
[10-28 00:26:41,163] http-80-21 (WARN) potential cross-site request forgery (CSRF) attack thwarted (user:test {user-id=878994, cookie=breezbreezkvga2ffczirnu6qn, account-id=7, ip=X.X.X.X}, ip:X.X.X.X, uri:/admin, error:tokenFromCookie is missing, tokenGeneratedFromCookie:null, tokenFromRequest:null)
[10-28 00:26:41,170] http-80-21 (INFO) [*********************nu6qn:142.150.192.244] GET http://[Domain_URL]/
[10-28 00:26:41,178] http-80-21 (INFO) [*********************nu6qn:142.150.192.244] DONE total: 8ms, queries: 2ms, conn: 0ms, xsl: 0
[10-28 00:26:41,184] http-80-21 (INFO) [*********************nu6qn:142.150.192.244] GET http://[Domain_URL]/admin?domain=[Domain_URL]
[10-28 00:26:41,192] http-80-21 (WARN) potential cross-site request forgery (CSRF) attack thwarted (user:test {user-id=878994, cookie=breezbreezkvga2ffczirnu6qn, account-id=7, ip=X.X.X.X}, ip:X.X.X.X, uri:/admin, error:tokenFromCookie is missing, tokenGeneratedFromCookie:null, tokenFromRequest:null)

Solution:

The issue has been identified as a bug in 9.5. The bug number is #4072617. Till the time this bug gets fixed we have a workaround in place.

Replace the web.xml file at C:\Connect\9.5.0\appserv\web\WEB-INF with the one that gets installed with the installer. The file that gets installed with the installer is web.xml.install

Take the backup of web.xml.install file and replace it with the original web.xml file.

Note: If the environment has NTLM/SSO enabled, those settings need to be put again in web.xml file

Large Seminar Sessions need to be scheduled 45 minutes before start of Session

In Adobe Connect 9.5 Seminar sessions need to be scheduled 45 minute(s) prior to the start time of large seminars to allow for implementing > 600 participants (license purchased for Webinar 1000 Managers)

9.5-SeminarSession

Instant sessions created from the large seminar room will only allow < 600 participants

9.5SeminarSessions

Previously with 9.4, the expected number of participants had to be selected in order to distinguish between regular Seminar Sessions (up to 600 Participants) and large Seminar Sessions (> 600 Participants)

Regular Sessions

This option is no longer listed with 9.5 since we now have dedicated webinar servers to accommodate large seminar licenses for both standby mode and scheduled sessions.

Attendee Pod in large Seminar appears in Presenter Only Area

In Adobe Connect 9.5, large Seminars (license purchased for Webinar 1000 Managers) are now by default launched on the Webinar Server (both in stand by mode and session scheduled)

Attendee Pod is functional only in “Presenter Only Area

AttendeePodListedEdited
Attendee Pod is NOT functional in “main meeting/seminar room area

AttendeePod

 (For reference only),

rtmps://na12wbn2.adobeconnect.com:443/?rtmp://PACNA12WBN02:8506/meetingas3app/1280557514/1424793651/

9.5 Large Seminar Attendee Pod

 

 

 

 

 

Benefits of this functionality

  • To conserve bandwidth and CPU load
  • Highly recommended when anticipating over 100 participants (drop down listing of participants viewed only by hosts/presenters)

Previously, with the 9.4 release, the Attendee List Pod was functional for large seminars only in the stand by mode (< 600 participants) when launched on FMS (Meeting server) and NOT on the Webinar Server

(For reference only),

rtmps://na12fms1.adobeconnect.com:443/?rtmp://PACNA12APP03:8506/meetingas3app/1280557514/1424426402/

9.4 LargeSeminar

 

CSO : DATE (19 OCT 2015) – Telephony and Meeting Issues in DA2 Datacenter

UPDATE #2:

As of 1:15 PM MDT, all Adobe Connect customers are now able to launch new meetings on all of the impacted clusters. After further investigation it was determined that clusters NA5, NA10, NA11 were not impacted during this incident. The affected clusters were NA3, NA4, and NA7.  This issue has been resolved and the service is currently operating normally. No further impact is anticipated at this time.

The technical teams made some configuration changes on the impacted servers followed by a reboot, which restored the service on the affected clusters.

A full summary of the root cause and recurrence prevention plan will follow.

UPDATE #1:

As of 8:45 AM MDT, it appears that most Adobe Connect clusters are having difficulties connecting to meetings.  This includes the entire DA2 datacenter as well as some clusters in the SJ1 datacenter.

The technical teams are still investigating.

 

 

Issue:

As of 6:10 AM MDT, some customers in North America in the DA2 datacenter  (Clusters: NA3, NA4, NA5, NA7, NA10, NA11) may be unable to start audio calls from their Connect meeting rooms.

Also, customers with an account hosted on our NA4 cluster are having difficulty with slow loading meetings or meetings that are hanging on connecting.

The technical teams are still investigating the cause of the issue.

Updates will follow below.

 

Moving Meeting Rooms to User Seminars Folder

In Adobe Connect 9.5, Meeting Hosts now have the ability to move normal Adobe Connect Meetings into their own User Seminars folder if they are part of a Named Webinar Manager group.

Previously, with the 9.4 release, we added new Seminar licensing called ‘Named Webiner Managers’. This essentially gives users in that new group, the ability to create and manage their own Seminar rooms (under their User Seminars folder).   In Adobe Connect 9.5, if you are a Named Webinar Manager and you are in the Meeting Hosts group and want to move one or more of your Meetings from your My Meetings folder (User Meetings > {username}) or the Shared Meetings folder to your individual User Seminars folder, you can now do this in the UI.

move

You will notice the new link location for User Seminars in the Move functionality.

Patch for Connect Mobile on iOS9

The previously existing workflows for installing Connect Mobile on iOS9 from the browser no longer work. A server-side Connect patch and an updated Connect Mobile application is required:

  • The server-side patch can be applied to any Connect 9 version except 9.5
    • Users on Connect9.5 won’t need the server-side patch
    • There in not currently a patch available for Connect 8.x servers
  • The Connect version, after the server patch is installed will have the following notation:
    • iOS9_Mobile_patch
    • View-able in the version.txt file on the updated Connect server
  • Users with older versions of Mobile application who have not upgraded to iOS9 will not need the patch
  • Adobe Connect hosted account holders need only update the client Mobile application to version 2.4.10 or higher

The Connect Server Patch and Updater page is here and contains the iOS9 patch: http://helpx.adobe.com/adobe-connect/kb/connect-90-patches.html

See the following reference: iOS 9 release date, features and news