Posts in Category "Database"

Providing Diagnostic Data to Expedite Solutions for Connect Meeting Issues

Issue: Anything that may happen during a meeting which has a pejorative effect on end-user experience.

Solution: In Connect 9.1 we have a great diagnostic option in the meeting room. You can immediately pull logs from any meeting to diagnose:

If you click Help>About Adobe Connect, while holding down the Ctrl key, the debug logs will appear int he meeting room and you will have the option to copy them to your clipboard.

log-mtg.fw

 

log-mtg1.fw

Sending me these, along with the RTMP string  Help> About Adobe Connect, while holding down the Shift key – this will be most helpful from the client experiencing the extreme latency.

rtmp-mtg.fw

Now if you want to take it even one step further and provide a client-side view of the meeting:

The instructions for enabling client-side logging are here: http://helpx.adobe.com/adobe-connect/kb/enable-logging-acrobat-connect-professional.html

Providing all this data along with the date and time (including timezone) and Meeting URL of any issue, will greatly expedite analysis and solution.

Resource Constraints cause Connection Read Error in Logs on Clustered Connect Servers

Issue: FCSj_IO:4 (x) – Connection read error: -1 LP: 5345 RP: 8506 URI: rtmp://localhost:8506/meetingapp/7/12345678

I have seen that in some VMWare environments that are very overtaxed for resources, latency between/among the clustered Connect servers on ports 8507 (and also 8506 though 8506 does not cause this error), can cause problems. Intra-cluster latency should never exceed 2-3ms. When it does we see intermittent errors and can also see crashes.

I had one unnamed customer who had a particularly weak infrastructure and  I could predict his crashes; he was doing back-ups and running other tasks at a certain time weekly that would severely hamper network connectivity for about an hour; these tasks were so all-consuming on the network, they turned every Connect cluster resource into an individual asset on its own island. The Connect logs bore this out and we knew with precision what was going on and could predict his call or email based on his maintenance schedule. He knew he needed to upgrade his infrastructure and in the meantime we worked out a reaction plan to deal with the issue; it included:

  • Place a higher than normal percentage of cache on each server to limit invoking shared storage during maintenance (see page 57)
  • Set the JDBC driver reconnection string for Database connectivity robustness
  • Plan heavy Connect usage around network and server maintenance activities and when possible, do your Connect server maintenance activities at the same time as well.

Brad’s Short-list for Connect Cluster SaaS Monitoring Options

There are many options on the monitoring theme that are worth considering when trying to decide how to keep trach of Connect server resources in a cluster. Articles describing clustered environments are on the Connect Users Community : http://www.connectusers.com  Simply search the User’s Community using the keywords: cluster, pool, edge, SSL, etc.

To effectively monitor your Connect cluster SaaS options can sometimes be cost effective than home-spun solutions; here are some staff picks with some commentary:

Sumologic- It resembles Splunk. The main difference is that Sumologic is hosted and managed externally and Splunk is hosted and managed on-premise. With Sumologic, there is not any need for software licensing, hardware investments or internal administrator expertise.  Splunk offers a similar service called splunk>storm, but it is not as mature as Sumologic and lacks some of the alerting capability found in Sumologic.

Loggly - An alternative to Sumologic could be Loggly which offers a similar service; it seems that the alerting service is not exactly built in.  It requires a little more work and is called AlertBirds.

Note: It is possible to take an on-premise option like Cacti and port it to Sumologic, so you could effectively kill 2 birds with one stone.  You can setup a forwarder in 30 seconds and be searching the logs in no time at all.

Monitis – Provides capabilities similar to those of Nagios along with external monitoring.  The Monitis community writes custom monitors thereby enriching the options.

LogicMonitor – An alternative to Monitis could be something like LogicMonitor.  You may be able to port your existing Nagios checks over to it (check and verify).  This si a simple solution, installing the monitor and having basic checks like CPU, Memory, Bandwidth, Disk Usage, Disk IO and external ping, http, https and udp monitors setup would take all of 20 minutes.

Pingdom- An alternative to RedAlert at a lesser cost.  It is trusted by millions and is easy to use and has more endpoints than comparable options.  It takes five minute setup.

The beauty of a SaaS monitoring solution is that you do not need to worry about scaling your monitoring solution every time you scale your Connect architecture.  You can have a single solution for 20 Connect Clusters vs having to add Cacti servers, Nagios servers, Splunk architecture and licensing to handle the additional monitoring needs commensurate with expansion.  With a SaaS solution, there in not any build-out time.  You can literally have 20 monitors up and running in under an hour, and work on adding additional ones at your leisure in between casts with your new Deceiver 8 Fly Combo.

With reference to basic on-premise monitoring, make sure you use standard perfmon counters for things like CPU, Memory etc. For meeting count and meeting user monitoring you may use the FMSAdmin API with scripts to make various calls and then parse the data and pass it to an option such as Cacti.  To insure robustness, the FMSAdmin service should be restarted routinely. You could also use similar counters to pull data directly from the Connect database, but this is not without risk as Connect updaters and upgrades can introduce changes that may require rework of your custom counters.

Adobe Connect Servers and Hardware-based Load-balancing Devices

This updated article offers a best-practice configuration of a basic Connect pool/cluster behind a high-end, application-aware HLD such as F5 BIG-IP LTM. This article does not discuss SSL-acceleration. This article does not describe all the possible configurations, but offers a general working example of a basic HTML/RTMP non-SSL cluster/pool of Connect servers.

Adobe Connect Server Pools and Hardware-based Load-balancing Devices: http://www.connectusers.com/tutorials/2009/04/load_balancing/index.php

Adobe Connect server pools/clusters and hardware-based load-balancing devices with SSL acceleration

The most robust means of implementing secure socket layer (SSL) with Adobe Connect servers is through a hardware-based SSL accelerator and similarly, the most robust means of clustering Connect servers is with a hardware-based load-balancing device (HLD). Since all enterprise-class HLDs are also SSL accelerators (any that are not both are either legacy or low-end by definition), this example-based article offers a best-practice configuration of a Connect Server pool or cluster running Connect Meetings, Adobe Presenter on-demand content, Adobe Connect Training, Curriculum and Virtual Classrooms securely behind a high-end, application-aware HLD and SSL acceleration device such as F5 BIG-IP LTM. This article does not exhaust the possible configurations, but offers a general working example.

The full tech-note is published here:  http://www.connectusers.com/tutorials/2011/09/connect_servers_ssl_acceleration/index.php

 

Clustering Connect Servers with Multiple Availability Zones

Challenge: When a Connect cluster/pool is split into two data centers it is prudent for backup meeting rooms to be created in an alternate datacenter; you do not want the backup and the primary meeting rooms hosted in the same datacenter.

Connect Edge servers fill the gap of extending resources out to remote offices in enterprise proxy mode or of splitting external client traffic from internal traffic in reverse proxy mode; there is not a federated option to allow an origin server cluster to be spread about geographically. Very little latency is tolerable between Connect origin servers. With that said, there is the possibility of splitting an origin cluster across two datacenters for additional redundancy if the latency between the two datacenters is consistently 3 milliseconds or less. Intra-cluster communication on ports 8507 and 8506 must be unhampered for a cluster to work properly.

Spreading a cluster across two local datacenters with less than 3ms of latency requires that all servers point to the same database. Basically I am describing a regular cluster except with half of the servers in one building and the other half in another building nearby. When you do this, it is prudent to make sure that the backup meeting rooms that are spawned in support of every primary active meeting room are always created on servers in the separate datacenter from the active primary meeting. You can do this in the Connect database in PPS_ENUM_DATA_HOSTS. Configure the hosts in one data center with one pool_id, and give the other servers in the second datacenter another pool_id. Then add this setting to the custom.ini on each origin server:

APP_SERVER_POOLING=true

This setting is necessary to configure failover to take advantage of multiple availability zones.

Java Exception Error while importing database in to the new server

Problem:

Java Exception Error while importing Database in to the new server

Kind of Environment: Licensed

Reason:

If you see this in debug logs:

05-26 01:40:50] console-8510-1 (ERROR) Exception thrown java.lang.NullPointerException at com.macromedia.airspeed.FeatureManager.updateFeatureSet(FeatureManager.java:172) at com.macromedia.airspeed.FeatureManager.isDisabled(FeatureManager.java:78)
Caused by: java.lang.NullPointerException at com.macromedia.airspeed.FeatureManager.updateFeatureSet(FeatureManager.java:172) at com.macromedia.airspeed.FeatureManager.isDisabled(FeatureManager.java:78

Solution:

PPS_ACCOUNTS_FEATURE table was not updated properly.

Please call Support in order to get further assistance.

 

Adobe Connect install fails with java.sql.SQLException

Problem :

My Adobe Connect installation completed successfully, however, it keeps getting failed each time at the database configuration page.

Following is the error message I encounter each time :

java.sql.SQLException: [SQLServer JDBC Driver][SQLServer]  SET LANGUAGE failed because ‘ENGLISH’ is not an official language name or a language alias on this SQL Server.

Reason :

If you are running into this error, you might want to check the collation settings on your SQL server. If this setting on the SQL server is not correctly setup to the right English language option, you are very likely to run into this issue.

There are several English-language settings available, and the collation setting on the SQL server must match the collation setting in the Adobe Connect database installation script.

Solution :

In order to resolve this issue, you may follow the following steps :

1. On your SQL Server, modify the collation setting to use SQL_Latin_GENERAL_CP1_C1_AS, as the default language. You may find more information on Collation fundamentals on the Microsoft’s website : http://msdn.microsoft.com/

2. Once done, open the Connect database configuration page again using the configuration console (http://localhost:8510/console) and complete the database configuration wizard.

 

Slower system performance on pulling up reports / slower database performance.

PROBLEM:

While pulling up connect reports/logging into connect central, system takes a lot time to display data.

 

REASON:

When connect application uses INSERT, UPDATE, and DELETE statements to modify table data, breeze database indexes can become fragmented. Index fragmentation can happen when the logical sequence of pages is disrupted or when an index page contains less than its maximum amount of data, creating a gap in the data page or index. As indexes become fragmented, connect app get inefficient data reads when accessing tables and slower database performance.

Performing routine database maintenance such as reindexing is important for keeping your breeze databases running at peak performance. 

If your connect server usage/web access is too high then Adobe recommends to re-index your connect database every weekend night to achieve optimal performance.

While searching through the debug.log files , You get the lines similar to  -:

 

[06-25 01:30:22] https-8443-1700 (INFO) SLOW QUERY: 109ms spid:393 com.macromedia.breeze.sco.DAL.getDetailedMeetingAttendance(meeting,”1046523363″,”1012293661″,Fri Jun 22 16:00:42 CDT 2012,Fri Jun 22 17:46:14 CDT 2012)
[06-25 01:30:24] https-8443-1700 (INFO) SLOW QUERY: 156ms spid:393 com.macromedia.breeze.sco.DAL.getDetailedMeetingAttendance(meeting,”1018351354″,”1012293661″,Fri Jun 22 11:28:15 CDT 2012,Fri Jun 22 12:14:15 CDT 2012)
[06-25 01:36:55] https-8443-1649 (INFO) SLOW QUERY: 437ms spid:926 com.macromedia.breeze.sco.Reports$HeaderQuizQuestionAnswerDistribution.execute(“1070892119″,slide-view,lesson-location,core,1,2,3)
[06-25 01:36:55] https-8443-1649 (INFO) SLOW QUERY: 594ms spid:393 com.macromedia.breeze.sco.Reports$ReportQuizQuestionAnswerDistribution.execute(“1070892119″,slide-view,lesson-location,core)
[06-25 01:41:29] https-8443-1641 (INFO) SLOW QUERY: 328ms spid:393 com.macromedia.breeze.sco.Reports$ReportMyMeetings.execute(“1013634615″,view,mini-host,host,”1012293661″,meeting,”1012293745″,”1013634618″)

Observe the query execution time in ms as highlighted in red fonts. If your server logs is having values equal/greater that the highlighted one then the database must be re-indexed.

 

SOLUTION:

1. Schedule downtime and stop the connect services.

2. If connect database is located within the connect server , Launch SQL Server Management Studio Express.Skip to point 5

3. If connect database is located on remote server (Typically called as central database server managed by DBA) , Ask your DBA to execute the database re-index command mentioned on point 6

4.By default connect database name is breeze, If you have specified any other database name , Navigate to connect console page which is accessible only in connect server on URL -: http://localhost:8510/console/application-settings/database-settings?account-id=7 Identify the database name and let DBA know about the database name.

5. Launch a new query editor window.

6.Select breeze database.

Execute the below database re-index command on query editor window.

EXEC sp_MSforeachtable @command1=”print ‘?’ DBCC DBREINDEX (‘?’, ‘ ‘, 80)”
GO

EXEC sp_MSforeachtable @command1=” UPDATE STATISTICS ?”
GO

 

7.Please note that the re-index process will take sometime depending on your database size.

8. Start the connect service when the re-index process is completed.

 

You can also follow the link :- http://blogs.adobe.com/connectsupport/adobe-connect-database-maintenance-activity to perform database maintenance activity to wipe off unused table records created on connect database.

 

 

After upgrade to Connect 8,database settings page loops back and never complete

Problem:

After upgrade to Connect 8, sometimes the wizard takes us to database settings page on the console, and when we click on next,it simply goes back to first screen again. In short, database settings loops back and never complete.

Reason:

If you see this in debug logs:

01-27 17:14:45] console-8510-1 (ERROR) ERROR OCCURRED IN fetch com.macromedia.breeze.model.Account
[01-27 17:14:45] console-8510-1 (INFO) Exception caught in DBRows.populate(), e=com.macromedia.airspeed.StatusException$Invalid$NoSuchItem:

[01-27 17:14:39] cps-startup (ERROR) ERROR OCCURRED IN fetch com.macromedia.breeze.model.Asset
[01-27 17:14:39] cps-startup (INFO) Exception caught in DBRows.populate(), e=com.macromedia.airspeed.StatusException$Invalid$NoSuchItem:

Solution:

Logs indicate that few DB columns in few tables do not get created during the upgrade process. You need to call Support for further help.