Posts in Category "Database"

Using a Named Instance of SQL Server with Adobe Connect

Issue: When using  a named instance of SQL server with Adobe Connect, if you enter the name of the instance during installation, the connection to the database will fail.

Workaround: Instead of using the name of the instance, you may enable TCPIP on the instance and use the IP address and then enter the port number for which the named instance is configured on the separate line as appropriate; if the named instance is listening on port 1833 (instead of 1433), then you would use the IPAddress (192.168.1.1)  and then the port number (1833) in the appropriate fields.

Using the instance name during installation after Connect version 8 will not work. The best approach is to use the port number of the instance and the IP Address of the database server.

To troubleshoot this, use SQL Server Configuration Manager:

Screen Shot 2014-02-21 at 7.59.05 AM

  • Make sure the instance has TCP/IP enabled.
  • Check to see what port the instance is listening on for that IP.
  • Use the IP address as server name (no instance name).  Put the port number in the port number field.
  • Make sure the named instance is listening on the port entered in Connect

Screen Shot 2014-02-21 at 7.59.41 AM

Screen Shot 2014-02-21 at 8.00.24 AM

Beginning with Connect version 8, the installer changed; in previous earlier versions, you would need to enter the the server instance and port on the same line.  The  newer installer has the port on a separate line:

HOST: INSTANCE-IP-ADDRESS
PORT: INSTANCE PORT-NUMBER
DATABASE NAME: SAMPLE-NAME
USER: SAMPLE-USER
PASSWORD: ****************
CONFIRM PASSWORD: ****************

The changes in the installer (beginning with Connect version 8) caused some confusion with named instances.

A named instance will work on an initial installation.  Sometimes, in an effort to troubleshoot you may initially point to a conventional instance of SQL in order to establish the installation and then point the established Connect installation to a named instance. The DB connection from an established Connect installation is more robust and forgiving than that of an initial installation.  After the installation is complete, you can modify custom.ini to include the instance name.

Note: You could use the server name, but you would need to ensure that the named instance has NAMED PIPES enabled.

Configuring Adobe Connect to take Advantage of Database Mirroring

Full redundancy requires that the Connect database be either mirrored or clustered; Adobe uses mirroring as the preferred solution.

The following example settings in the custom.ini file are needed to configure Connect to take advantage of SQL Mirroring:

DB_NAME=ConnectDBName

DB_HOST=ConnectDBPrimaryHostName

DB_BACKUP_HOST=ConnectDBSecondaryHostName

DB_URL=jdbc:macromedia:sqlserver://{DB_HOST}:{DB_PORT};databaseName={DB_NAME};user={DB_USER};password={DB_PASSWORD};AlternateServers=({DB_BACKUP_HOST}:{DB_PORT};DatabaseName={DB_NAME});ConnectionRetryCount=12;ConnectionRetryDelay=10;FailoverMode=extended;FailoverPreconnect=false;FailoverGranularity=atomic

Note: Change the first three variables as appropriate, but do not make any changes to the DB_URL.  It is all one line and it pulls the values from the other three entries in custom.ini:

The follwoing setting is always pudent whether using mirroring or clustering, but it is particularly important if you are clustering SQL. If you are running the Connect SQL database in a SQL cluster rather than in a mirrored environment, you will want to make sure that Connect makes multiple database connection attempts during SQL fail-over. If Connect loses its SQL database, the entire Connect cluster will go down and it will wait for an administrator to manually reconnect to the database through launching the Connect configuration console on port 8510. Add the following to the custom.ini file to support any delays in clustered SQL fail-over:

DB_URL_CONNECTION_RETRY_COUNT = 15
DB_URL_CONNECTION_RETRY_DELAY= 30

The actual JDBC string That invokes these variables is in the config.ini file:

DB_URL=jdbc:macromedia:sqlserver://{DB_HOST}:{DB_PORT};databaseName={DB_NAME};user={DB_USER};password={DB_PASSWORD};ConnectionRetryCount={DB_URL_CONNECTION_RETRY_COUNT};ConnectionRetryDelay={DB_URL_CONNECTION_RETRY_DELAY}

Save the custom.ini and cycle the services.

 

 

Adobe Connect Server Licensing for Disaster Recovery

This question is commonly asked: Does my license for On-Premise Adobe Connect allow me install Adobe Connect servers for disaster recovery purposes?

First let’s define the terms: Disaster Recovery Environment refers to your technical environment designed solely to allow you to respond to an interruption in service due to an event beyond your control that creates an inability on your part to provide critical business functions for a material period of time. That is to say, it refers to a secondary site that would not be utilized in production unless the primary site went offline due to a natural or human-inflicted disaster that is beyond your control. Use of Adobe Connect servers in Disaster Recovery Environments is within the scope of your license and no additional fees are due to Adobe Systems Incorporated. For example, for the architecture depicted here, you would need four Adobe Connect server licenses. 

 

Connect_DR_cluster

 

However, adding one or more Adobe Connect servers to a local cluster is outside the scope of your license, and you will need to purchase additional licenses from Adobe Systems Incorporated to accomplish this.  Additional licenses are needed when adding any Adobe Connect servers that increase scalability in the form of:

  • Availability — What percentage of time is Connect available to geographically distributed users?
  • Reliability — How often does Connect experience problems that affect availability?
  • Performance — How fast does Connect consistently and qualitatively respond to user requests?
  • Concurrency – How many users can a Connect deployment handle concurrently?

Information around cluster expansion is here: Adobe® Connect™ server pools/clusters and hardware-based load-balancing devices with SSL acceleration

If you were to geographically distribute an active Connect cluster by placing Adobe Connect servers into two separate data centers, that would also require additional licensing. Connect servers in a cluster cannot have more than 2-3ms of latency between and among Connect servers.  Generally you would not geographically distribute Adobe Connect servers into different data centers, however, there is a chapter in the aforementioned clustering article on the topic. With that said, the architecture depicted below, is an example of a distributed active Adobe Connect cluster that is is spread between two local data-centers with nominal latency between those data-centers (less than 3ms of latency). All four servers are in production and all are actively hosting meetings and serving on-demand content.  This Connect architecture example depicted in the diagram below requires a four-server Connect cluster license:

 

Cross-DC-CLUSTER

 

Connect on VMWare – some deployment tips

Issue: VMWare is ubiquitous in the enterprise and while it opens up huge potential for management of the Connect infrastructure, it must be planned and executed with an eye toward robustness.

This advice is gleaned from conversations with senior persons on our operations team as well as from support cases generated by various customers with on-premise VMWare deployments of Connect.

One of the most important and often overlooked variables about virtualization is to make certain that  VMware is compatible with all the underlying components of the server and network architecture. The infrastructure supporting VMWare must be verified by VMware under their Hardware Certification Program or Partner Verified and Supported Products (PSVP) program; be sure to use certified hardware.

Here is the link to the compatibility reference:  http://www.vmware.com/resources/compatibility

With Connect you must consider both Tomcat and  FMS; the former can run on most anything, while the latter is a bit more demanding; RTMP can be acutely;y affected by latency and packet transmissions. If you notice unpredicted latency or a surprise crash of FMS with Connect 9.1, a good test would be to check the network components; sniff for packet transmission issues – have the vNIC of the guest VMs configured to use VMXNET3; this is a good place to start.

With reference to recommendations and best practices, it really depends on the VMware infrastructure adopted. The following references serve as a guide for an enhanced environment:

Enterprise Java Applications on VMware – Best Practices Guide: http://www.vmware.com/resources/techresources/1087

Best Practices for Performance Tuning of Latency-Sensitive Workloads in vSphere VMs: https://www.vmware.com/resources/techresources/10220

Performance Best Practices for VMware vSphere 5.1: https://www.vmware.com/resources/techresources/10329

The key with Network Storage is speed. If you lose connectivity to the shared storage then only what is cached on the origins will be available.

Shared storage requirements

  • Disk specs: 10,000–15,000 RPM — Fibre Channel preferred
  • Network link: TCP/IP — 1GB I/O throughput or better
  • Controller: Dual controllers with Active/Active multipatch capability
  • Protocol: CIFS or equivalent

Avoid, virtualizing the Connect database if possible.

I have seen that in some customer-based VMWare environments that are overtaxed, that latency among the servers on 8507 (and 8506), can cause problems. Intra-cluster latency (server to server communication) should never exceed 2-3ms. When it does we see intermittent crashes. I had one customer who had a particularly weak infrastructure and for whom I could predict his crashes; he was doing back-ups and running other tasks at a certain time weekly that would tax and hamper network connectivity for about an hour; these tasks were so all-consuming on the network, they turned every cluster resource into an individual asset on its own island. The log traces bore this out and we knew with precision what was going on. He knew he needed to upgrade his infrastructure and in the meantime we worked out a reaction plan to deal with the issue; it included:

  1. Place a higher than normal percentage of cache on each server to limit invoking shared storage
  2. Set the JDBC driver reconnection string for Database connectivity
  3. Plan Connect usage around these maintenance activities and when possible, do Connect maintenance activities at the same time as well – not very difficult as these were after hours, but being a  global operation, still not a given.

Providing Diagnostic Data to Expedite Solutions for Connect Meeting Issues

Issue: Anything that may happen during a meeting which has a pejorative effect on end-user experience.

Solution: In Connect 9.1 we have a great diagnostic option in the meeting room. You can immediately pull logs from any meeting to diagnose:

If you click Help>About Adobe Connect, while holding down the Ctrl key, the debug logs will appear int he meeting room and you will have the option to copy them to your clipboard.

log-mtg.fw

 

log-mtg1.fw

Sending me these, along with the RTMP string  Help> About Adobe Connect, while holding down the Shift key – this will be most helpful from the client experiencing the extreme latency.

rtmp-mtg.fw

Now if you want to take it even one step further and provide a client-side view of the meeting:

The instructions for enabling client-side logging are here: http://helpx.adobe.com/adobe-connect/kb/enable-logging-acrobat-connect-professional.html

Providing all this data along with the date and time (including timezone) and Meeting URL of any issue, will greatly expedite analysis and solution.

Resource Constraints cause Connection Read Error in Logs on Clustered Connect Servers

Issue: FCSj_IO:4 (x) – Connection read error: -1 LP: 5345 RP: 8506 URI: rtmp://localhost:8506/meetingapp/7/12345678

I have seen that in some VMWare environments that are very overtaxed for resources, latency between/among the clustered Connect servers on ports 8507 (and also 8506 though 8506 does not cause this error), can cause problems. Intra-cluster latency should never exceed 2-3ms. When it does we see intermittent errors and can also see crashes.

I had one unnamed customer who had a particularly weak infrastructure and  I could predict his crashes; he was doing back-ups and running other tasks at a certain time weekly that would severely hamper network connectivity for about an hour; these tasks were so all-consuming on the network, they turned every Connect cluster resource into an individual asset on its own island. The Connect logs bore this out and we knew with precision what was going on and could predict his call or email based on his maintenance schedule. He knew he needed to upgrade his infrastructure and in the meantime we worked out a reaction plan to deal with the issue; it included:

  • Place a higher than normal percentage of cache on each server to limit invoking shared storage during maintenance (see page 57)
  • Set the JDBC driver reconnection string for Database connectivity robustness
  • Plan heavy Connect usage around network and server maintenance activities and when possible, do your Connect server maintenance activities at the same time as well.

Brad’s Short-list for Connect Cluster SaaS Monitoring Options

There are many options on the monitoring theme that are worth considering when trying to decide how to keep trach of Connect server resources in a cluster. Articles describing clustered environments are on the Connect Users Community : http://www.connectusers.com  Simply search the User’s Community using the keywords: cluster, pool, edge, SSL, etc.

To effectively monitor your Connect cluster SaaS options can sometimes be cost effective than home-spun solutions; here are some staff picks with some commentary:

Sumologic– It resembles Splunk. The main difference is that Sumologic is hosted and managed externally and Splunk is hosted and managed on-premise. With Sumologic, there is not any need for software licensing, hardware investments or internal administrator expertise.  Splunk offers a similar service called splunk>storm, but it is not as mature as Sumologic and lacks some of the alerting capability found in Sumologic.

Loggly An alternative to Sumologic could be Loggly which offers a similar service; it seems that the alerting service is not exactly built in.  It requires a little more work and is called AlertBirds.

Note: It is possible to take an on-premise option like Cacti and port it to Sumologic, so you could effectively kill 2 birds with one stone.  You can setup a forwarder in 30 seconds and be searching the logs in no time at all.

Monitis – Provides capabilities similar to those of Nagios along with external monitoring.  The Monitis community writes custom monitors thereby enriching the options.

LogicMonitor – An alternative to Monitis could be something like LogicMonitor.  You may be able to port your existing Nagios checks over to it (check and verify).  This si a simple solution, installing the monitor and having basic checks like CPU, Memory, Bandwidth, Disk Usage, Disk IO and external ping, http, https and udp monitors setup would take all of 20 minutes.

Pingdom– An alternative to RedAlert at a lesser cost.  It is trusted by millions and is easy to use and has more endpoints than comparable options.  It takes five minute setup.

The beauty of a SaaS monitoring solution is that you do not need to worry about scaling your monitoring solution every time you scale your Connect architecture.  You can have a single solution for 20 Connect Clusters vs having to add Cacti servers, Nagios servers, Splunk architecture and licensing to handle the additional monitoring needs commensurate with expansion.  With a SaaS solution, there in not any build-out time.  You can literally have 20 monitors up and running in under an hour, and work on adding additional ones at your leisure in between casts with your new Deceiver 8 Fly Combo.

With reference to basic on-premise monitoring, make sure you use standard perfmon counters for things like CPU, Memory etc. For meeting count and meeting user monitoring you may use the FMSAdmin API with scripts to make various calls and then parse the data and pass it to an option such as Cacti.  To insure robustness, the FMSAdmin service should be restarted routinely. You could also use similar counters to pull data directly from the Connect database, but this is not without risk as Connect updaters and upgrades can introduce changes that may require rework of your custom counters.

Adobe Connect Servers and Hardware-based Load-balancing Devices

This updated article offers a best-practice configuration of a basic Connect pool/cluster behind a high-end, application-aware HLD such as F5 BIG-IP LTM. This article does not discuss SSL-acceleration. This article does not describe all the possible configurations, but offers a general working example of a basic HTML/RTMP non-SSL cluster/pool of Connect servers.

Adobe Connect Server Pools and Hardware-based Load-balancing Devices: http://www.connectusers.com/tutorials/2009/04/load_balancing/index.php

Adobe Connect server pools/clusters and hardware-based load-balancing devices with SSL acceleration

The most robust means of implementing secure socket layer (SSL) with Adobe Connect servers is through a hardware-based SSL accelerator and similarly, the most robust means of clustering Connect servers is with a hardware-based load-balancing device (HLD). Since all enterprise-class HLDs are also SSL accelerators (any that are not both are either legacy or low-end by definition), this example-based article offers a best-practice configuration of a Connect Server pool or cluster running Connect Meetings, Adobe Presenter on-demand content, Adobe Connect Training, Curriculum and Virtual Classrooms securely behind a high-end, application-aware HLD and SSL acceleration device such as F5 BIG-IP LTM. This article does not exhaust the possible configurations, but offers a general working example.

The full tech-note is published here:  http://www.connectusers.com/tutorials/2011/09/connect_servers_ssl_acceleration/index.php

 

Clustering Connect Servers with Multiple Availability Zones

Challenge: When a Connect cluster/pool is split into two data centers it is prudent for backup meeting rooms to be created in an alternate datacenter; you do not want the backup and the primary meeting rooms hosted in the same datacenter.

Connect Edge servers fill the gap of extending resources out to remote offices in enterprise proxy mode or of splitting external client traffic from internal traffic in reverse proxy mode; there is not a federated option to allow an origin server cluster to be spread about geographically. Very little latency is tolerable between Connect origin servers. With that said, there is the possibility of splitting an origin cluster across two datacenters for additional redundancy if the latency between the two datacenters is consistently 3 milliseconds or less. Intra-cluster communication on ports 8507 and 8506 must be unhampered for a cluster to work properly.

Spreading a cluster across two local datacenters with less than 3ms of latency requires that all servers point to the same database. Basically I am describing a regular cluster except with half of the servers in one building and the other half in another building nearby. When you do this, it is prudent to make sure that the backup meeting rooms that are spawned in support of every primary active meeting room are always created on servers in the separate datacenter from the active primary meeting. You can do this in the Connect database in PPS_ENUM_DATA_HOSTS. Configure the hosts in one data center with one pool_id, and give the other servers in the second datacenter another pool_id. Then add this setting to the custom.ini on each origin server:

APP_SERVER_POOLING=true

This setting is necessary to configure failover to take advantage of multiple availability zones.