APIC-EM does not restart after restore

Have experimented with the backup and restore functions of APIC-EM. Created a backup, which seemed to be fine, and then attempted to restore it.  APIC-EM seemed to get as far as the "Starting Controller (3/4)" stage, and then no further for over 30 minutes.

Restarted the APIC-EM VM, and APIC-EM eventually, kind of, came back such that could log in. But when tried to view device inventory, was presented with a message that the inventory service had not started. That lasted for over 30 min also, then  tried a restore again, and seeing same "3/4" stage message, 

using 1.2GA. --not trying to upgrade APIC-EM. Unable to Get RCA..

For a problem when asked to run the RCA command. This is what you should do to get the file:

1. You need to login in to the server CLI with the user grapevine and the password you gave during the installation.
2. Then you see a prompt  ($ in my case)
3. then you type: rca (lowercase)
4. Again you need to enter the password of the user grapevine
5. A tar file is created with the name "grapevine-root-{xxx.xxx.xxx.xxx}-rca-2016-03-16_20-39-16_UTC+0000'"
6. The file is stored in the "root/tmp" directory, you can use WinSCP (grapevine user) to grap the file to local disk.

Restored the APIC-EM instance to the default snapshot, which was taken after initial install. Note that the initial install was based on a database restore using the following instructions:

 "How to load the pre-populated DB on the APIC-EM 1.2 code:

 1. Download both the files from the following box folder and copy it to the sandbox APIC-EM /tmp directory.

 link sent separately in email

 2. Cd to the /tmp directory - "cd /tmp"

3. Untar the backup.tar file – "sudo tar –zxvf backup.tar" (you should see the /tmp/backup created folder on your APIC-EM)

4. Run the following command to load this DB: "sudo java -jar /tmp/dbrecovery-4.0.0-SNAPSHOT.jar restore /tmp/backup"

 Before you do this, be sure to run reset_grapevine on this APIC-EM to ensure that the DB is clean."

 After reverting to the snapshot, I did the following:

 - Created a discovery job.

- Ran the discovery and discovered a CSR1Kv running in my test lab.

- Backed up the system

- Restored from the latest backup

 The restore got as far as stage 3/4 and has been there for ~24 hours.

Could you also try ( if time permits ) , to create a new VM with APIC-EM cluster and do a fresh install and then try backup/restore. We (engineering team) have tried the backup/restore seems to be working for us with the 1.2.x build.

• Here is the response from Dev team. NTP was the issue
•  There is a NTP connectivity error
•  Error occurred while running restore operation
• [task_id=c444b23a-4a7d-11e6-be89-0a01c0a8010d]: Unable to configure NTP.
• Please confirm NTP server connectectivity and settings.
•  You can check /var/log/grapevine_manager_activity.log and /var/log/grapevine_manager.log for restore logs.

To start again config_wizard, reboot and wait more than 1h  to get APIC-EM  recovered.

 This is my log (sorry if it is too long)

Note:  apic-em-jboss-ejbca takes 20min to get up.

 2016-10-18 08:53:04,221 |    Running [4/37]: grapevine grapevine-coordinator-service rabbitmq cassandra

2016-10-18 08:53:19,338 |    Running [5/37]: router

2016-10-18 08:54:24,890 |    Running [6/37]: reverse-proxy

2016-10-18 08:54:45,068 |    Running [8/37]: postgres election-service

2016-10-18 08:54:50,114 |    Running [9/37]: log-aggregator

2016-10-18 08:55:15,356 |    Running [10/37]: cas-service

2016-10-18 08:56:10,829 |    Running [11/37]: node-ui

2016-10-18 08:56:56,301 |    Running [12/37]: rbac-service

2016-10-18 08:58:17,093 |    Running [13/37]: ipgeo-service

2016-10-18 09:01:59,413 |    Running [14/37]: remote-ras

2016-10-18 09:06:06,965 |    Running [15/37]: topology-service

2016-10-18 09:06:17,057 |    Running [16/37]: scheduler-service

2016-10-18 09:08:23,342 |    Running [17/37]: policy-manager-service

2016-10-18 09:08:28,413 |    Running [18/37]: apic-em-pki-broker-service

2016-10-18 09:09:54,862 |    Running [19/37]: task-service

2016-10-18 09:10:12,619 |    Running [21/37]: pnp-service access-policy-programmer-service

2016-10-18 09:10:25,549 |    Running [22/37]: telemetry-service

2016-10-18 09:10:44,655 |    Running [23/37]: qos-policy-programmer-service

2016-10-18 09:11:26,234 |    Running [24/37]: identity-manager-pxgrid-service file-service

2016-10-18 09:11:47,746 |    Running [25/37]: nbar-policy-programmer-service

2016-10-18 09:12:01,325 |    Running [26/37]: node-ui

2016-10-18 09:12:26,052 |    Running [27/37]: visibility-service

2016-10-18 09:13:55,137 |    Running [28/37]: app-vis-policy-programmer-service

2016-10-18 09:15:11,462 |    Running [29/37]: pfr-policy-programmer-service

2016-10-18 09:15:34,205 |    Running [30/37]: ip-pool-manager-service

2016-10-18 09:15:45,073 |    Running [31/37]: policy-analysis-service

2016-10-18 09:16:53,754 |    Running [32/37]: qos-lan-policy-programmer-service

2016-10-18 09:21:37,240 |    Running [33/37]: apic-em-event-service

2016-10-18 09:22:48,321 |    Running [34/37]: apic-em-jboss-ejbca

2016-10-18 09:42:41,412 |    Running [35/37]: apic-em-network-programmer-service

2016-10-18 09:45:02,824 |    Running [36/37]: network-poller-service

2016-10-18 09:47:09,147 |    Running [37/37]: apic-em-inventory-manager-service

2016-10-18 09:47:09,207 |    Service re-balancing not required

2016-10-18 09:47:09,207 | Configuring Update Service...

2016-10-18 09:47:12,044 | CONFIGURATION COMPLETED

 The configuration wizard has completed successfully!