Devices stuck in UNKNOWN state

Got 2 devices stuck in unknown state. Tried to re-discover them, but there was no change in their status. Tried to delete them and  got the following error message:

Restarted the APIC-EM server, but still getting the same error messages. The discovery doesn't show any errors.  It's a single server setup.

Take the instance logs of apic-em-inventory-manager-service

and share apic-em-inventory-manager-service.log* logs.(assuming it is 2.0.X build)

 If it is earlier build (1.4.X) , please share apic-em-inventory-manager-service.log, apic-inventory.log, xde.log, existenceInventory.log, inventory.log

Follow the  steps mentioned below for taking the logs:

Log in as grapevine.

execute:  cd bin

execute: grape instance status

This will display all of the services.  Look for the apic-em-inventory-manager-service and note the ip address.

execute: ssh <ip of inventory service>

execute: tar czvf logs4tac.tgz /var/log/grapevine/services/apic-em-inventory-manager-service/5.1.3.1009/

NOTE: the version at the end of the path differs depending on what you are running

execute: exit

execute: scp <ip of inventory service>:logs4tac.tgz .

execute: exit

Now the file can be retrieved with filezilla or such.

sftp://<ip of inventory service>:logs4tac.tgz.

Took a backup of APIC-EM and a snapshot in VMWare, delete all of the devices . clear the logs and then performed discovery, resync and delete operations on those remaining devices that were stuck.

There are some exceptions seen with respect to RawCliInfo table entries and found that got defect open for this issue of devices being stuck at 'Unknown' state.

 To check the delete device issue, there is no relevant logs available. Enabling DEBUG logs  and sharing the same set of logs  would help to debug the issue with delete.

 Command to enable DEBUG logs:

sudo /opt/CSCOlumos/bin/setLogLevel.sh inventory DEBUG

sudo /opt/CSCOlumos/bin/setLogLevel.sh disocvery DEBUG

 This has to be run on the VM where apic-em-inventory-manager-service is running.

 The workaround for the issues is to cleanup the corrupted entries from DB manually.

ended up doing a reset grapevine to get the entries cleared and that worked.