Use of the management interface in APIC-EM PnP

• Tried to use the management interface on Catalyst switches with PnP but it looks like its not supported and you must use the front user interfaces. 
• On devices such as nexus 3K and 9K SFP+ version only, can you use the management interfaces or does the same apply and need optics as an uplink switch that can handle 10G and Bidis as well.
•  Why is the management interface not an option on Catalyst (or both if that is the case).

Short answer is due to vrf interaction.  There is a workaround with a bootstrap config.

• Due to the clock signaling field notice we are in the situation to replace multiple hundred of ISR43XX Routers (IWAN deployment).
•  As the config on the new routers will be the very same as on the old routers, we would like to utilize the MGMT interface for this.
•  Could you please elaborate on the workaround you mentioned.

1) How the device gets assigned an IP address (is it DHCP or static)

2) How the device discovers the APIC-EM controller.

3) what is the interaction between the management interface and the transport interfaces?  i.e. if you are connecting to the device over the management interface, how does that get back to the corporate network (does it require the transport interfaces to be up?)

 In a WAN environment, #1 is often SP controlled.  e.g. MPLS link you will often need to assign a static address/route on the CE -> PE.

You might not be able to use DHCP/DNS for APIC-EM discovery.

 For this situation (the worst case), where you need to assign and IP address and the PNP profile, you would use a "bootstrap config file".  This can be delivered on a USB, or using the smartphone app (and a special console cable to connect to smartphone).

 The things you need to add are just "source interface gigabitethernet0" commands.

here is an example:

         interface GigabitEthernet0

vrf forwarding Mgmt-vrf

ip address <STATIC_IP>

no shutdown

!

ip http client source-interface GigabitEthernet0

ip route 0.0.0.0 0.0.0.0 <PE_IP>

!

pnp profile PNP_Profile

transport http ipv4 <APIC_IP> port 80 source GigabitEthernet0

!

end

Now Q#3 comes into play... how does the management interface connect back to APIC-EM?  is that an out

Any chance on using DHCP for the MGMT interface without touching the device?

saw a similar discussion here: https://communities.cisco.com/thread/59744

 This would be the expected workflow:

- Existing working network in branch sites (full connectivity to APIC and rest of network)

- New router gets delivered to branch (no configuration applied)

- Connect new router via MGMT Interface to switch (no other interfaces connected)

- Receive IP address via DHCP on MGMT Interface

- DHCP Option 43 to discover APIC-EM

- Receive configuration via PnP on MGMT Interface

- Swap Router manually afterwards

  PnP will try to use the global VRF routing to get to APIC and therefore fail. Is the EEM Script workaround still the desired way for this case?

 We've been trying to figure out how to leverage the MGMT interface also.

 Given the vrf issues which are understandable I'm wondering if this might be a way forward

- Existing Working Branch with full connectivity

- New router is delivered (no config right out of box)

- New router is connected on one of the primary interfaces (in band) and NOT the MGMT interface

- PnP connectivity is achieved via DHCP and Option 43 to APIC-EM

- Device is claimed or identified as part of a project and the right image is applied.  A configuration is also applied which re-configures the primary interface (so that the APIC-EM looses connectivity with the new router) but configures the MGMT interface so once it times out on the APIC-EM it can be connected to the MGMT interface and it should come up and now the router is accessible via its MGMT interface.

- Connect MGMT interface to configured network and attempt access

If Access:

- Review and save config

- Swap out router

If no Access:

- power cycle router and try again

 From our testing in this situation, with the APIC-EM losing connectivity with the device it will not save the config so if something should go wrong a simple reboot should bring it back to a clean starting point.

 A better understanding of how the config is applied would be idea here. Is the config TFTP or otherwise copied to running as a whole?

 The only other option, which is one we've used is to load a "partial" config which keeps the interface used for PnP intact and viable.  Once image and this partial config are complete via the APIC-EM, I have scripts that will "wr mem" and then copy the config over via TFTP.  I then update the configuration of the interface used for PnP and TFTP that config back to the startup.  That maintains my connectivity but once the router reloads it will have the proper config.

 Its a bit risky to be sure but so far these are the only workarounds we've come up with.

if the config applies successfully, then the pnp agent does a "wr mem" and tells the controller it has been successful, and it status will be "Provisioned".

 You can change the interface the device uses to talk to APIC-EM during provisioning (I call this interface switchover).  The reason it works is all communications starts from device -> APIC.  I have used this in a Non-VLAN1 for management work around (without startup-vlan).  You can change interface/IP, as long as the new interface can communicate to APIC, it should be fine.

when you did your testing for the management interface, I think you also need to change the pnp profile, so the pnp connection uses the management interface.

i.e. change

 pnp profile pnp-zero-touch

transport https ipv4 W.X.Y.Z port 443

to

pnp profile pnp-zero-touch

transport https ipv4 W.X.Y.Z port 443 interface g0

 otherwise when you change over the configuration, the pnp agent is still going to use a non management interface to try to connect to the controller.

Notice that you can configure a VRF here too.  That might also work.

• We used the APIC-EM for the initial provisioning and then the management interface for non-pnp access and flipped over to Ansible and scripts for the "finishing touches" including running a bunch of verification commands.  This is even better because we can maintain connectivity to the APIC-EM! (and still do non-controller based things if we need to).
•  How will the controller deal with the change In this situation, connectivity via in-line interface, config applied via https which then knocks out that "session" but allows a secondary session via MGMT to come up so likely new IP and new session established to controller.  what happens at this point. Assuming the "logs" are tied together via SN.

The controller just sees a series of requests coming in from the device.  The source IP address can change.  The device is identified by it's serial number and model number. Both the device and the controller have state machines that they work through.

 Remember the discussion we had about not displaying the IP address for project devices.... because i said it might change during provisioning.... :-)

•  Showing or otherwise getting the IP will still be valuable in many uses cases (that we will see).   So when defining a project (or Claiming a device) you can fill in a "bootstrap" field.  Is that an indication to look for a file in USB flash or the Mobile APP. or could that be a small file that sets up the communication channel you want to use in production (ie. interface switch over) delivered via the initial in-band connection?  Whats a good link for this type of info?... I keep thinking I'm asking questions I should be able to figure out but the documentation can be a bit confusing (to me anyway) because some is APIC-EM and some is PnP and at least initially didn't appreciate the distinction! 

 In short a bootstrap config file can be in two places:

1) on the controller (and bound to a device rule - based on name/serial number)

2) on a USB flash in a fixed file "router.cfg"

For case #1, you can use the mobile app to connect to the PnP server and then download the file to the device via a special serial cable that connects your smart phone to the network-device.

 For case #2, it is automated process. if the bootstrap file is present, the device will use it.  NOTE: this is not supported on all devices (e.g. switches).  Definitely works on 4k routers.