video and preview views rendering green

Hi team,

Below is the sentence from the Advanced CSDK Developer's Guide:

"When there is no video stream being sent or received, the video view and preview view renders a full frame of green. Only when streaming does video appear."

It seems this green screen is by design. However, I wanted to ask whether there is a way to change this?

Thanks and regards,

Murat

Hi Murat.

Thank you for your question.

I don't believe it is possible to change the colour of this but let me find out more information for you.

Can I just confirm If you are interested in any platform in particular?

Cheers,

Rach D

Hi Rach,

Thanks for the reply. Sorry, I forgot to mention in my first message. This is only for Android platform.

iOS is working fine but in Android we see the green screen.

Thanks and regards,

Murat

Hi Murat.

Thank you for the update, I've just had a quick discussion with the team here and they asked me to check if you have tried this:

"Register as a VideoSurfaceListener, and implement the onSurfaceRenderingStarted() method, and only display the VideoSurface in the app once rendering has started."

Our Sample application uses this approach and doesn't display the green frames.

If this doesn't work for you please do let us know and we can investigate into this further. We will in that case however need to know more information on what you are trying to achieve and where the issue is occurring.

Cheers,

Rach

Hi Rach,

I will try this and let you know about the results.

Thanks and regards,

Murat

Hi Rach,

We tried but it did not work. Do you have a sample code snippet for this?

Thanks and regards,

Murat

Hi Murat.

That's no problem, do you have access to our sample application? You can use the code from there if so, it may even be worth checking that is what you are looking for by previewing the sample.

Cheers,

Rach D

Hi Rach,

Do you mean the sample application coming with the SDK?

Thanks and regards,

Murat

Hi Rach,

The related code in the SDK is as follows:

----------------------------------------------------------------------------------------------------------------------------

        @Override

        public void onSurfaceRenderingStarted(VideoSurface videoSurface)

        {

            if (videoViews.contains(videoSurface))

            {

                assignVideoSurfaceToCall(videoSurface);

            }

            if (listener != null)

            {

                listener.onSurfaceRenderingStarted(videoSurface);

            }

        }

----------------------------------------------------------------------------------------------------------------------------      

My customer's code is as follows:

----------------------------------------------------------------------------------------------------------------------------    

        public void onSurfaceRenderingStarted(VideoSurface videoSurface) {

        new Handler(Looper.getMainLooper()).post(new Runnable() {

            @Override

            public void run() {

                remotePreviewContainer.setVisibility(View.VISIBLE);

            }

        });

----------------------------------------------------------------------------------------------------------------------------

Do you see a problem with this code?

Thanks and regards,

Murat

Hi Murat.

Would you be able to give a little more information on the customers code? a larger snippet would be good but we are looking to find out of the video view was visible before the renderingStarted event?

Cheers,

Rach

Hi Rach,

I could not attach the file itself. It is little bit long but I am pasting the code of the class here. Thank you very much for your support.

Regards,

Murat

------------------------------------------------------------------------------------------------------------------------------------------------------------

public class MainActivity extends AppCompatActivity implements UCListener, VideoSurfaceListener {

    private UC uc;

    private Phone phone;

    private RelativeLayout container;

    private RelativeLayout remotePreviewContainer;

    private VideoSurface remotePreview;

    private RelativeLayout selfPreviewContainer;

    private VideoSurface selfPreview;

    private Button startStopButton;

    private com.alicecallsbob.fcsdk.android.phone.Call activeCall;

    @Override

    protected void onCreate(Bundle savedInstanceState) {

        super.onCreate(savedInstanceState);

        container = new RelativeLayout(this);

        setContentView(container);

        encryptionTest();

        remotePreviewContainer = new RelativeLayout(this);

        remotePreviewContainer.setBackgroundColor(Color.BLACK);

        remotePreviewContainer.setLayoutParams(new RelativeLayout.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.MATCH_PARENT));

        container.addView(remotePreviewContainer);

        selfPreviewContainer = new RelativeLayout(this);

        selfPreviewContainer.setBackgroundColor(Color.BLACK);

        RelativeLayout.LayoutParams selfPreviewContainerParams = new RelativeLayout.LayoutParams(Utils.convertPx(80), Utils.convertPx(100));

        selfPreviewContainerParams.addRule(RelativeLayout.ALIGN_PARENT_BOTTOM);

        selfPreviewContainerParams.addRule(RelativeLayout.ALIGN_PARENT_RIGHT);

        selfPreviewContainerParams.setMargins(0, 0, Utils.convertPx(10), Utils.convertPx(10));

        selfPreviewContainer.setLayoutParams(selfPreviewContainerParams);

        container.addView(selfPreviewContainer);

        RelativeLayout.LayoutParams startStopLayoutParams = new RelativeLayout.LayoutParams(ViewGroup.LayoutParams.WRAP_CONTENT, ViewGroup.LayoutParams.WRAP_CONTENT);

        startStopLayoutParams.addRule(RelativeLayout.ALIGN_PARENT_BOTTOM);

        startStopLayoutParams.addRule(RelativeLayout.CENTER_HORIZONTAL);

        startStopLayoutParams.setMargins(0, 0, 0, Utils.convertPx(10));

        startStopButton = new Button(this);

        startStopButton.setText("Start");

        container.addView(startStopButton, startStopLayoutParams);

        startStopButton.setOnClickListener(new View.OnClickListener() {

            @Override

            public void onClick(View view) {

                postToGetSessionId();

            }

        });

        PermissionManager pm = PermissionManager.create(this);

        pm.with(android.Manifest.permission.CAMERA, android.Manifest.permission.RECORD_AUDIO).request();

        AudioManager audioManager = (AudioManager) getSystemService(Context.AUDIO_SERVICE);

        audioManager.setMode(AudioManager.MODE_IN_COMMUNICATION);

        audioManager.setSpeakerphoneOn(true);

    }

    private void postToGetSessionId(){

        String postDataJSON =

        "{\n" +

        "  \"webAppId\": \"OdeaRemoteExpertProject\",\n" +

        "  \"allowedOrigins\": [\"*\"],\n" +

        "  \"urlSchemeDetails\": {\n" +

        "      \"host\": \"remoteexpert.odeabank.com.tr\",\n" +

        "      \"port\": \"443\",\n" +

        "      \"secure\": true\n" +

        "  },\n" +

        "  \"voice\": {\n" +

        "      \"username\": \"assist-m2v7r3jpb0jsk5j28ok4b5o4s\",\n" +

        "      \"displayName\": \"An Anonymous User\",\n" +

        "      \"domain\": \"10.224.60.211\"\n" +

        "  },\n" +

        "  \"uuiData\": \"deneme1\"\n" +

        "};";

        OkHttpClient httpClient = new OkHttpClient();

        Request request = new Request.Builder()

                .url("https://remoteexpert.odeabank.com.tr/gateway/sessions/session")

                .post(RequestBody.create(MediaType.parse("application/json; charset=utf-8"), postDataJSON))

                .build();

        httpClient.newCall(request).enqueue(new Callback() {

            @Override

            public void onFailure(Call call, IOException e) {

                Log.v("Error", "Error");

            }

            @Override

            public void onResponse(Call call, Response response) throws IOException {

                Log.v("Response", response.toString());

                JSONObject responseData = null;

                try {

                    responseData = new JSONObject(response.body().string());

                    if (responseData != null) {

                        final String sessionId = responseData.getString("sessionid");

                        new Handler(Looper.getMainLooper()).post(new Runnable() {

                            @Override

                            public void run() {

                                startSession(sessionId);

                            }

                        });

                    }

                } catch (JSONException e) {

                    e.printStackTrace();

                }

            }

        });

    }

    private void startSession(String sessionId) {

        uc = UCFactory.createUc(this, sessionId, this);

        uc.setNetworkReachable(true);

        uc.setTrustManager(new TrustAllCerts());

        uc.setHostnameVerifier(new NullHostNameVerifier());

        uc.startSession();

        phone = uc.getPhone();

        phone.addListener(new PhoneListener() {

            @Override

            public void onIncomingCall(com.alicecallsbob.fcsdk.android.phone.Call call) {

            }

            @Override

            public void onCaptureSettingChange(PhoneVideoCaptureSetting phoneVideoCaptureSetting, int i) {

            }

            @Override

            public void onLocalMediaStream() {

            }

        });

        phone.setCamera(CAMERA_FACING_FRONT);

        //phone.setVideoOrientation(Surface.ROTATION_0);

        phone.setVideoOrientation(270);

        phone.setPreferredCaptureResolution(PhoneVideoCaptureResolution.RESOLUTION_960x720);

        remotePreview = phone.createVideoSurface(MainActivity.this, new Point(remotePreviewContainer.getWidth(), remotePreviewContainer.getHeight()), MainActivity.this);

        remotePreviewContainer.addView(remotePreview);

        remotePreviewContainer.setVisibility(View.GONE);

        selfPreview = phone.createVideoSurface(MainActivity.this, new Point(selfPreviewContainer.getWidth(), selfPreviewContainer.getHeight()), MainActivity.this);

        selfPreview.setZOrderOnTop(true);

        selfPreviewContainer.addView(selfPreview);

        phone.setPreviewView(selfPreview);

    }

    @Override

    public void onSessionStarted() {

        new Handler(Looper.getMainLooper()).post(new Runnable() {

            @Override

            public void run() {

                try {

                    activeCall = phone.createCall("sip:80003@10.224.60.211", true, true, new CallListener() {

                        @Override

                        public void onDialFailed(com.alicecallsbob.fcsdk.android.phone.Call call, String s, CallStatus callStatus) {

                        }

                        @Override

                        public void onCallFailed(com.alicecallsbob.fcsdk.android.phone.Call call, String s, CallStatus callStatus) {

                        }

                        @Override

                        public void onMediaChangeRequested(com.alicecallsbob.fcsdk.android.phone.Call call, boolean b, boolean b1) {

                        }

                        @Override

                        public void onStatusChanged(com.alicecallsbob.fcsdk.android.phone.Call call, CallStatus callStatus) {

                        }

                        @Override

                        public void onStatusChanged(com.alicecallsbob.fcsdk.android.phone.Call call, CallStatusInfo callStatusInfo) {

                        }

                        @Override

                        public void onRemoteDisplayNameChanged(com.alicecallsbob.fcsdk.android.phone.Call call, String s) {

                        }

                        @Override

                        public void onRemoteMediaStream(com.alicecallsbob.fcsdk.android.phone.Call call) {

                        }

                        @Override

                        public void onInboundQualityChanged(com.alicecallsbob.fcsdk.android.phone.Call call, int i) {

                        }

                        @Override

                        public void onRemoteHeld(com.alicecallsbob.fcsdk.android.phone.Call call) {

                        }

                        @Override

                        public void onRemoteUnheld(com.alicecallsbob.fcsdk.android.phone.Call call) {

                        }

                    });

                    activeCall.setVideoView(remotePreview);

                } catch (CallCreationWithErrorException e) {

                    e.printStackTrace();

                }

            }

        });

    }

    @Override

    public void onSessionNotStarted() {

    }

    @Override

    public void onSystemFailure() {

    }

    @Override

    public void onConnectionLost() {

    }

    @Override

    public void onConnectionRetry(int i, long l) {

    }

    @Override

    public void onConnectionReestablished() {

    }

    @Override

    public void onGenericError(String s, String s1) {

    }

    @Override

    public void onFrameSizeChanged(int i, int i1, VideoSurface.Endpoint endpoint, VideoSurface videoSurface) {

    }

    @Override

    public void onSurfaceRenderingStarted(VideoSurface videoSurface) {

        new Handler(Looper.getMainLooper()).post(new Runnable() {

            @Override

            public void run() {

                remotePreviewContainer.setVisibility(View.VISIBLE);

            }

        });

    }

    private void encryptionTest() {

        String userID = "************";

        String password = "***************";

        String encryptred = null;

        try {

            encryptred = EncryptionManager.encrypt(userID, password);

            Log.v("EncryptionManager::encryptData", encryptred);

        }

        catch (Exception e) {

            e.printStackTrace();

        }

        try {

            String decrypted = EncryptionManager.decrypt(encryptred, password);

            Log.v("EncryptionManager::decryptData", decrypted);

        }

        catch (Exception e) {

            e.printStackTrace();

        }

    }

}

------------------------------------------------------------------------------------------------------------------------------------------------------------

Thanks Murat.

Will get back to you as soon as we can.

Cheers,

Rach

A potential issue we see is with the 'onSurfaceRenderingStarted' callback. It seems to be being called for both local preview and the remote preview. There doesn't seem to be a distinction between the two and it's possible that the remote view is being made visible when the local preview has started rendering.

Could you try

if (videoSurface == myRemoteSurface) {//make remote visible}

and see if that helps? 

We're hoping by making a clear definitive between the two you may not see the green screen issue any longer.

Cheers,

Rach D

Hi Rach,

We tried this but unfortunately still the same. We are receiving the onSurfaceRenderingStarted event and we are checking if videoSurface == myRemoteSurface. However, again we are seeing the green background for a few seconds.

When is the onSurfaceRenderingStarted event coming? Is it triggered by the SIP OK message or by the RTP packets coming?

Thanks and regards,

Murat

Hi Murat.

Thank you for your reply. Can I just confirm if the green background has improved to only a few seconds or a few seconds was always the case?
Was there any change saw after you added the above suggestion?

For the SurfaceRenderingStarted I believe it's the incoming RTP packets but let me clarify that with the team to be 100% sure.

Cheers,

Rach

Hi Rach,

No, there is no improvement. It was same as before. Now, I made a few more tests. Previously, we were sending the calls to CVP or to auto-answer agents. That's why we were seeing the green window for a very short period.

Now, I disabled auto-answer and tested again. So, it is much clearer. As soon as I click the call button, I see the green screen and if I don't answer the call on the agent side (it keeps ringing), I continue seeing the green screen. When the agent answers, then green screen disappears.

Thanks and regards,

Murat

Hi Murat.

I just got an update from our team. If the green screen is visible before the call is answered then it would seem that the 'view' is being made visible before the call is answered.
Have you been able to test the sample application from the play store to see if you experience the same there? 

The only other suggestion we can think of at this time would be to to use the 'onStatusChanged()' callback in the callListener interface so you are only making the video view visible when the call is established. i.e 'CallStatus.IN_CALL' .

If this doesn't work I think we will need to go back to the drawing board and raise a defect on this one to get it properly looked into.

Let us know how you get on.

Cheers,

Rach D

Hi Rach,

We added this control. Now, we are checking CalStatus.IN_CALL and then making the view visible. Actually, this improved the experience. Previously, green screen was showing during the ringing state. It was taking around 5-6 seconds. Now, we get rid the of green screen while ringing. The background is now black during screen. However, now green is shown only for a second (between the call is answered and the actual RTP media packets are received).

Do you have any idea how we can improve more?

Thanks and regards,

Murat

Hi Murat.

I'm glad to hear the recent change has improved your experience.

As for it still showing green screen for a second before answering, this should not be happening with the changes we have advised to you. Are you able to clean out your project to see if there are any old configs being called in from somewhere?

As I said in the previous e-mail we are at the end of our ideas of what could be causing the issue to happen, so in order to investigate this further we would need to raise a defect to get some dedicated developer time to take a look at this.

Would it be possible for you to reach out to your Cisco BU representative to raise this with us? It would need approval from the BU before we are able to investigate things at our end.

Apologies we were not able to get this entirely resolved for you here.

Cheers,

Rach D

Hi Rach,

I will try this with a clean project one more time. You helped a lot, thank you very much for all your support and your time.

Regards,

Murat

Hi Murat.

Not a problem at all, always a pleasure. Thank you for your patience in waiting for our updates.

Good luck with the clearing. It may even be worth commenting out the onSurfaceRendering in both places individually see if this makes a difference for you.

Cheers,

Rach

Would it be possible to create and attach a very simple sample project that demonstrates the issue so that Rach and the team can easily reproduce and troubleshoot..?

Hi David,

I cannot attach here but If it is OK for you, I can share the project with you via email.

Thanks and regards,

Murat

Hey, Murat. Was thinking you might try leveraging the onRemoteMediaStream callback to help with this. You can map the video to your video element when the call is setup, but leave that element hidden until the onRemoteMediaStream callback is triggered, at which time you show or unhide your video surface. Not sure if this will work for you, just a thought...

Hi Murat,

Did you resolve this issue?

I heard one  customer was happy with the results of using the following advice:

Change the Z order of the remote preview to hide it in the background and when onSurfaceRenderingStarted is fired, change the Z order to make it viewable?

Regards,

Rob

Hi Murat,

Thank you for the reply, let me check with my team and I'll get back to you soon.

Cheers,

Rach D

Hi Brad,

Thanks for the suggestion, I will try this.

Regards,

Murat

Hi Murat

We got same issue and still can't resolved, could you please share with us if your issue already resolved?

regards

Van.

All,

The REM team at my end as discovered the short 'burst' of green seen before a call is answered seems to be an issue with the application when making a call through a CUCM (direct call to browser does not see the issue hence why this did not show in our testing).

We are currently working on this defect at our end, please bear with us.

Cheers,

Rach D

Hi Rach,

We are looking forward to your good news. Thanks for your time and effort.

Regards,

Murat

Hi Rach

Thank for your investigate. we got other issue with IOS, When setup call, the video frame display black screen (about 2 sec) before displayed agent video, can we change this background to other color or picture?

Regards

Van.

Hi Van.

Thank you for reaching out. Would it be possible for you to open a new thread for this so we can track it separately to help other customers in the future?

If you could specify your call flow too in the new ticket that would be great.

Cheers,

Rach D

Hi Rach,

Do you have any updates on this? Do we have a Cisco defect ID on this?

Thanks and regards,

Murat

Hi Murat.

This is not something I could tell you, you'll have to follow up on the ticket you created back with my team.

Cheers,

Rach D

Hi Rach,

The following recommendation fixed the issue.

“Change the Z order of the remote preview to hide it in the background and when onSurfaceRenderingStarted is fired, change the Z order to make it viewable?”

Thank you very much for your support.

Best regards,

Murat

Thank you for your reply Murat, that's great news!

Thank you for the update, you're more than welcome.

Cheers,

Rach D

Hi Rob,

We tried several different things, none of them worked but finally "changing the Z order of remote preview" resolved the issue.

Thanks and regards,

Murat