"Template service broker apiserver fails to start" during Ansible MCP startup

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

"Template service broker apiserver fails to start" during Ansible MCP startup

John Stafford
Hi,

1) when I run "ansible-playbook playbook.yml -e "dockerhub_username=$DOCKERHUB_USERNAME" -e "dockerhub_password=$DOCKERHUB_PASSWORD" --ask-become-pass"

2) the execution gets as far as "TASK [oc-cluster-up : Cluster up] " and then terminates with an error[1] (after hanging for about 4-5 mins)

3) the docker image[2] in cockpit contains the the message "Unable to connect to the server: dial tcp 172.30.0.1:443: getsockopt: no route to host disconnected"

Any ideas on how to troubleshoot/work around this?

FYI - I completed a 'make clean' against the '/home/jstaffor/MCP/mcp-standalone/' folder and executed the command above again but unfortunately the same situation occurred.

[1]

“Unable to connect to the server: dial tcp 172.30.0.1:443: getsockopt: no route to host disconnected”


fatal: [localhost]: FAILED! => {"changed": false, "cmd": "oc cluster up --service-catalog=true --host-config-dir=/home/jstaffor/MCP/mcp-standalone/ui --host-data-dir=/home/jstaffor/MCP/mcp-standalone/ui/openshift-data --host-pv-dir=/home/jstaffor/MCP/mcp-standalone/ui/openshift-pvs --host-volumes-dir=/home/jstaffor/MCP/mcp-standalone/ui/openshift-volumes --routing-suffix=192.168.37.1.nip.io --public-hostname=192.168.37.1 --version=v3.7.0-rc.0 --image=openshift/origin", "delta": "0:10:42.022010", "end": "2017-11-07 11:45:48.927503", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2017-11-07 11:35:06.905493", "stderr": "", "stderr_lines": [], "stdout": "Starting OpenShift using openshift/origin:v3.7.0-rc.0 ...\n-- Checking OpenShift client ... OK\n-- Checking Docker client ... OK\n-- Checking Docker version ... OK\n-- Checking for existing OpenShift container ... OK\n-- Checking for openshift/origin:v3.7.0-rc.0 image ... OK\n-- Checking Docker daemon configuration ... OK\n-- Checking for available ports ... \n   WARNING: Binding DNS on port 8053 instead of 53, which may not be resolvable from all clients.\n-- Checking type of volume mount ... \n   Using nsenter mounter for OpenShift volumes\n-- Creating host directories ... OK\n-- Finding server IP ... \n   Using public hostname IP 192.168.37.1 as the host IP\n   Using 192.168.37.1 as the server IP\n-- Checking service catalog version requirements ... OK\n-- Starting OpenShift container ... \n   Creating initial OpenShift configuration\n   Starting OpenShift using container 'origin'\n   Waiting for API server to start listening\n   OpenShift server started\n-- Adding default OAuthClient redirect URIs ... OK\n-- Installing registry ... \n   scc \"privileged\" added to: [\"system:serviceaccount:default:registry\"]\n-- Installing router ... OK\n-- Importing image streams ... OK\n-- Importing templates ... OK\n-- Importing service catalog templates ... OK\n-- Installing service catalog ... OK\n-- Installing template service broker ... FAIL\n   Error: failed to start the template service broker apiserver: timed out waiting for the condition", "stdout_lines": ["Starting OpenShift using openshift/origin:v3.7.0-rc.0 ...", "-- Checking OpenShift client ... OK", "-- Checking Docker client ... OK", "-- Checking Docker version ... OK", "-- Checking for existing OpenShift container ... OK", "-- Checking for openshift/origin:v3.7.0-rc.0 image ... OK", "-- Checking Docker daemon configuration ... OK", "-- Checking for available ports ... ", "   WARNING: Binding DNS on port 8053 instead of 53, which may not be resolvable from all clients.", "-- Checking type of volume mount ... ", "   Using nsenter mounter for OpenShift volumes", "-- Creating host directories ... OK", "-- Finding server IP ... ", "   Using public hostname IP 192.168.37.1 as the host IP", "   Using 192.168.37.1 as the server IP", "-- Checking service catalog version requirements ... OK", "-- Starting OpenShift container ... ", "   Creating initial OpenShift configuration", "   Starting OpenShift using container 'origin'", "   Waiting for API server to start listening", "   OpenShift server started", "-- Adding default OAuthClient redirect URIs ... OK", "-- Installing registry ... ", "   scc \"privileged\" added to: [\"system:serviceaccount:default:registry\"]", "-- Installing router ... OK", "-- Importing image streams ... OK", "-- Importing templates ... OK", "-- Importing service catalog templates ... OK", "-- Installing service catalog ... OK", "-- Installing template service broker ... FAIL", "   Error: failed to start the template service broker apiserver: timed out waiting for the condition"]}

-- 

​[2]
Id:bc3e4b7320ec40571486521712ad33420f7c04b2f44e3b7ecdcaee0b768a3554
Created:2017-11-07T11:55:52.864111572Z
Image:
sha256:76d9e4ab142b310fa8adf1427f99f98fdc627ae8e460b92f55878ba68e6a9679
Command:/bin/bash -c "#/bin/bash set -e function generate_pv() { local basedir=\"${1}\" local name=\"${2}\" cat <<EOF apiVersion: v1 kind: PersistentVolume metadata: name: ${name} labels: volume: ${name} spec: capacity: storage: 100Gi accessModes: - ReadWriteOnce - ReadWriteMany - ReadOnlyMany hostPath: path: ${basedir}/${name} persistentVolumeReclaimPolicy: Recycle EOF } function setup_pv_dir() { local dir=\"${1}\" if [[ ! -d \"${dir}\" ]]; then mkdir -p \"${dir}\" fi if ! chcon -t svirt_sandbox_file_t \"${dir}\" &> /dev/null; then echo \"Not setting SELinux content for ${dir}\" fi chmod 770 \"${dir}\" } function create_pv() { local basedir=\"${1}\" local name=\"${2}\" setup_pv_dir \"${basedir}/${name}\" if ! oc get pv \"${name}\" &> /dev/null; then generate_pv \"${basedir}\" \"${name}\" | oc create -f - else echo \"persistentvolume ${name} already exists\" fi } basedir=\"/home/jstaffor/MCP/mcp-standalone/ui/openshift-pvs\" setup_pv_dir \"${basedir}/registry\" for i in $(seq -f \"%04g\" 1 100); do create_pv \"${basedir}\" \"pv${i}\" done "
State:Exited 1
Restart Policy:
IP Address:
IP Prefix Length:0
Gateway:
MAC Address:
 


JOHN STAFFORD

TECHNICAL WRITER, CCS

Red Hat 


_______________________________________________
feedhenry-dev mailing list
[hidden email]
https://www.redhat.com/mailman/listinfo/feedhenry-dev
Reply | Threaded
Open this post in threaded view
|

Re: "Template service broker apiserver fails to start" during Ansible MCP startup

John Frizelle
Hi John,

From looking at the output and my own experience, this is OpenShift itself not starting up correctly - "Error: failed to start the template service broker apiserver: timed out waiting for the condition"]}"

If you manually run oc cluster up, you often get a bit more info (and in a more readable format) than what is output from the ansible installer.

Since the problem here is specific to oc, we might end up needing input from some openshift folks.

Regards,
John.

--
John Frizelle
Chief Architect, Red Hat Mobile
Consulting Engineer

mobile: <a href="tel://+353872901644" target="_blank">+353 87 290 1644
twitter: @johnfriz
skype: john_frizelle




On 7 November 2017 at 12:08, John Stafford <[hidden email]> wrote:
Hi,

1) when I run "ansible-playbook playbook.yml -e "dockerhub_username=$DOCKERHUB_USERNAME" -e "dockerhub_password=$DOCKERHUB_PASSWORD" --ask-become-pass"

2) the execution gets as far as "TASK [oc-cluster-up : Cluster up] " and then terminates with an error[1] (after hanging for about 4-5 mins)

3) the docker image[2] in cockpit contains the the message "Unable to connect to the server: dial tcp 172.30.0.1:443: getsockopt: no route to host disconnected"

Any ideas on how to troubleshoot/work around this?

FYI - I completed a 'make clean' against the '/home/jstaffor/MCP/mcp-standalone/' folder and executed the command above again but unfortunately the same situation occurred.

[1]

“Unable to connect to the server: dial tcp 172.30.0.1:443: getsockopt: no route to host disconnected”


fatal: [localhost]: FAILED! => {"changed": false, "cmd": "oc cluster up --service-catalog=true --host-config-dir=/home/jstaffor/MCP/mcp-standalone/ui --host-data-dir=/home/jstaffor/MCP/mcp-standalone/ui/openshift-data --host-pv-dir=/home/jstaffor/MCP/mcp-standalone/ui/openshift-pvs --host-volumes-dir=/home/jstaffor/MCP/mcp-standalone/ui/openshift-volumes --routing-suffix=192.168.37.1.nip.io --public-hostname=192.168.37.1 --version=v3.7.0-rc.0 --image=openshift/origin", "delta": "0:10:42.022010", "end": "2017-11-07 11:45:48.927503", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2017-11-07 11:35:06.905493", "stderr": "", "stderr_lines": [], "stdout": "Starting OpenShift using openshift/origin:v3.7.0-rc.0 ...\n-- Checking OpenShift client ... OK\n-- Checking Docker client ... OK\n-- Checking Docker version ... OK\n-- Checking for existing OpenShift container ... OK\n-- Checking for openshift/origin:v3.7.0-rc.0 image ... OK\n-- Checking Docker daemon configuration ... OK\n-- Checking for available ports ... \n   WARNING: Binding DNS on port 8053 instead of 53, which may not be resolvable from all clients.\n-- Checking type of volume mount ... \n   Using nsenter mounter for OpenShift volumes\n-- Creating host directories ... OK\n-- Finding server IP ... \n   Using public hostname IP 192.168.37.1 as the host IP\n   Using 192.168.37.1 as the server IP\n-- Checking service catalog version requirements ... OK\n-- Starting OpenShift container ... \n   Creating initial OpenShift configuration\n   Starting OpenShift using container 'origin'\n   Waiting for API server to start listening\n   OpenShift server started\n-- Adding default OAuthClient redirect URIs ... OK\n-- Installing registry ... \n   scc \"privileged\" added to: [\"system:serviceaccount:default:registry\"]\n-- Installing router ... OK\n-- Importing image streams ... OK\n-- Importing templates ... OK\n-- Importing service catalog templates ... OK\n-- Installing service catalog ... OK\n-- Installing template service broker ... FAIL\n   Error: failed to start the template service broker apiserver: timed out waiting for the condition", "stdout_lines": ["Starting OpenShift using openshift/origin:v3.7.0-rc.0 ...", "-- Checking OpenShift client ... OK", "-- Checking Docker client ... OK", "-- Checking Docker version ... OK", "-- Checking for existing OpenShift container ... OK", "-- Checking for openshift/origin:v3.7.0-rc.0 image ... OK", "-- Checking Docker daemon configuration ... OK", "-- Checking for available ports ... ", "   WARNING: Binding DNS on port 8053 instead of 53, which may not be resolvable from all clients.", "-- Checking type of volume mount ... ", "   Using nsenter mounter for OpenShift volumes", "-- Creating host directories ... OK", "-- Finding server IP ... ", "   Using public hostname IP 192.168.37.1 as the host IP", "   Using 192.168.37.1 as the server IP", "-- Checking service catalog version requirements ... OK", "-- Starting OpenShift container ... ", "   Creating initial OpenShift configuration", "   Starting OpenShift using container 'origin'", "   Waiting for API server to start listening", "   OpenShift server started", "-- Adding default OAuthClient redirect URIs ... OK", "-- Installing registry ... ", "   scc \"privileged\" added to: [\"system:serviceaccount:default:registry\"]", "-- Installing router ... OK", "-- Importing image streams ... OK", "-- Importing templates ... OK", "-- Importing service catalog templates ... OK", "-- Installing service catalog ... OK", "-- Installing template service broker ... FAIL", "   Error: failed to start the template service broker apiserver: timed out waiting for the condition"]}

-- 

​[2]
Id:bc3e4b7320ec40571486521712ad33420f7c04b2f44e3b7ecdcaee0b768a3554
Created:2017-11-07T11:55:52.864111572Z
Image:
sha256:76d9e4ab142b310fa8adf1427f99f98fdc627ae8e460b92f55878ba68e6a9679
Command:/bin/bash -c "#/bin/bash set -e function generate_pv() { local basedir=\"${1}\" local name=\"${2}\" cat <<EOF apiVersion: v1 kind: PersistentVolume metadata: name: ${name} labels: volume: ${name} spec: capacity: storage: 100Gi accessModes: - ReadWriteOnce - ReadWriteMany - ReadOnlyMany hostPath: path: ${basedir}/${name} persistentVolumeReclaimPolicy: Recycle EOF } function setup_pv_dir() { local dir=\"${1}\" if [[ ! -d \"${dir}\" ]]; then mkdir -p \"${dir}\" fi if ! chcon -t svirt_sandbox_file_t \"${dir}\" &> /dev/null; then echo \"Not setting SELinux content for ${dir}\" fi chmod 770 \"${dir}\" } function create_pv() { local basedir=\"${1}\" local name=\"${2}\" setup_pv_dir \"${basedir}/${name}\" if ! oc get pv \"${name}\" &> /dev/null; then generate_pv \"${basedir}\" \"${name}\" | oc create -f - else echo \"persistentvolume ${name} already exists\" fi } basedir=\"/home/jstaffor/MCP/mcp-standalone/ui/openshift-pvs\" setup_pv_dir \"${basedir}/registry\" for i in $(seq -f \"%04g\" 1 100); do create_pv \"${basedir}\" \"pv${i}\" done "
State:Exited 1
Restart Policy:
IP Address:
IP Prefix Length:0
Gateway:
MAC Address:
 


JOHN STAFFORD

TECHNICAL WRITER, CCS

Red Hat 


_______________________________________________
feedhenry-dev mailing list
[hidden email]
https://www.redhat.com/mailman/listinfo/feedhenry-dev



_______________________________________________
feedhenry-dev mailing list
[hidden email]
https://www.redhat.com/mailman/listinfo/feedhenry-dev
Reply | Threaded
Open this post in threaded view
|

Re: "Template service broker apiserver fails to start" during Ansible MCP startup

John Stafford
@ John F - thanks for feedback. I was working with John R on running 'oc cluster up'  and working with info coming back from this, which is:

Logged into "https://127.0.0.1:8443" as "system:admin" using existing credentials.

You have access to the following projects and can switch between them with 'oc project <projectname>':

    default
    kube-public
    kube-system
  * myproject
    openshift
    openshift-infra
    openshift-node


It looks like I am missing two services, Ansible being one.

@John R - the following is the error message from the image that was running at the very end of execution:

Failed to list *template.Template: Get https://172.30.0.1:443/apis/template.openshift.io/v1/templates?resourceVersion=0: dial tcp 172.30.0.1:443: getsockopt: no route to host




User "system:anonymous" cannot list templates.template.openshift.io at the cluster scope: User "system:anonymous" cannot list all templates.template.openshift.io in the cluster




On Tue, Nov 7, 2017 at 12:23 PM, John Frizelle <[hidden email]> wrote:
Hi John,

From looking at the output and my own experience, this is OpenShift itself not starting up correctly - "Error: failed to start the template service broker apiserver: timed out waiting for the condition"]}"

If you manually run oc cluster up, you often get a bit more info (and in a more readable format) than what is output from the ansible installer.

Since the problem here is specific to oc, we might end up needing input from some openshift folks.

Regards,
John.

--
John Frizelle
Chief Architect, Red Hat Mobile
Consulting Engineer

mobile: <a href="tel://+353872901644" target="_blank">+353 87 290 1644
twitter: @johnfriz
skype: john_frizelle




On 7 November 2017 at 12:08, John Stafford <[hidden email]> wrote:
Hi,

1) when I run "ansible-playbook playbook.yml -e "dockerhub_username=$DOCKERHUB_USERNAME" -e "dockerhub_password=$DOCKERHUB_PASSWORD" --ask-become-pass"

2) the execution gets as far as "TASK [oc-cluster-up : Cluster up] " and then terminates with an error[1] (after hanging for about 4-5 mins)

3) the docker image[2] in cockpit contains the the message "Unable to connect to the server: dial tcp 172.30.0.1:443: getsockopt: no route to host disconnected"

Any ideas on how to troubleshoot/work around this?

FYI - I completed a 'make clean' against the '/home/jstaffor/MCP/mcp-standalone/' folder and executed the command above again but unfortunately the same situation occurred.

[1]

“Unable to connect to the server: dial tcp 172.30.0.1:443: getsockopt: no route to host disconnected”


fatal: [localhost]: FAILED! => {"changed": false, "cmd": "oc cluster up --service-catalog=true --host-config-dir=/home/jstaffor/MCP/mcp-standalone/ui --host-data-dir=/home/jstaffor/MCP/mcp-standalone/ui/openshift-data --host-pv-dir=/home/jstaffor/MCP/mcp-standalone/ui/openshift-pvs --host-volumes-dir=/home/jstaffor/MCP/mcp-standalone/ui/openshift-volumes --routing-suffix=192.168.37.1.nip.io --public-hostname=192.168.37.1 --version=v3.7.0-rc.0 --image=openshift/origin", "delta": "0:10:42.022010", "end": "2017-11-07 11:45:48.927503", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2017-11-07 11:35:06.905493", "stderr": "", "stderr_lines": [], "stdout": "Starting OpenShift using openshift/origin:v3.7.0-rc.0 ...\n-- Checking OpenShift client ... OK\n-- Checking Docker client ... OK\n-- Checking Docker version ... OK\n-- Checking for existing OpenShift container ... OK\n-- Checking for openshift/origin:v3.7.0-rc.0 image ... OK\n-- Checking Docker daemon configuration ... OK\n-- Checking for available ports ... \n   WARNING: Binding DNS on port 8053 instead of 53, which may not be resolvable from all clients.\n-- Checking type of volume mount ... \n   Using nsenter mounter for OpenShift volumes\n-- Creating host directories ... OK\n-- Finding server IP ... \n   Using public hostname IP 192.168.37.1 as the host IP\n   Using 192.168.37.1 as the server IP\n-- Checking service catalog version requirements ... OK\n-- Starting OpenShift container ... \n   Creating initial OpenShift configuration\n   Starting OpenShift using container 'origin'\n   Waiting for API server to start listening\n   OpenShift server started\n-- Adding default OAuthClient redirect URIs ... OK\n-- Installing registry ... \n   scc \"privileged\" added to: [\"system:serviceaccount:default:registry\"]\n-- Installing router ... OK\n-- Importing image streams ... OK\n-- Importing templates ... OK\n-- Importing service catalog templates ... OK\n-- Installing service catalog ... OK\n-- Installing template service broker ... FAIL\n   Error: failed to start the template service broker apiserver: timed out waiting for the condition", "stdout_lines": ["Starting OpenShift using openshift/origin:v3.7.0-rc.0 ...", "-- Checking OpenShift client ... OK", "-- Checking Docker client ... OK", "-- Checking Docker version ... OK", "-- Checking for existing OpenShift container ... OK", "-- Checking for openshift/origin:v3.7.0-rc.0 image ... OK", "-- Checking Docker daemon configuration ... OK", "-- Checking for available ports ... ", "   WARNING: Binding DNS on port 8053 instead of 53, which may not be resolvable from all clients.", "-- Checking type of volume mount ... ", "   Using nsenter mounter for OpenShift volumes", "-- Creating host directories ... OK", "-- Finding server IP ... ", "   Using public hostname IP 192.168.37.1 as the host IP", "   Using 192.168.37.1 as the server IP", "-- Checking service catalog version requirements ... OK", "-- Starting OpenShift container ... ", "   Creating initial OpenShift configuration", "   Starting OpenShift using container 'origin'", "   Waiting for API server to start listening", "   OpenShift server started", "-- Adding default OAuthClient redirect URIs ... OK", "-- Installing registry ... ", "   scc \"privileged\" added to: [\"system:serviceaccount:default:registry\"]", "-- Installing router ... OK", "-- Importing image streams ... OK", "-- Importing templates ... OK", "-- Importing service catalog templates ... OK", "-- Installing service catalog ... OK", "-- Installing template service broker ... FAIL", "   Error: failed to start the template service broker apiserver: timed out waiting for the condition"]}

-- 

​[2]
Id:bc3e4b7320ec40571486521712ad33420f7c04b2f44e3b7ecdcaee0b768a3554
Created:2017-11-07T11:55:52.864111572Z
Image:
sha256:76d9e4ab142b310fa8adf1427f99f98fdc627ae8e460b92f55878ba68e6a9679
Command:/bin/bash -c "#/bin/bash set -e function generate_pv() { local basedir=\"${1}\" local name=\"${2}\" cat <<EOF apiVersion: v1 kind: PersistentVolume metadata: name: ${name} labels: volume: ${name} spec: capacity: storage: 100Gi accessModes: - ReadWriteOnce - ReadWriteMany - ReadOnlyMany hostPath: path: ${basedir}/${name} persistentVolumeReclaimPolicy: Recycle EOF } function setup_pv_dir() { local dir=\"${1}\" if [[ ! -d \"${dir}\" ]]; then mkdir -p \"${dir}\" fi if ! chcon -t svirt_sandbox_file_t \"${dir}\" &> /dev/null; then echo \"Not setting SELinux content for ${dir}\" fi chmod 770 \"${dir}\" } function create_pv() { local basedir=\"${1}\" local name=\"${2}\" setup_pv_dir \"${basedir}/${name}\" if ! oc get pv \"${name}\" &> /dev/null; then generate_pv \"${basedir}\" \"${name}\" | oc create -f - else echo \"persistentvolume ${name} already exists\" fi } basedir=\"/home/jstaffor/MCP/mcp-standalone/ui/openshift-pvs\" setup_pv_dir \"${basedir}/registry\" for i in $(seq -f \"%04g\" 1 100); do create_pv \"${basedir}\" \"pv${i}\" done "
State:Exited 1
Restart Policy:
IP Address:
IP Prefix Length:0
Gateway:
MAC Address:
 


JOHN STAFFORD

TECHNICAL WRITER, CCS

Red Hat 


_______________________________________________
feedhenry-dev mailing list
[hidden email]
https://www.redhat.com/mailman/listinfo/feedhenry-dev





--

JOHN STAFFORD

TECHNICAL WRITER, CCS

Red Hat 


_______________________________________________
feedhenry-dev mailing list
[hidden email]
https://www.redhat.com/mailman/listinfo/feedhenry-dev