[DISCUSS] Out of Band VR migration, should we reboot VR or not?

[Rohit Yadav]

Hi all,

Recently a behaviour was reported for ACS 4.5.1, where out of band VR migration would cause rebooting of VR. It seems this is a desired behaviour as per this issue: https://issues.apache.org/jira/browse/CLOUDSTACK-7994

It was shared on the thread on users ML that since CloudStack now supports aggregate commands for VR, this behaviour is unnecessary.

The VR in 4.5+ supports aggregated execution of commands on VRs to allows us to achieve eventual consistency of VR state without actually rebooting it, see this for details: https://issues.apache.org/jira/browse/CLOUDSTACK-6047

Please share your comments on whether if we should revert the fix or not, and the best way to do it. Thanks.

Regards,
Rohit Yadav
Software Architect, ShapeBlue
M. [phone] | ***@shapeblue.com
Blog: bhaisaab.org | Twitter: @_bhaisaab



Find out more about ShapeBlue and our range of CloudStack related services

IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from S

[Remi Bergsma]

Hi all,

If I understand correctly, this is also in 4.4.3 (and the upcoming 4.4.4). The report on user list: http://markmail.org/message/nth7qck27s2chwoc?q=VMsync+issues+with+VRs%3F<http://markmail.org/message/nth7qck27s2chwoc?q=VMsync+issues+with+VRs?>

As VPCs are non-redundant, a sudden reboot impacts users. I’d like to do this in a non-impacting way.

Why out-of-band migrations? When patching XenServer at scale, it is much faster to set the cluster to “unmanaged” in CloudStack and “evacuate” a XenServer host directly (i.e. making it empty by live-migrating), turn off HA, disable it, patch, reboot, enable, and do this for all hypervisors in the pool. Finally, turning HA on again and set CloudStack to “manage” the cluster again. I have scripted this, so we can automatically patch clusters at scale.

In CloudStack 4.4.2 this works and the database gets updated with the current information on where VMs live. In CloudStack versions which include this fix, CloudStack will reboot all routers “to be sure”. This prevents me from doing maintenance without impact.

I’d propose to either limit the fix to VMware, or come up with a way that does not require a reboot.
CLOUDSTACK-6047 sounds interesting, just curious how to do it in 4.4 for now.

Regards,
Remi


On 03 Jun 2015, at 13:48, Rohit Yadav <***@shapeblue.com<mailto:***@shapeblue.com>> wrote:

Hi all,

Recently a behaviour was reported for ACS 4.5.1, where out of band VR migration would cause rebooting of VR. It seems this is a desired behaviour as per this issue: https://issues.apache.org/jira/browse/CLOUDSTACK-7994

It was shared on the thread on users ML that since CloudStack now supports aggregate commands for VR, this behaviour is unnecessary.

The VR in 4.5+ supports aggregated execution of commands on VRs to allows us to achieve eventual consistency of VR state without actually rebooting it, see this for details: https://issues.apache.org/jira/browse/CLOUDSTACK-6047

Please share your comments on whether if we should revert the fix or not, and the best way to do it. Thanks.

Regards,
Rohit Yadav
Software Architect, ShapeBlue
M. [phone] | ***@shapeblue.com<mailto:***@shapeblue.com>
Blog: bhaisaab.org<http://bhaisaab.org> | Twitter: @_bhaisaab



Find out more about ShapeBlue and our range of CloudStack related services

IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.

[Sudhansu Sahu]

The aggregate command was introduced in 4.4 so we can implement this in
4.4 as well.


Regards
Sudhansu Sahu
CPG-Orchestration
T: [phone] | M: [phone]
***@citrix.com <mailto:%***@citrix.com>

<http://www.citrix.com>

Powering mobile workstyles and cloud services

Post by Rohit Yadav
Hi all,
If I understand correctly, this is also in 4.4.3 (and the upcoming
http://markmail.org/message/nth7qck27s2chwoc?q=VMsync+issues+with+VRs%3F<h
ttp://markmail.org/message/nth7qck27s2chwoc?q=VMsync+issues+with+VRs?>
As VPCs are non-redundant, a sudden reboot impacts users. I¹d like to do
this in a non-impacting way.
Why out-of-band migrations? When patching XenServer at scale, it is much
faster to set the cluster to ³unmanaged² in CloudStack and ³evacuate² a
XenServer host directly (i.e. making it empty by live-migrating), turn
off HA, disable it, patch, reboot, enable, and do this for all
hypervisors in the pool. Finally, turning HA on again and set CloudStack
to ³manage² the cluster again. I have scripted this, so we can
automatically patch clusters at scale.
In CloudStack 4.4.2 this works and the database gets updated with the
current information on where VMs live. In CloudStack versions which
include this fix, CloudStack will reboot all routers ³to be sure². This
prevents me from doing maintenance without impact.
I¹d propose to either limit the fix to VMware, or come up with a way that
does not require a reboot.
CLOUDSTACK-6047 sounds interesting, just curious how to do it in 4.4 for now.
Regards,
Remi
On 03 Jun 2015, at 13:48, Rohit Yadav
Hi all,
Recently a behaviour was reported for ACS 4.5.1, where out of band VR
migration would cause rebooting of VR. It seems this is a desired
https://issues.apache.org/jira/browse/CLOUDSTACK-7994
It was shared on the thread on users ML that since CloudStack now
supports aggregate commands for VR, this behaviour is unnecessary.
The VR in 4.5+ supports aggregated execution of commands on VRs to allows
us to achieve eventual consistency of VR state without actually rebooting
https://issues.apache.org/jira/browse/CLOUDSTACK-6047
Please share your comments on whether if we should revert the fix or not,
and the best way to do it. Thanks.
Regards,
Rohit Yadav
Software Architect, ShapeBlue
M. [phone] |
Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design &
Build<http://secure-web.cisco.com/1IT9RaYLUZQ7chW-ARcRoDX3w8TYuNItsW60qZQi
l7aWzHZFqJrFxNiP2VzZqELd-h6T1__-zg-Kyen2Zd69cUxRpIBmAWagY9rwoDCh5R7GHlYCcG
Rwi-D9DorUzAXVWz5n3oN2zwfSTW3ed4ocuu4UNsecm9oITVtWSL3QgVRpRHkT3UFf1zDYQXNR
G3XqR/http%3A%2F%2Fshapeblue.com%2Fiaas-cloud-design-and-build%2F%2F>
CSForge ­ rapid IaaS deployment
framework<http://secure-web.cisco.com/14UFU_a4mRs8mk63Pv_UPdDv4W0GDdbuUt5s
OS7E4ItlYkAlflu_KsLkcOuzACQ3y7NaLNJ7tYIf1c1IHeC92mLjh3izOh5RqEGP3SgnycXSxi
rsTFmNTfUBWtqDnHuRMYngTjUIoLbALtF1fhpX2FA3_vZBRmZJnumWhZTLXAASxXfMliY0MKD2
D2677-hCM/http%3A%2F%2Fshapeblue.com%2Fcsforge%2F>
CloudStack
Consulting<http://secure-web.cisco.com/14OAw2w9F6PgJ7tF5Ued8EsvlPYVACEpMvI
z1zvbYOynBFgSPev-ETG0lB7jxmMUDsgjlWDk0dwmrBFCedYIHx4QX45kG6w0Dfb_C53KJosYK
4Z8f2Qyec5x_Nljvn7D_ZfdjEfPl4xF6_iylDvj_80TL1ZBHOAIpPoZwDYiwNPUGSk-liakENF
jSikM93RrP/http%3A%2F%2Fshapeblue.com%2Fcloudstack-consultancy%2F>
CloudStack Software
Engineering<http://secure-web.cisco.com/1-9DE8xL74Rj9MTkhEw1oEcRxAawJfQcb2
nPTPaBPz-WyCU0TQKIStZ4ENC3UTOInjHv8ib3FVGNs8jYjBR4SEC1BOCnmLTx2QPbZQJ6ILdI
YLqTv8hbRO0hu8vXCLThbK9cmnUG--m-CwlJHl-u4Gnnt9aUqHvFnncH1J22tPVjB39Ue7zD1t
1bmAW9a8L-n/http%3A%2F%2Fshapeblue.com%2Fcloudstack-software-engineering%2
F>
CloudStack Infrastructure
Support<http://secure-web.cisco.com/1TaOtSW5oZP_85G2h7rPrOKHU9vpxKSi5GYu5_
LzzFlxgSP-l0961xIsF01DNsOleIWwEnzL87WCkADfsnCMFrYizGORiePG58AGdm9tNeFfwGPm
Tvljsh2YzPmSPG30N2IGjj7l-SvGWW2iqK7xYtN27YcH0xH4BJJWlBKWZkz1xrjTb09O_AtniQ
JZYUn9h/http%3A%2F%2Fshapeblue.com%2Fcloudstack-infrastructure-support%2F>
CloudStack Bootcamp Training
Courses<http://secure-web.cisco.com/1XNQSG5fM2DYgGy0yo40peRo2quLqxDy5jDend
sblnHT9UZvZuAdMawmT0oEZqCYjV_esxYJ2olrZX7ZXe6Ne9tBxpsxiaAeczVG1vXqhnZsSXyB
orW97iSLeKLXLYXHLsikCgDp5bGmR9JQ02_Mv61o0Flye31gMfK2x0tNbI9gjHEFMgMCDGaxY8
zWKG3zk/http%3A%2F%2Fshapeblue.com%2Fcloudstack-training%2F>
This email and any attachments to it may be confidential and are intended
solely for the use of the individual to whom it is addressed. Any views
or opinions expressed are solely those of the author and do not
necessarily represent those of Shape Blue Ltd or related companies. If
you are not the intended recipient of this email, you must neither take
any action based upon its contents, nor copy or show it to anyone. Please
contact the sender if you believe you have received this email in error.
Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
Services India LLP is a company incorporated in India and is operated
under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is
a company incorporated in Brasil and is operated under license from Shape
Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
South Africa and is traded under license from Shape Blue Ltd. ShapeBlue
is a registered trademark.

[Remi Bergsma]

That’s great, sounds like the best way to do it then :-)

Post by Sudhansu Sahu
The aggregate command was introduced in 4.4 so we can implement this in
4.4 as well.
Regards
Sudhansu Sahu
CPG-Orchestration
T: [phone] | M: [phone]
<http://www.citrix.com>
Powering mobile workstyles and cloud services

Post by Rohit Yadav
Hi all,
If I understand correctly, this is also in 4.4.3 (and the upcoming
http://markmail.org/message/nth7qck27s2chwoc?q=VMsync+issues+with+VRs%3F<h
ttp://markmail.org/message/nth7qck27s2chwoc?q=VMsync+issues+with+VRs?>
As VPCs are non-redundant, a sudden reboot impacts users. I¹d like to do
this in a non-impacting way.
Why out-of-band migrations? When patching XenServer at scale, it is much
faster to set the cluster to ³unmanaged² in CloudStack and ³evacuate² a
XenServer host directly (i.e. making it empty by live-migrating), turn
off HA, disable it, patch, reboot, enable, and do this for all
hypervisors in the pool. Finally, turning HA on again and set CloudStack
to ³manage² the cluster again. I have scripted this, so we can
automatically patch clusters at scale.
In CloudStack 4.4.2 this works and the database gets updated with the
current information on where VMs live. In CloudStack versions which
include this fix, CloudStack will reboot all routers ³to be sure². This
prevents me from doing maintenance without impact.
I¹d propose to either limit the fix to VMware, or come up with a way that
does not require a reboot.
CLOUDSTACK-6047 sounds interesting, just curious how to do it in 4.4 for now.
Regards,
Remi
On 03 Jun 2015, at 13:48, Rohit Yadav
Hi all,
Recently a behaviour was reported for ACS 4.5.1, where out of band VR
migration would cause rebooting of VR. It seems this is a desired
https://issues.apache.org/jira/browse/CLOUDSTACK-7994
It was shared on the thread on users ML that since CloudStack now
supports aggregate commands for VR, this behaviour is unnecessary.
The VR in 4.5+ supports aggregated execution of commands on VRs to allows
us to achieve eventual consistency of VR state without actually rebooting
https://issues.apache.org/jira/browse/CLOUDSTACK-6047
Please share your comments on whether if we should revert the fix or not,
and the best way to do it. Thanks.
Regards,
Rohit Yadav
Software Architect, ShapeBlue
M. [phone] |
Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design &
Build<http://secure-web.cisco.com/1IT9RaYLUZQ7chW-ARcRoDX3w8TYuNItsW60qZQi
l7aWzHZFqJrFxNiP2VzZqELd-h6T1__-zg-Kyen2Zd69cUxRpIBmAWagY9rwoDCh5R7GHlYCcG
Rwi-D9DorUzAXVWz5n3oN2zwfSTW3ed4ocuu4UNsecm9oITVtWSL3QgVRpRHkT3UFf1zDYQXNR
G3XqR/http%3A%2F%2Fshapeblue.com%2Fiaas-cloud-design-and-build%2F%2F>
CSForge ­ rapid IaaS deployment
framework<http://secure-web.cisco.com/14UFU_a4mRs8mk63Pv_UPdDv4W0GDdbuUt5s
OS7E4ItlYkAlflu_KsLkcOuzACQ3y7NaLNJ7tYIf1c1IHeC92mLjh3izOh5RqEGP3SgnycXSxi
rsTFmNTfUBWtqDnHuRMYngTjUIoLbALtF1fhpX2FA3_vZBRmZJnumWhZTLXAASxXfMliY0MKD2
D2677-hCM/http%3A%2F%2Fshapeblue.com%2Fcsforge%2F>
CloudStack
Consulting<http://secure-web.cisco.com/14OAw2w9F6PgJ7tF5Ued8EsvlPYVACEpMvI
z1zvbYOynBFgSPev-ETG0lB7jxmMUDsgjlWDk0dwmrBFCedYIHx4QX45kG6w0Dfb_C53KJosYK
4Z8f2Qyec5x_Nljvn7D_ZfdjEfPl4xF6_iylDvj_80TL1ZBHOAIpPoZwDYiwNPUGSk-liakENF
jSikM93RrP/http%3A%2F%2Fshapeblue.com%2Fcloudstack-consultancy%2F>
CloudStack Software
Engineering<http://secure-web.cisco.com/1-9DE8xL74Rj9MTkhEw1oEcRxAawJfQcb2
nPTPaBPz-WyCU0TQKIStZ4ENC3UTOInjHv8ib3FVGNs8jYjBR4SEC1BOCnmLTx2QPbZQJ6ILdI
YLqTv8hbRO0hu8vXCLThbK9cmnUG--m-CwlJHl-u4Gnnt9aUqHvFnncH1J22tPVjB39Ue7zD1t
1bmAW9a8L-n/http%3A%2F%2Fshapeblue.com%2Fcloudstack-software-engineering%2
F>
CloudStack Infrastructure
Support<http://secure-web.cisco.com/1TaOtSW5oZP_85G2h7rPrOKHU9vpxKSi5GYu5_
LzzFlxgSP-l0961xIsF01DNsOleIWwEnzL87WCkADfsnCMFrYizGORiePG58AGdm9tNeFfwGPm
Tvljsh2YzPmSPG30N2IGjj7l-SvGWW2iqK7xYtN27YcH0xH4BJJWlBKWZkz1xrjTb09O_AtniQ
JZYUn9h/http%3A%2F%2Fshapeblue.com%2Fcloudstack-infrastructure-support%2F>
CloudStack Bootcamp Training
Courses<http://secure-web.cisco.com/1XNQSG5fM2DYgGy0yo40peRo2quLqxDy5jDend
sblnHT9UZvZuAdMawmT0oEZqCYjV_esxYJ2olrZX7ZXe6Ne9tBxpsxiaAeczVG1vXqhnZsSXyB
orW97iSLeKLXLYXHLsikCgDp5bGmR9JQ02_Mv61o0Flye31gMfK2x0tNbI9gjHEFMgMCDGaxY8
zWKG3zk/http%3A%2F%2Fshapeblue.com%2Fcloudstack-training%2F>
This email and any attachments to it may be confidential and are intended
solely for the use of the individual to whom it is addressed. Any views
or opinions expressed are solely those of the author and do not
necessarily represent those of Shape Blue Ltd or related companies. If
you are not the intended recipient of this email, you must neither take
any action based upon its contents, nor copy or show it to anyone. Please
contact the sender if you believe you have received this email in error.
Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
Services India LLP is a company incorporated in India and is operated
under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is
a company incorporated in Brasil and is operated under license from Shape
Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
South Africa and is traded under license from Shape Blue Ltd. ShapeBlue
is a registered trademark.

[Daan Hoogland]

I think we should revert for 4.4 as it creates a serious regression. As
reverting it would create a regession for vmware users on 4.4.3 we must
create an alternative. For 4.5 we should use the aggregate command and as
Sudhansu says we can for 4.4. as well. that and for 4.6+ we have persistent
configuration in VRs.

That’s great, sounds like the best way to do it then :-)

Post by Sudhansu Sahu
The aggregate command was introduced in 4.4 so we can implement this in
4.4 as well.
Regards
Sudhansu Sahu
CPG-Orchestration
T: [phone] | M: [phone]
<http://www.citrix.com>
Powering mobile workstyles and cloud services

Post by Rohit Yadav
Hi all,
If I understand correctly, this is also in 4.4.3 (and the upcoming

http://markmail.org/message/nth7qck27s2chwoc?q=VMsync+issues+with+VRs%3F<h

Post by Sudhansu Sahu

Post by Rohit Yadav
ttp://markmail.org/message/nth7qck27s2chwoc?q=VMsync+issues+with+VRs?>
As VPCs are non-redundant, a sudden reboot impacts users. I¹d like to do
this in a non-impacting way.
Why out-of-band migrations? When patching XenServer at scale, it is much
faster to set the cluster to ³unmanaged² in CloudStack and ³evacuate² a
XenServer host directly (i.e. making it empty by live-migrating), turn
off HA, disable it, patch, reboot, enable, and do this for all
hypervisors in the pool. Finally, turning HA on again and set CloudStack
to ³manage² the cluster again. I have scripted this, so we can
automatically patch clusters at scale.
In CloudStack 4.4.2 this works and the database gets updated with the
current information on where VMs live. In CloudStack versions which
include this fix, CloudStack will reboot all routers ³to be sure². This
prevents me from doing maintenance without impact.
I¹d propose to either limit the fix to VMware, or come up with a way

that

Post by Sudhansu Sahu

Post by Rohit Yadav
does not require a reboot.
CLOUDSTACK-6047 sounds interesting, just curious how to do it in 4.4 for now.
Regards,
Remi
On 03 Jun 2015, at 13:48, Rohit Yadav
Hi all,
Recently a behaviour was reported for ACS 4.5.1, where out of band VR
migration would cause rebooting of VR. It seems this is a desired
https://issues.apache.org/jira/browse/CLOUDSTACK-7994
It was shared on the thread on users ML that since CloudStack now
supports aggregate commands for VR, this behaviour is unnecessary.
The VR in 4.5+ supports aggregated execution of commands on VRs to

allows

Post by Sudhansu Sahu

Post by Rohit Yadav
us to achieve eventual consistency of VR state without actually

rebooting

Post by Sudhansu Sahu

Post by Rohit Yadav
https://issues.apache.org/jira/browse/CLOUDSTACK-6047
Please share your comments on whether if we should revert the fix or

not,

Post by Sudhansu Sahu

Post by Rohit Yadav
and the best way to do it. Thanks.
Regards,
Rohit Yadav
Software Architect, ShapeBlue
M. [phone] |
Find out more about ShapeBlue and our range of CloudStack related

services

Post by Sudhansu Sahu

Post by Rohit Yadav
IaaS Cloud Design &
Build<

http://secure-web.cisco.com/1IT9RaYLUZQ7chW-ARcRoDX3w8TYuNItsW60qZQi
l7aWzHZFqJrFxNiP2VzZqELd-h6T1__-zg-Kyen2Zd69cUxRpIBmAWagY9rwoDCh5R7GHlYCcG
Rwi-D9DorUzAXVWz5n3oN2zwfSTW3ed4ocuu4UNsecm9oITVtWSL3QgVRpRHkT3UFf1zDYQXNR

Post by Sudhansu Sahu

Post by Rohit Yadav
G3XqR/http%3A%2F%2Fshapeblue.com%2Fiaas-cloud-design-and-build%2F%2F>
CSForge ­ rapid IaaS deployment
framework<

http://secure-web.cisco.com/14UFU_a4mRs8mk63Pv_UPdDv4W0GDdbuUt5s
OS7E4ItlYkAlflu_KsLkcOuzACQ3y7NaLNJ7tYIf1c1IHeC92mLjh3izOh5RqEGP3SgnycXSxi
rsTFmNTfUBWtqDnHuRMYngTjUIoLbALtF1fhpX2FA3_vZBRmZJnumWhZTLXAASxXfMliY0MKD2

Post by Sudhansu Sahu

Post by Rohit Yadav
D2677-hCM/http%3A%2F%2Fshapeblue.com%2Fcsforge%2F>
CloudStack
Consulting<

http://secure-web.cisco.com/14OAw2w9F6PgJ7tF5Ued8EsvlPYVACEpMvI
z1zvbYOynBFgSPev-ETG0lB7jxmMUDsgjlWDk0dwmrBFCedYIHx4QX45kG6w0Dfb_C53KJosYK
4Z8f2Qyec5x_Nljvn7D_ZfdjEfPl4xF6_iylDvj_80TL1ZBHOAIpPoZwDYiwNPUGSk-liakENF

Post by Sudhansu Sahu

Post by Rohit Yadav
jSikM93RrP/http%3A%2F%2Fshapeblue.com%2Fcloudstack-consultancy%2F>
CloudStack Software
Engineering<

http://secure-web.cisco.com/1-9DE8xL74Rj9MTkhEw1oEcRxAawJfQcb2
nPTPaBPz-WyCU0TQKIStZ4ENC3UTOInjHv8ib3FVGNs8jYjBR4SEC1BOCnmLTx2QPbZQJ6ILdI
YLqTv8hbRO0hu8vXCLThbK9cmnUG--m-CwlJHl-u4Gnnt9aUqHvFnncH1J22tPVjB39Ue7zD1t
1bmAW9a8L-n/http%3A%2F%2Fshapeblue.com%2Fcloudstack-software-engineering%2

Post by Sudhansu Sahu

Post by Rohit Yadav
F>
CloudStack Infrastructure
Support<

http://secure-web.cisco.com/1TaOtSW5oZP_85G2h7rPrOKHU9vpxKSi5GYu5_
LzzFlxgSP-l0961xIsF01DNsOleIWwEnzL87WCkADfsnCMFrYizGORiePG58AGdm9tNeFfwGPm
Tvljsh2YzPmSPG30N2IGjj7l-SvGWW2iqK7xYtN27YcH0xH4BJJWlBKWZkz1xrjTb09O_AtniQ
JZYUn9h/http%3A%2F%2Fshapeblue.com%2Fcloudstack-infrastructure-support%2F>

Post by Sudhansu Sahu

Post by Rohit Yadav
CloudStack Bootcamp Training
Courses<

http://secure-web.cisco.com/1XNQSG5fM2DYgGy0yo40peRo2quLqxDy5jDend
sblnHT9UZvZuAdMawmT0oEZqCYjV_esxYJ2olrZX7ZXe6Ne9tBxpsxiaAeczVG1vXqhnZsSXyB
orW97iSLeKLXLYXHLsikCgDp5bGmR9JQ02_Mv61o0Flye31gMfK2x0tNbI9gjHEFMgMCDGaxY8

Post by Sudhansu Sahu

Post by Rohit Yadav
zWKG3zk/http%3A%2F%2Fshapeblue.com%2Fcloudstack-training%2F>
This email and any attachments to it may be confidential and are

intended

Post by Sudhansu Sahu

Post by Rohit Yadav
solely for the use of the individual to whom it is addressed. Any views
or opinions expressed are solely those of the author and do not
necessarily represent those of Shape Blue Ltd or related companies. If
you are not the intended recipient of this email, you must neither take
any action based upon its contents, nor copy or show it to anyone.

Please

Post by Sudhansu Sahu

Post by Rohit Yadav
contact the sender if you believe you have received this email in error.
Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
Services India LLP is a company incorporated in India and is operated
under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is
a company incorporated in Brasil and is operated under license from

Shape

Post by Sudhansu Sahu

Post by Rohit Yadav
Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic

of

Post by Sudhansu Sahu

Post by Rohit Yadav
South Africa and is traded under license from Shape Blue Ltd. ShapeBlue
is a registered trademark.

[Sudhansu Sahu]

Instead of RebootTask we should implement an AgrregationExecutionTask
which will push the network rules.

If we revert the current change(RebootTask), without implementing
AgrregationExecutionTask then VR will not have a desired state(missing
rules) after HA.


Regards
Sudhansu Sahu
CPG-Orchestration
T: [phone] | M: [phone]
***@citrix.com <mailto:%***@citrix.com>

<http://www.citrix.com>

Powering mobile workstyles and cloud services

Post by Rohit Yadav
Hi all,
Recently a behaviour was reported for ACS 4.5.1, where out of band VR
migration would cause rebooting of VR. It seems this is a desired
https://issues.apache.org/jira/browse/CLOUDSTACK-7994
It was shared on the thread on users ML that since CloudStack now
supports aggregate commands for VR, this behaviour is unnecessary.
The VR in 4.5+ supports aggregated execution of commands on VRs to allows
us to achieve eventual consistency of VR state without actually rebooting
https://issues.apache.org/jira/browse/CLOUDSTACK-6047
Please share your comments on whether if we should revert the fix or not,
and the best way to do it. Thanks.
Regards,
Rohit Yadav
Software Architect, ShapeBlue
Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design &
Build<http://secure-web.cisco.com/1bbIQxZPXaFxSxrciKEB1oAY3Vx1oqSLqrCUTk-M
4Z6TR1CYdHZDWU4MYbL4C1gTpYCMgJYvDB-ZuAdhfJuTIQz49aGLvaa7ljlziBdfTp0l4L94vd
2TlvX4JdCtVuwUTenAe8FG-81zUcgcvuAuauh4AIXgtBk2X1BPknNmxs9RaeB5p4c4bGonBXnh
P7Z9I/http%3A%2F%2Fshapeblue.com%2Fiaas-cloud-design-and-build%2F%2F>
CSForge ­ rapid IaaS deployment
framework<http://secure-web.cisco.com/1NuRL5disPYgkPu0tQ2zeKAps2lcrHUScvrO
wGOd9LEij4ZBFgmJZeMSGMmHHlz5VUWz_umaBbvd38Z3j97a-iV6bgrm9HPSgLxM329zUSjNgD
K-i3lSKurbu-ZXRApmH3dyjM3-ZRKYYWywPt-1cp3IB5nuAl-iewlBO9Q8nAdmxSIdtILdUKv7
_2chQP_92/http%3A%2F%2Fshapeblue.com%2Fcsforge%2F>
CloudStack
Consulting<http://secure-web.cisco.com/1FsVJFEHLO8LddHEfPpJLbIzXZAZ76FwvQc
FSccz2nHMYUfdHy943VXRJ4ihxj-3XM1cLihwYpsMRPONWlcIyV9KQotH7fmExhj_e--JTNLiI
l61qah5jYmPEbVu314PxVOSjLPSUOOv_vYwnW8MUr1h0wcwltE_-Y_ZCYue5fftLLEXQwrkNsi
aWQwyJHziF/http%3A%2F%2Fshapeblue.com%2Fcloudstack-consultancy%2F>
CloudStack Software
Engineering<http://secure-web.cisco.com/1KRERGuVoj6zZOTmYG-VRrv2AJwg5UPzAX
WLzWVUCjXz-3cp7B86Fy4YrYl_JLYgoxVn0j88wAwXLEy9xBmm2OK8Vs6r0GS2hXdpEm-wuw3B
KwzURUdwS_5xcA74w6bNOeu4daywzUS-3X_6aGeb8JAMf_4YEcfIoXhYFY8ADg06khhhrlRwI8
ipuCoKAN9DL/http%3A%2F%2Fshapeblue.com%2Fcloudstack-software-engineering%2
F>
CloudStack Infrastructure
Support<http://secure-web.cisco.com/19vLu_3N-eW2worynFIv16NAgR0nRs9CV13xer
MUXvmUJxTaY4sRuLgHITwY16HqlruSFjTzBkYsLWAvEECWv1ofQ1G1mQ5mZjzTrDwkL7b0hViE
xIdjpNhSbf7Q-S9MIl5Myhdm3QVm5raqtdBkM7ijIoQ6mt22GKvkjfnqOrfz872VqqYjZ51oLD
ldCEUVQ/http%3A%2F%2Fshapeblue.com%2Fcloudstack-infrastructure-support%2F>
CloudStack Bootcamp Training
Courses<http://secure-web.cisco.com/14E9HdB2qYEGWrZXx-9WF7jUJrEEh90OnHIVKV
RmrYnxNK3H7le0LI6XawllVbjJVL__srKczYwa7I5Ty1lYnm9VB2ef5kw0jiMhgQc8tOwXEv9e
J9oli9TKT8GSDeoPL9Q8vPQ1gtMA0f18-Uua2kJ2DsR-N_8b_L8T7HjGddjnX7Lh4se9DUns20
mfr_EUi/http%3A%2F%2Fshapeblue.com%2Fcloudstack-training%2F>
This email and any attachments to it may be confidential and are intended
solely for the use of the individual to whom it is addressed. Any views
or opinions expressed are solely those of the author and do not
necessarily represent those of Shape Blue Ltd or related companies. If
you are not the intended recipient of this email, you must neither take
any action based upon its contents, nor copy or show it to anyone. Please
contact the sender if you believe you have received this email in error.
Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
Services India LLP is a company incorporated in India and is operated
under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is
a company incorporated in Brasil and is operated under license from Shape
Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
South Africa and is traded under license from Shape Blue Ltd. ShapeBlue
is a registered trademark.

[Koushik Das]

In case the VR is moved out of band (say as part of Vmware DRS), all network rules are lost. Rebooting VR from CS re-applies all the rules. Either the reboot is done manually from UI/API or automatically as was done as part of CLOUDSTACK-7994.

I haven't looked at the aggregate command. If it can be used to apply rules on VR without a reboot then that should work as well.

-Koushik

-----Original Message-----
From: Rohit Yadav [mailto:***@shapeblue.com]
Sent: Wednesday, 3 June 2015 17:18
To: dev
Subject: [DISCUSS] Out of Band VR migration, should we reboot VR or not?

Hi all,

Recently a behaviour was reported for ACS 4.5.1, where out of band VR migration would cause rebooting of VR. It seems this is a desired behaviour as per this issue: https://issues.apache.org/jira/browse/CLOUDSTACK-7994

It was shared on the thread on users ML that since CloudStack now supports aggregate commands for VR, this behaviour is unnecessary.

The VR in 4.5+ supports aggregated execution of commands on VRs to allows us to achieve eventual consistency of VR state without actually rebooting it, see this for details: https://issues.apache.org/jira/browse/CLOUDSTACK-6047

Please share your comments on whether if we should revert the fix or not, and the best way to do it. Thanks.

Regards,
Rohit Yadav
Software Architect, ShapeBlue
M. [phone] | ***@shapeblue.com
Blog: bhaisaab.org | Twitter: @_bhaisaab



Find out more about ShapeBlue and our range of CloudStack related services

IaaS Cloud Design & Build<http://secure-web.cisco.com/1dxwCI-iCtF6Z_nBrHijVPBNYijQKrOiFTNAyoWGJy4rj0SYtw0-E0Y0pAkH3D-NB0b01e1nt_XkrPm26uaNt3xXaVo4ZbAU7coVeFNM7kzqwOki5MLTTck7Y8OTbP1c-Zv9PMiy-M43LI_uAvDx-mT_q0vQcDZCt4fKB5W9A3-8Bdv5rzE0m03ik1kkWENKj/http%3A%2F%2Fshapeblue.com%2Fiaas-cloud-design-and-build%2F%2F>
CSForge – rapid IaaS deployment framework<http://secure-web.cisco.com/1KBMX-P7Uz8rk1CFZiKEJ6XOBtwCL6e4cxHjHhKOIeqCjjNcPHHrb0GoqH0CpXLUo42d8MsSS0X8jUVxeUQUuCEywQ8cHO7-KakvmeWrimKoqc3jlsmu_OgNQqdMRNwrO_uw71z7aOol5O-nshuUYfVgArT2kp2roAFkOUfEGQ2uyy18JLB5k2WhCMiyVyNs1/http%3A%2F%2Fshapeblue.com%2Fcsforge%2F>
CloudStack Consulting<http://secure-web.cisco.com/18B_NmraREMmC8vTWKXxDVFNi6uVPHolLmQtOC5pxOCqqVKFHBRYmXzR0e97TssbyoSyR90ruMa9xhxhyhPzd1nHvmYZXJWuiWx5eG0625EWlv3ZmrPpKJQcMrxeXCP4RXYkxVrnqKj-V0pH1AMKLwP5WRSrohlcx2oht8DsGOcyHmpFGdATnwmsBwW15ubts/http%3A%2F%2Fshapeblue.com%2Fcloudstack-consultancy%2F>
CloudStack Software Engineering<http://secure-web.cisco.com/1YTx4foSk2LL4qOkJ6hp8arX_tkHSgGLRhcrIh4tiJcMv4yOrG9LDzy23u-Of4to5Vw7pzPe7Z2BtX6UOObpoMHTu1Ygga8bXXaPX063jdDnTXVIeAzV86Qiya0QaVMLi8Yr7h619NJTW99-b2MVlgfrhYr2y0-xhqyX8x_qwxJsJ7ImNvcIeqtzluHsyEMwW/http%3A%2F%2Fshapeblue.com%2Fcloudstack-software-engineering%2F>
CloudStack Infrastructure Support<http://secure-web.cisco.com/1mWtUTi0OKF8weMnbfj_o1Xj1zQYn6dXlewUgkdu2L6UuSQi2hZdVPstsZ4PKm8seKRW9wKYnvF70hbKmhFdfk6XVguYW3ZXOBYvkNdiBK-yy31X-1Y4f2ESLclUvykFEl7X-hNmgBZkXpgP-uBL1ohAu36e0HtgdNE1zuX04JWN1YQzfKcYUHCb7GeKszF3x/http%3A%2F%2Fshapeblue.com%2Fcloudstack-infrastructure-support%2F>
CloudStack Bootcamp Training Courses<http://secure-web.cisco.com/16ybXVrdBe17I4mnCuuKp4rP7MHtjPD8iKc-dybhmY02WlxPCR05bGrGVQcpjdJyLH0Iw_X1oC5orlEkNVEpIQLjDWOdf3rsjI2WOQTECMHhA89ZuuE_96g0HQ-eLw_ylw4XsSOUq6KPyLjYryreJO38qidzmHjY6bxYb1MpUr9E7sCwzq_XBVDBdZFH5gdWF/http%3A%2F%2Fshapeblue.com%2Fcloudstack-training%2F>

This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape

[Remi Bergsma]

This is a VMware specific problem, right?
We could also limit the fix to this hypervisor only.

Regards,
Remi


On 03 Jun 2015, at 14:12, Koushik Das <***@citrix.com<mailto:***@citrix.com>> wrote:

In case the VR is moved out of band (say as part of Vmware DRS), all network rules are lost. Rebooting VR from CS re-applies all the rules. Either the reboot is done manually from UI/API or automatically as was done as part of CLOUDSTACK-7994.

I haven't looked at the aggregate command. If it can be used to apply rules on VR without a reboot then that should work as well.

-Koushik

-----Original Message-----
From: Rohit Yadav [mailto:***@shapeblue.com]
Sent: Wednesday, 3 June 2015 17:18
To: dev
Subject: [DISCUSS] Out of Band VR migration, should we reboot VR or not?

Hi all,

Recently a behaviour was reported for ACS 4.5.1, where out of band VR migration would cause rebooting of VR. It seems this is a desired behaviour as per this issue: https://issues.apache.org/jira/browse/CLOUDSTACK-7994

It was shared on the thread on users ML that since CloudStack now supports aggregate commands for VR, this behaviour is unnecessary.

The VR in 4.5+ supports aggregated execution of commands on VRs to allows us to achieve eventual consistency of VR state without actually rebooting it, see this for details: https://issues.apache.org/jira/browse/CLOUDSTACK-6047

Please share your comments on whether if we should revert the fix or not, and the best way to do it. Thanks.

Regards,
Rohit Yadav
Software Architect, ShapeBlue
M. [phone] | ***@shapeblue.com<mailto:***@shapeblue.com>
Blog: bhaisaab.org<http://bhaisaab.org> | Twitter: @_bhaisaab



Find out more about ShapeBlue and our range of CloudStack related services

IaaS Cloud Design & Build<http://secure-web.cisco.com/1dxwCI-iCtF6Z_nBrHijVPBNYijQKrOiFTNAyoWGJy4rj0SYtw0-E0Y0pAkH3D-NB0b01e1nt_XkrPm26uaNt3xXaVo4ZbAU7coVeFNM7kzqwOki5MLTTck7Y8OTbP1c-Zv9PMiy-M43LI_uAvDx-mT_q0vQcDZCt4fKB5W9A3-8Bdv5rzE0m03ik1kkWENKj/http%3A%2F%2Fshapeblue.com%2Fiaas-cloud-design-and-build%2F%2F>
CSForge – rapid IaaS deployment framework<http://secure-web.cisco.com/1KBMX-P7Uz8rk1CFZiKEJ6XOBtwCL6e4cxHjHhKOIeqCjjNcPHHrb0GoqH0CpXLUo42d8MsSS0X8jUVxeUQUuCEywQ8cHO7-KakvmeWrimKoqc3jlsmu_OgNQqdMRNwrO_uw71z7aOol5O-nshuUYfVgArT2kp2roAFkOUfEGQ2uyy18JLB5k2WhCMiyVyNs1/http%3A%2F%2Fshapeblue.com%2Fcsforge%2F>
CloudStack Consulting<http://secure-web.cisco.com/18B_NmraREMmC8vTWKXxDVFNi6uVPHolLmQtOC5pxOCqqVKFHBRYmXzR0e97TssbyoSyR90ruMa9xhxhyhPzd1nHvmYZXJWuiWx5eG0625EWlv3ZmrPpKJQcMrxeXCP4RXYkxVrnqKj-V0pH1AMKLwP5WRSrohlcx2oht8DsGOcyHmpFGdATnwmsBwW15ubts/http%3A%2F%2Fshapeblue.com%2Fcloudstack-consultancy%2F>
CloudStack Software Engineering<http://secure-web.cisco.com/1YTx4foSk2LL4qOkJ6hp8arX_tkHSgGLRhcrIh4tiJcMv4yOrG9LDzy23u-Of4to5Vw7pzPe7Z2BtX6UOObpoMHTu1Ygga8bXXaPX063jdDnTXVIeAzV86Qiya0QaVMLi8Yr7h619NJTW99-b2MVlgfrhYr2y0-xhqyX8x_qwxJsJ7ImNvcIeqtzluHsyEMwW/http%3A%2F%2Fshapeblue.com%2Fcloudstack-software-engineering%2F>
CloudStack Infrastructure Support<http://secure-web.cisco.com/1mWtUTi0OKF8weMnbfj_o1Xj1zQYn6dXlewUgkdu2L6UuSQi2hZdVPstsZ4PKm8seKRW9wKYnvF70hbKmhFdfk6XVguYW3ZXOBYvkNdiBK-yy31X-1Y4f2ESLclUvykFEl7X-hNmgBZkXpgP-uBL1ohAu36e0HtgdNE1zuX04JWN1YQzfKcYUHCb7GeKszF3x/http%3A%2F%2Fshapeblue.com%2Fcloudstack-infrastructure-support%2F>
CloudStack Bootcamp Training Courses<http://secure-web.cisco.com/16ybXVrdBe17I4mnCuuKp4rP7MHtjPD8iKc-dybhmY02WlxPCR05bGrGVQcpjdJyLH0Iw_X1oC5orlEkNVEpIQLjDWOdf3rsjI2WOQTECMHhA89ZuuE_96g0HQ-eLw_ylw4XsSOUq6KPyLjYryreJO38qidzmHjY6bxYb1MpUr9E7sCwzq_XBVDBdZFH5gdWF/http%3A%2F%2Fshapeblue.com%2Fcloudstack-training%2F>

This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.

[Koushik Das]

I think as a design principle we shouldn't introduce HV specific checks in the orchestration/API layers.
I am not sure if the problem is specific to Vmware. Any out of band VR movement can lead to this issue. For now it is seen in Vmware but what about other HVs where CS relies on native HA provided by HV.

-----Original Message-----
From: Remi Bergsma [mailto:***@schubergphilis.com]
Sent: Wednesday, 3 June 2015 17:54
To: ***@cloudstack.apache.org
Subject: Re: [DISCUSS] Out of Band VR migration, should we reboot VR or not?

This is a VMware specific problem, right?
We could also limit the fix to this hypervisor only.

Regards,
Remi


On 03 Jun 2015, at 14:12, Koushik Das <***@citrix.com<mailto:***@citrix.com>> wrote:

In case the VR is moved out of band (say as part of Vmware DRS), all network rules are lost. Rebooting VR from CS re-applies all the rules. Either the reboot is done manually from UI/API or automatically as was done as part of CLOUDSTACK-7994.

I haven't looked at the aggregate command. If it can be used to apply rules on VR without a reboot then that should work as well.

-Koushik

-----Original Message-----
From: Rohit Yadav [mailto:***@shapeblue.com]
Sent: Wednesday, 3 June 2015 17:18
To: dev
Subject: [DISCUSS] Out of Band VR migration, should we reboot VR or not?

Hi all,

Recently a behaviour was reported for ACS 4.5.1, where out of band VR migration would cause rebooting of VR. It seems this is a desired behaviour as per this issue: https://issues.apache.org/jira/browse/CLOUDSTACK-7994

It was shared on the thread on users ML that since CloudStack now supports aggregate commands for VR, this behaviour is unnecessary.

The VR in 4.5+ supports aggregated execution of commands on VRs to allows us to achieve eventual consistency of VR state without actually rebooting it, see this for details: https://issues.apache.org/jira/browse/CLOUDSTACK-6047

Please share your comments on whether if we should revert the fix or not, and the best way to do it. Thanks.

Regards,
Rohit Yadav
Software Architect, ShapeBlue
M. [phone] | ***@shapeblue.com<mailto:***@shapeblue.com>
Blog: bhaisaab.org<http://bhaisaab.org> | Twitter: @_bhaisaab



Find out more about ShapeBlue and our range of CloudStack related services

IaaS Cloud Design & Build<http://secure-web.cisco.com/1dxwCI-iCtF6Z_nBrHijVPBNYijQKrOiFTNAyoWGJy4rj0SYtw0-E0Y0pAkH3D-NB0b01e1nt_XkrPm26uaNt3xXaVo4ZbAU7coVeFNM7kzqwOki5MLTTck7Y8OTbP1c-Zv9PMiy-M43LI_uAvDx-mT_q0vQcDZCt4fKB5W9A3-8Bdv5rzE0m03ik1kkWENKj/http%3A%2F%2Fshapeblue.com%2Fiaas-cloud-design-and-build%2F%2F>
CSForge – rapid IaaS deployment framework<http://secure-web.cisco.com/1KBMX-P7Uz8rk1CFZiKEJ6XOBtwCL6e4cxHjHhKOIeqCjjNcPHHrb0GoqH0CpXLUo42d8MsSS0X8jUVxeUQUuCEywQ8cHO7-KakvmeWrimKoqc3jlsmu_OgNQqdMRNwrO_uw71z7aOol5O-nshuUYfVgArT2kp2roAFkOUfEGQ2uyy18JLB5k2WhCMiyVyNs1/http%3A%2F%2Fshapeblue.com%2Fcsforge%2F>
CloudStack Consulting<http://secure-web.cisco.com/18B_NmraREMmC8vTWKXxDVFNi6uVPHolLmQtOC5pxOCqqVKFHBRYmXzR0e97TssbyoSyR90ruMa9xhxhyhPzd1nHvmYZXJWuiWx5eG0625EWlv3ZmrPpKJQcMrxeXCP4RXYkxVrnqKj-V0pH1AMKLwP5WRSrohlcx2oht8DsGOcyHmpFGdATnwmsBwW15ubts/http%3A%2F%2Fshapeblue.com%2Fcloudstack-consultancy%2F>
CloudStack Software Engineering<http://secure-web.cisco.com/1YTx4foSk2LL4qOkJ6hp8arX_tkHSgGLRhcrIh4tiJcMv4yOrG9LDzy23u-Of4to5Vw7pzPe7Z2BtX6UOObpoMHTu1Ygga8bXXaPX063jdDnTXVIeAzV86Qiya0QaVMLi8Yr7h619NJTW99-b2MVlgfrhYr2y0-xhqyX8x_qwxJsJ7ImNvcIeqtzluHsyEMwW/http%3A%2F%2Fshapeblue.com%2Fcloudstack-software-engineering%2F>
CloudStack Infrastructure Support<http://secure-web.cisco.com/1mWtUTi0OKF8weMnbfj_o1Xj1zQYn6dXlewUgkdu2L6UuSQi2hZdVPstsZ4PKm8seKRW9wKYnvF70hbKmhFdfk6XVguYW3ZXOBYvkNdiBK-yy31X-1Y4f2ESLclUvykFEl7X-hNmgBZkXpgP-uBL1ohAu36e0HtgdNE1zuX04JWN1YQzfKcYUHCb7GeKszF3x/http%3A%2F%2Fshapeblue.com%2Fcloudstack-infrastructure-support%2F>
CloudStack Bootcamp Training Courses<http://secure-web.cisco.com/16ybXVrdBe17I4mnCuuKp4rP7MHtjPD8iKc-dybhmY02WlxPCR05bGrGVQcpjdJyLH0Iw_X1oC5orlEkNVEpIQLjDWOdf3rsjI2WOQTECMHhA89ZuuE_96g0HQ-eLw_ylw4XsSOUq6KPyLjYryreJO38qidzmHjY6bxYb1MpUr9E7sCwzq_XBVDBdZFH5gdWF/http%3A%2F%2Fshapeblue.com%2Fcloudstack-training%2F>

This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a register

[Daan Hoogland]

that does negate the regression in xen, i'm afraid

Post by Koushik Das
I think as a design principle we shouldn't introduce HV specific checks in
the orchestration/API layers.
I am not sure if the problem is specific to Vmware. Any out of band VR
movement can lead to this issue. For now it is seen in Vmware but what
about other HVs where CS relies on native HA provided by HV.
-----Original Message-----
Sent: Wednesday, 3 June 2015 17:54
Subject: Re: [DISCUSS] Out of Band VR migration, should we reboot VR or not?
This is a VMware specific problem, right?
We could also limit the fix to this hypervisor only.
Regards,
Remi
In case the VR is moved out of band (say as part of Vmware DRS), all
network rules are lost. Rebooting VR from CS re-applies all the rules.
Either the reboot is done manually from UI/API or automatically as was done
as part of CLOUDSTACK-7994.
I haven't looked at the aggregate command. If it can be used to apply
rules on VR without a reboot then that should work as well.
-Koushik
-----Original Message-----
Sent: Wednesday, 3 June 2015 17:18
To: dev
Subject: [DISCUSS] Out of Band VR migration, should we reboot VR or not?
Hi all,
Recently a behaviour was reported for ACS 4.5.1, where out of band VR
migration would cause rebooting of VR. It seems this is a desired behaviour
as per this issue: https://issues.apache.org/jira/browse/CLOUDSTACK-7994
It was shared on the thread on users ML that since CloudStack now supports
aggregate commands for VR, this behaviour is unnecessary.
The VR in 4.5+ supports aggregated execution of commands on VRs to allows
us to achieve eventual consistency of VR state without actually rebooting
https://issues.apache.org/jira/browse/CLOUDSTACK-6047
Please share your comments on whether if we should revert the fix or not,
and the best way to do it. Thanks.
Regards,
Rohit Yadav
Software Architect, ShapeBlue
Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design & Build<
http://secure-web.cisco.com/1dxwCI-iCtF6Z_nBrHijVPBNYijQKrOiFTNAyoWGJy4rj0SYtw0-E0Y0pAkH3D-NB0b01e1nt_XkrPm26uaNt3xXaVo4ZbAU7coVeFNM7kzqwOki5MLTTck7Y8OTbP1c-Zv9PMiy-M43LI_uAvDx-mT_q0vQcDZCt4fKB5W9A3-8Bdv5rzE0m03ik1kkWENKj/http%3A%2F%2Fshapeblue.com%2Fiaas-cloud-design-and-build%2F%2F
CSForge – rapid IaaS deployment framework<
http://secure-web.cisco.com/1KBMX-P7Uz8rk1CFZiKEJ6XOBtwCL6e4cxHjHhKOIeqCjjNcPHHrb0GoqH0CpXLUo42d8MsSS0X8jUVxeUQUuCEywQ8cHO7-KakvmeWrimKoqc3jlsmu_OgNQqdMRNwrO_uw71z7aOol5O-nshuUYfVgArT2kp2roAFkOUfEGQ2uyy18JLB5k2WhCMiyVyNs1/http%3A%2F%2Fshapeblue.com%2Fcsforge%2F
CloudStack Consulting<
http://secure-web.cisco.com/18B_NmraREMmC8vTWKXxDVFNi6uVPHolLmQtOC5pxOCqqVKFHBRYmXzR0e97TssbyoSyR90ruMa9xhxhyhPzd1nHvmYZXJWuiWx5eG0625EWlv3ZmrPpKJQcMrxeXCP4RXYkxVrnqKj-V0pH1AMKLwP5WRSrohlcx2oht8DsGOcyHmpFGdATnwmsBwW15ubts/http%3A%2F%2Fshapeblue.com%2Fcloudstack-consultancy%2F
CloudStack Software Engineering<
http://secure-web.cisco.com/1YTx4foSk2LL4qOkJ6hp8arX_tkHSgGLRhcrIh4tiJcMv4yOrG9LDzy23u-Of4to5Vw7pzPe7Z2BtX6UOObpoMHTu1Ygga8bXXaPX063jdDnTXVIeAzV86Qiya0QaVMLi8Yr7h619NJTW99-b2MVlgfrhYr2y0-xhqyX8x_qwxJsJ7ImNvcIeqtzluHsyEMwW/http%3A%2F%2Fshapeblue.com%2Fcloudstack-software-engineering%2F
CloudStack Infrastructure Support<
http://secure-web.cisco.com/1mWtUTi0OKF8weMnbfj_o1Xj1zQYn6dXlewUgkdu2L6UuSQi2hZdVPstsZ4PKm8seKRW9wKYnvF70hbKmhFdfk6XVguYW3ZXOBYvkNdiBK-yy31X-1Y4f2ESLclUvykFEl7X-hNmgBZkXpgP-uBL1ohAu36e0HtgdNE1zuX04JWN1YQzfKcYUHCb7GeKszF3x/http%3A%2F%2Fshapeblue.com%2Fcloudstack-infrastructure-support%2F
CloudStack Bootcamp Training Courses<
http://secure-web.cisco.com/16ybXVrdBe17I4mnCuuKp4rP7MHtjPD8iKc-dybhmY02WlxPCR05bGrGVQcpjdJyLH0Iw_X1oC5orlEkNVEpIQLjDWOdf3rsjI2WOQTECMHhA89ZuuE_96g0HQ-eLw_ylw4XsSOUq6KPyLjYryreJO38qidzmHjY6bxYb1MpUr9E7sCwzq_XBVDBdZFH5gdWF/http%3A%2F%2Fshapeblue.com%2Fcloudstack-training%2F
This email and any attachments to it may be confidential and are intended
solely for the use of the individual to whom it is addressed. Any views or
opinions expressed are solely those of the author and do not necessarily
represent those of Shape Blue Ltd or related companies. If you are not the
intended recipient of this email, you must neither take any action based
upon its contents, nor copy or show it to anyone. Please contact the sender
if you believe you have received this email in error. Shape Blue Ltd is a
company incorporated in England & Wales. ShapeBlue Services India LLP is a
company incorporated in India and is operated under license from Shape Blue
Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil
and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is
a company registered by The Republic of South Africa and is traded under
license from Shape Blue Ltd. ShapeBlue is a registered trademark.

[Koushik Das]

Is there any issue reported in XS due to this?

-----Original Message-----
From: Daan Hoogland [mailto:***@gmail.com]
Sent: Wednesday, 3 June 2015 18:10
To: ***@cloudstack.apache.org
Subject: Re: [DISCUSS] Out of Band VR migration, should we reboot VR or not?

that does negate the regression in xen, i'm afraid

Post by Koushik Das
I think as a design principle we shouldn't introduce HV specific
checks in the orchestration/API layers.
I am not sure if the problem is specific to Vmware. Any out of band VR
movement can lead to this issue. For now it is seen in Vmware but what
about other HVs where CS relies on native HA provided by HV.
-----Original Message-----
Sent: Wednesday, 3 June 2015 17:54
Subject: Re: [DISCUSS] Out of Band VR migration, should we reboot VR or not?
This is a VMware specific problem, right?
We could also limit the fix to this hypervisor only.
Regards,
Remi
In case the VR is moved out of band (say as part of Vmware DRS), all
network rules are lost. Rebooting VR from CS re-applies all the rules.
Either the reboot is done manually from UI/API or automatically as was
done as part of CLOUDSTACK-7994.
I haven't looked at the aggregate command. If it can be used to apply
rules on VR without a reboot then that should work as well.
-Koushik
-----Original Message-----
Sent: Wednesday, 3 June 2015 17:18
To: dev
Subject: [DISCUSS] Out of Band VR migration, should we reboot VR or not?
Hi all,
Recently a behaviour was reported for ACS 4.5.1, where out of band VR
migration would cause rebooting of VR. It seems this is a desired
https://issues.apache.org/jira/browse/CLOUDSTACK-7994
It was shared on the thread on users ML that since CloudStack now
supports aggregate commands for VR, this behaviour is unnecessary.
The VR in 4.5+ supports aggregated execution of commands on VRs to
allows us to achieve eventual consistency of VR state without actually
https://issues.apache.org/jira/browse/CLOUDSTACK-6047
Please share your comments on whether if we should revert the fix or
not, and the best way to do it. Thanks.
Regards,
Rohit Yadav
Software Architect, ShapeBlue
Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design & Build<
http://secure-web.cisco.com/1dxwCI-iCtF6Z_nBrHijVPBNYijQKrOiFTNAyoWGJy
4rj0SYtw0-E0Y0pAkH3D-NB0b01e1nt_XkrPm26uaNt3xXaVo4ZbAU7coVeFNM7kzqwOki
5MLTTck7Y8OTbP1c-Zv9PMiy-M43LI_uAvDx-mT_q0vQcDZCt4fKB5W9A3-8Bdv5rzE0m0
3ik1kkWENKj/http%3A%2F%2Fshapeblue.com%2Fiaas-cloud-design-and-build%2
F%2F
CSForge – rapid IaaS deployment framework<
http://secure-web.cisco.com/1KBMX-P7Uz8rk1CFZiKEJ6XOBtwCL6e4cxHjHhKOIe
qCjjNcPHHrb0GoqH0CpXLUo42d8MsSS0X8jUVxeUQUuCEywQ8cHO7-KakvmeWrimKoqc3j
lsmu_OgNQqdMRNwrO_uw71z7aOol5O-nshuUYfVgArT2kp2roAFkOUfEGQ2uyy18JLB5k2
WhCMiyVyNs1/http%3A%2F%2Fshapeblue.com%2Fcsforge%2F
CloudStack Consulting<
http://secure-web.cisco.com/18B_NmraREMmC8vTWKXxDVFNi6uVPHolLmQtOC5pxO
CqqVKFHBRYmXzR0e97TssbyoSyR90ruMa9xhxhyhPzd1nHvmYZXJWuiWx5eG0625EWlv3Z
mrPpKJQcMrxeXCP4RXYkxVrnqKj-V0pH1AMKLwP5WRSrohlcx2oht8DsGOcyHmpFGdATnw
msBwW15ubts/http%3A%2F%2Fshapeblue.com%2Fcloudstack-consultancy%2F
CloudStack Software Engineering<
http://secure-web.cisco.com/1YTx4foSk2LL4qOkJ6hp8arX_tkHSgGLRhcrIh4tiJ
cMv4yOrG9LDzy23u-Of4to5Vw7pzPe7Z2BtX6UOObpoMHTu1Ygga8bXXaPX063jdDnTXVI
eAzV86Qiya0QaVMLi8Yr7h619NJTW99-b2MVlgfrhYr2y0-xhqyX8x_qwxJsJ7ImNvcIeq
tzluHsyEMwW/http%3A%2F%2Fshapeblue.com%2Fcloudstack-software-engineeri
ng%2F
CloudStack Infrastructure Support<
http://secure-web.cisco.com/1mWtUTi0OKF8weMnbfj_o1Xj1zQYn6dXlewUgkdu2L
6UuSQi2hZdVPstsZ4PKm8seKRW9wKYnvF70hbKmhFdfk6XVguYW3ZXOBYvkNdiBK-yy31X
-1Y4f2ESLclUvykFEl7X-hNmgBZkXpgP-uBL1ohAu36e0HtgdNE1zuX04JWN1YQzfKcYUH
Cb7GeKszF3x/http%3A%2F%2Fshapeblue.com%2Fcloudstack-infrastructure-sup
port%2F
CloudStack Bootcamp Training Courses<
http://secure-web.cisco.com/16ybXVrdBe17I4mnCuuKp4rP7MHtjPD8iKc-dybhmY
02WlxPCR05bGrGVQcpjdJyLH0Iw_X1oC5orlEkNVEpIQLjDWOdf3rsjI2WOQTECMHhA89Z
uuE_96g0HQ-eLw_ylw4XsSOUq6KPyLjYryreJO38qidzmHjY6bxYb1MpUr9E7sCwzq_XBV
DBdZFH5gdWF/http%3A%2F%2Fshapeblue.com%2Fcloudstack-training%2F
This email and any attachments to it may be confidential and are
intended solely for the use of the individual to whom it is addressed.
Any views or opinions expressed are solely those of the author and do
not necessarily represent those of Shape Blue Ltd or related
companies. If you are not the intended recipient of this email, you
must neither take any action based upon its contents, nor copy or show
it to anyone. Please contact the sender if you believe you have
received this email in error. Shape Blue Ltd is a company incorporated
in England & Wales. ShapeBlue Services India LLP is a company
incorporated in India and is operated under license from Shape Blue
Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA
Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. Shap

[Daan Hoogland]

Koushik,

See http://markmail.org/message/6ofb74lzemq2byhw

Post by Koushik Das
Is there any issue reported in XS due to this?
-----Original Message-----
Sent: Wednesday, 3 June 2015 18:10
Subject: Re: [DISCUSS] Out of Band VR migration, should we reboot VR or not?
that does negate the regression in xen, i'm afraid

Post by Koushik Das
I think as a design principle we shouldn't introduce HV specific
checks in the orchestration/API layers.
I am not sure if the problem is specific to Vmware. Any out of band VR
movement can lead to this issue. For now it is seen in Vmware but what
about other HVs where CS relies on native HA provided by HV.
-----Original Message-----
Sent: Wednesday, 3 June 2015 17:54
Subject: Re: [DISCUSS] Out of Band VR migration, should we reboot VR or not?
This is a VMware specific problem, right?
We could also limit the fix to this hypervisor only.
Regards,
Remi
In case the VR is moved out of band (say as part of Vmware DRS), all
network rules are lost. Rebooting VR from CS re-applies all the rules.
Either the reboot is done manually from UI/API or automatically as was
done as part of CLOUDSTACK-7994.
I haven't looked at the aggregate command. If it can be used to apply
rules on VR without a reboot then that should work as well.
-Koushik
-----Original Message-----
Sent: Wednesday, 3 June 2015 17:18
To: dev
Subject: [DISCUSS] Out of Band VR migration, should we reboot VR or not?
Hi all,
Recently a behaviour was reported for ACS 4.5.1, where out of band VR
migration would cause rebooting of VR. It seems this is a desired
https://issues.apache.org/jira/browse/CLOUDSTACK-7994
It was shared on the thread on users ML that since CloudStack now
supports aggregate commands for VR, this behaviour is unnecessary.
The VR in 4.5+ supports aggregated execution of commands on VRs to
allows us to achieve eventual consistency of VR state without actually
https://issues.apache.org/jira/browse/CLOUDSTACK-6047
Please share your comments on whether if we should revert the fix or
not, and the best way to do it. Thanks.
Regards,
Rohit Yadav
Software Architect, ShapeBlue
Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design & Build<
http://secure-web.cisco.com/1dxwCI-iCtF6Z_nBrHijVPBNYijQKrOiFTNAyoWGJy
4rj0SYtw0-E0Y0pAkH3D-NB0b01e1nt_XkrPm26uaNt3xXaVo4ZbAU7coVeFNM7kzqwOki
5MLTTck7Y8OTbP1c-Zv9PMiy-M43LI_uAvDx-mT_q0vQcDZCt4fKB5W9A3-8Bdv5rzE0m0
3ik1kkWENKj/http%3A%2F%2Fshapeblue.com%2Fiaas-cloud-design-and-build%2
F%2F
CSForge – rapid IaaS deployment framework<
http://secure-web.cisco.com/1KBMX-P7Uz8rk1CFZiKEJ6XOBtwCL6e4cxHjHhKOIe
qCjjNcPHHrb0GoqH0CpXLUo42d8MsSS0X8jUVxeUQUuCEywQ8cHO7-KakvmeWrimKoqc3j
lsmu_OgNQqdMRNwrO_uw71z7aOol5O-nshuUYfVgArT2kp2roAFkOUfEGQ2uyy18JLB5k2
WhCMiyVyNs1/http%3A%2F%2Fshapeblue.com%2Fcsforge%2F
CloudStack Consulting<
http://secure-web.cisco.com/18B_NmraREMmC8vTWKXxDVFNi6uVPHolLmQtOC5pxO
CqqVKFHBRYmXzR0e97TssbyoSyR90ruMa9xhxhyhPzd1nHvmYZXJWuiWx5eG0625EWlv3Z
mrPpKJQcMrxeXCP4RXYkxVrnqKj-V0pH1AMKLwP5WRSrohlcx2oht8DsGOcyHmpFGdATnw
msBwW15ubts/http%3A%2F%2Fshapeblue.com%2Fcloudstack-consultancy%2F
CloudStack Software Engineering<
http://secure-web.cisco.com/1YTx4foSk2LL4qOkJ6hp8arX_tkHSgGLRhcrIh4tiJ
cMv4yOrG9LDzy23u-Of4to5Vw7pzPe7Z2BtX6UOObpoMHTu1Ygga8bXXaPX063jdDnTXVI
eAzV86Qiya0QaVMLi8Yr7h619NJTW99-b2MVlgfrhYr2y0-xhqyX8x_qwxJsJ7ImNvcIeq
tzluHsyEMwW/http%3A%2F%2Fshapeblue.com%2Fcloudstack-software-engineeri
ng%2F
CloudStack Infrastructure Support<
http://secure-web.cisco.com/1mWtUTi0OKF8weMnbfj_o1Xj1zQYn6dXlewUgkdu2L
6UuSQi2hZdVPstsZ4PKm8seKRW9wKYnvF70hbKmhFdfk6XVguYW3ZXOBYvkNdiBK-yy31X
-1Y4f2ESLclUvykFEl7X-hNmgBZkXpgP-uBL1ohAu36e0HtgdNE1zuX04JWN1YQzfKcYUH
Cb7GeKszF3x/http%3A%2F%2Fshapeblue.com%2Fcloudstack-infrastructure-sup
port%2F
CloudStack Bootcamp Training Courses<
http://secure-web.cisco.com/16ybXVrdBe17I4mnCuuKp4rP7MHtjPD8iKc-dybhmY
02WlxPCR05bGrGVQcpjdJyLH0Iw_X1oC5orlEkNVEpIQLjDWOdf3rsjI2WOQTECMHhA89Z
uuE_96g0HQ-eLw_ylw4XsSOUq6KPyLjYryreJO38qidzmHjY6bxYb1MpUr9E7sCwzq_XBV
DBdZFH5gdWF/http%3A%2F%2Fshapeblue.com%2Fcloudstack-training%2F
This email and any attachments to it may be confidential and are
intended solely for the use of the individual to whom it is addressed.
Any views or opinions expressed are solely those of the author and do
not necessarily represent those of Shape Blue Ltd or related
companies. If you are not the intended recipient of this email, you
must neither take any action based upon its contents, nor copy or show
it to anyone. Please contact the sender if you believe you have
received this email in error. Shape Blue Ltd is a company incorporated
in England & Wales. ShapeBlue Services India LLP is a company
incorporated in India and is operated under license from Shape Blue
Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA
Pty Ltd is a company registered by The Republic of South Africa and is

traded under license from Shape Blue Ltd. ShapeBlue is a registered
trademark.

[Remi Bergsma]

I think the aggregate approach is much better (missed that mail at). Great it is available in 4.4 as well so we can fix it in both 4.4 and 4.5 in the same way.

Post by Koushik Das
I think as a design principle we shouldn't introduce HV specific checks in the orchestration/API layers.
I am not sure if the problem is specific to Vmware. Any out of band VR movement can lead to this issue. For now it is seen in Vmware but what about other HVs where CS relies on native HA provided by HV.
-----Original Message-----
Sent: Wednesday, 3 June 2015 17:54
Subject: Re: [DISCUSS] Out of Band VR migration, should we reboot VR or not?
This is a VMware specific problem, right?
We could also limit the fix to this hypervisor only.
Regards,
Remi
In case the VR is moved out of band (say as part of Vmware DRS), all network rules are lost. Rebooting VR from CS re-applies all the rules. Either the reboot is done manually from UI/API or automatically as was done as part of CLOUDSTACK-7994.
I haven't looked at the aggregate command. If it can be used to apply rules on VR without a reboot then that should work as well.
-Koushik
-----Original Message-----
Sent: Wednesday, 3 June 2015 17:18
To: dev
Subject: [DISCUSS] Out of Band VR migration, should we reboot VR or not?
Hi all,
Recently a behaviour was reported for ACS 4.5.1, where out of band VR migration would cause rebooting of VR. It seems this is a desired behaviour as per this issue: https://issues.apache.org/jira/browse/CLOUDSTACK-7994
It was shared on the thread on users ML that since CloudStack now supports aggregate commands for VR, this behaviour is unnecessary.
The VR in 4.5+ supports aggregated execution of commands on VRs to allows us to achieve eventual consistency of VR state without actually rebooting it, see this for details: https://issues.apache.org/jira/browse/CLOUDSTACK-6047
Please share your comments on whether if we should revert the fix or not, and the best way to do it. Thanks.
Regards,
Rohit Yadav
Software Architect, ShapeBlue
Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design & Build<http://secure-web.cisco.com/1dxwCI-iCtF6Z_nBrHijVPBNYijQKrOiFTNAyoWGJy4rj0SYtw0-E0Y0pAkH3D-NB0b01e1nt_XkrPm26uaNt3xXaVo4ZbAU7coVeFNM7kzqwOki5MLTTck7Y8OTbP1c-Zv9PMiy-M43LI_uAvDx-mT_q0vQcDZCt4fKB5W9A3-8Bdv5rzE0m03ik1kkWENKj/http%3A%2F%2Fshapeblue.com%2Fiaas-cloud-design-and-build%2F%2F>
CSForge – rapid IaaS deployment framework<http://secure-web.cisco.com/1KBMX-P7Uz8rk1CFZiKEJ6XOBtwCL6e4cxHjHhKOIeqCjjNcPHHrb0GoqH0CpXLUo42d8MsSS0X8jUVxeUQUuCEywQ8cHO7-KakvmeWrimKoqc3jlsmu_OgNQqdMRNwrO_uw71z7aOol5O-nshuUYfVgArT2kp2roAFkOUfEGQ2uyy18JLB5k2WhCMiyVyNs1/http%3A%2F%2Fshapeblue.com%2Fcsforge%2F>
CloudStack Consulting<http://secure-web.cisco.com/18B_NmraREMmC8vTWKXxDVFNi6uVPHolLmQtOC5pxOCqqVKFHBRYmXzR0e97TssbyoSyR90ruMa9xhxhyhPzd1nHvmYZXJWuiWx5eG0625EWlv3ZmrPpKJQcMrxeXCP4RXYkxVrnqKj-V0pH1AMKLwP5WRSrohlcx2oht8DsGOcyHmpFGdATnwmsBwW15ubts/http%3A%2F%2Fshapeblue.com%2Fcloudstack-consultancy%2F>
CloudStack Software Engineering<http://secure-web.cisco.com/1YTx4foSk2LL4qOkJ6hp8arX_tkHSgGLRhcrIh4tiJcMv4yOrG9LDzy23u-Of4to5Vw7pzPe7Z2BtX6UOObpoMHTu1Ygga8bXXaPX063jdDnTXVIeAzV86Qiya0QaVMLi8Yr7h619NJTW99-b2MVlgfrhYr2y0-xhqyX8x_qwxJsJ7ImNvcIeqtzluHsyEMwW/http%3A%2F%2Fshapeblue.com%2Fcloudstack-software-engineering%2F>
CloudStack Infrastructure Support<http://secure-web.cisco.com/1mWtUTi0OKF8weMnbfj_o1Xj1zQYn6dXlewUgkdu2L6UuSQi2hZdVPstsZ4PKm8seKRW9wKYnvF70hbKmhFdfk6XVguYW3ZXOBYvkNdiBK-yy31X-1Y4f2ESLclUvykFEl7X-hNmgBZkXpgP-uBL1ohAu36e0HtgdNE1zuX04JWN1YQzfKcYUHCb7GeKszF3x/http%3A%2F%2Fshapeblue.com%2Fcloudstack-infrastructure-support%2F>
CloudStack Bootcamp Training Courses<http://secure-web.cisco.com/16ybXVrdBe17I4mnCuuKp4rP7MHtjPD8iKc-dybhmY02WlxPCR05bGrGVQcpjdJyLH0Iw_X1oC5orlEkNVEpIQLjDWOdf3rsjI2WOQTECMHhA89ZuuE_96g0HQ-eLw_ylw4XsSOUq6KPyLjYryreJO38qidzmHjY6bxYb1MpUr9E7sCwzq_XBVDBdZFH5gdWF/http%3A%2F%2Fshapeblue.com%2Fcloudstack-training%2F>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue

[Koushik Das]

Yes.

-----Original Message-----
From: Remi Bergsma [mailto:***@schubergphilis.com]
Sent: Wednesday, 3 June 2015 18:13
To: ***@cloudstack.apache.org
Subject: Re: [DISCUSS] Out of Band VR migration, should we reboot VR or not?

I think the aggregate approach is much better (missed that mail at). Great it is available in 4.4 as well so we can fix it in both 4.4 and 4.5 in the same way.

Post by Koushik Das
I think as a design principle we shouldn't introduce HV specific checks in the orchestration/API layers.
I am not sure if the problem is specific to Vmware. Any out of band VR movement can lead to this issue. For now it is seen in Vmware but what about other HVs where CS relies on native HA provided by HV.
-----Original Message-----
Sent: Wednesday, 3 June 2015 17:54
Subject: Re: [DISCUSS] Out of Band VR migration, should we reboot VR or not?
This is a VMware specific problem, right?
We could also limit the fix to this hypervisor only.
Regards,
Remi
In case the VR is moved out of band (say as part of Vmware DRS), all network rules are lost. Rebooting VR from CS re-applies all the rules. Either the reboot is done manually from UI/API or automatically as was done as part of CLOUDSTACK-7994.
I haven't looked at the aggregate command. If it can be used to apply rules on VR without a reboot then that should work as well.
-Koushik
-----Original Message-----
Sent: Wednesday, 3 June 2015 17:18
To: dev
Subject: [DISCUSS] Out of Band VR migration, should we reboot VR or not?
Hi all,
Recently a behaviour was reported for ACS 4.5.1, where out of band VR
migration would cause rebooting of VR. It seems this is a desired
https://issues.apache.org/jira/browse/CLOUDSTACK-7994
It was shared on the thread on users ML that since CloudStack now supports aggregate commands for VR, this behaviour is unnecessary.
The VR in 4.5+ supports aggregated execution of commands on VRs to
allows us to achieve eventual consistency of VR state without actually
https://issues.apache.org/jira/browse/CLOUDSTACK-6047
Please share your comments on whether if we should revert the fix or not, and the best way to do it. Thanks.
Regards,
Rohit Yadav
Software Architect, ShapeBlue
M. [phone] |
Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design &
Build<http://secure-web.cisco.com/1dxwCI-iCtF6Z_nBrHijVPBNYijQKrOiFTNA
yoWGJy4rj0SYtw0-E0Y0pAkH3D-NB0b01e1nt_XkrPm26uaNt3xXaVo4ZbAU7coVeFNM7k
zqwOki5MLTTck7Y8OTbP1c-Zv9PMiy-M43LI_uAvDx-mT_q0vQcDZCt4fKB5W9A3-8Bdv5
rzE0m03ik1kkWENKj/http%3A%2F%2Fshapeblue.com%2Fiaas-cloud-design-and-b
uild%2F%2F> CSForge – rapid IaaS deployment
framework<http://secure-web.cisco.com/1KBMX-P7Uz8rk1CFZiKEJ6XOBtwCL6e4
cxHjHhKOIeqCjjNcPHHrb0GoqH0CpXLUo42d8MsSS0X8jUVxeUQUuCEywQ8cHO7-Kakvme
WrimKoqc3jlsmu_OgNQqdMRNwrO_uw71z7aOol5O-nshuUYfVgArT2kp2roAFkOUfEGQ2u
yy18JLB5k2WhCMiyVyNs1/http%3A%2F%2Fshapeblue.com%2Fcsforge%2F>
CloudStack
Consulting<http://secure-web.cisco.com/18B_NmraREMmC8vTWKXxDVFNi6uVPHo
lLmQtOC5pxOCqqVKFHBRYmXzR0e97TssbyoSyR90ruMa9xhxhyhPzd1nHvmYZXJWuiWx5e
G0625EWlv3ZmrPpKJQcMrxeXCP4RXYkxVrnqKj-V0pH1AMKLwP5WRSrohlcx2oht8DsGOc
yHmpFGdATnwmsBwW15ubts/http%3A%2F%2Fshapeblue.com%2Fcloudstack-consult
ancy%2F> CloudStack Software
Engineering<http://secure-web.cisco.com/1YTx4foSk2LL4qOkJ6hp8arX_tkHSg
GLRhcrIh4tiJcMv4yOrG9LDzy23u-Of4to5Vw7pzPe7Z2BtX6UOObpoMHTu1Ygga8bXXaP
X063jdDnTXVIeAzV86Qiya0QaVMLi8Yr7h619NJTW99-b2MVlgfrhYr2y0-xhqyX8x_qwx
JsJ7ImNvcIeqtzluHsyEMwW/http%3A%2F%2Fshapeblue.com%2Fcloudstack-softwa
re-engineering%2F> CloudStack Infrastructure
Support<http://secure-web.cisco.com/1mWtUTi0OKF8weMnbfj_o1Xj1zQYn6dXle
wUgkdu2L6UuSQi2hZdVPstsZ4PKm8seKRW9wKYnvF70hbKmhFdfk6XVguYW3ZXOBYvkNdi
BK-yy31X-1Y4f2ESLclUvykFEl7X-hNmgBZkXpgP-uBL1ohAu36e0HtgdNE1zuX04JWN1Y
QzfKcYUHCb7GeKszF3x/http%3A%2F%2Fshapeblue.com%2Fcloudstack-infrastruc
ture-support%2F> CloudStack Bootcamp Training
Courses<http://secure-web.cisco.com/16ybXVrdBe17I4mnCuuKp4rP7MHtjPD8iK
c-dybhmY02WlxPCR05bGrGVQcpjdJyLH0Iw_X1oC5orlEkNVEpIQLjDWOdf3rsjI2WOQTE
CMHhA89ZuuE_96g0HQ-eLw_ylw4XsSOUq6KPyLjYryreJO38qidzmHjY6bxYb1MpUr9E7s
Cwzq_XBVDBdZFH5gdWF/http%3A%2F%2Fshapeblue.com%2Fcloudstack-training%2
F>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered tradem

[Sateesh Chodapuneedi]

Post by Koushik Das
-----Original Message-----
Sent: Wednesday, June 3, 2015 6:08 PM
Subject: RE: [DISCUSS] Out of Band VR migration, should we reboot VR or not?
I think as a design principle we shouldn't introduce HV specific checks in the
orchestration/API layers.
I am not sure if the problem is specific to Vmware. Any out of band VR
movement can lead to this issue. For now it is seen in Vmware but what about
other HVs where CS relies on native HA provided by HV.

[Sateesh] Same is applicable to Hyper-v if HA is enabled
Also in case of VMware, even enabling DRS causes out of band movement of VMs from host to host.
I think it'd be better to address this irrespective of underlying hypervisor.

Regards,
Sateesh

Post by Koushik Das
-----Original Message-----
Sent: Wednesday, 3 June 2015 17:54
Subject: Re: [DISCUSS] Out of Band VR migration, should we reboot VR or not?
This is a VMware specific problem, right?
We could also limit the fix to this hypervisor only.
Regards,
Remi
On 03 Jun 2015, at 14:12, Koushik Das
In case the VR is moved out of band (say as part of Vmware DRS), all network
rules are lost. Rebooting VR from CS re-applies all the rules. Either the reboot is
done manually from UI/API or automatically as was done as part of
CLOUDSTACK-7994.
I haven't looked at the aggregate command. If it can be used to apply rules on
VR without a reboot then that should work as well.
-Koushik
-----Original Message-----
Sent: Wednesday, 3 June 2015 17:18
To: dev
Subject: [DISCUSS] Out of Band VR migration, should we reboot VR or not?
Hi all,
Recently a behaviour was reported for ACS 4.5.1, where out of band VR
migration would cause rebooting of VR. It seems this is a desired behaviour as
per this issue: https://issues.apache.org/jira/browse/CLOUDSTACK-7994
It was shared on the thread on users ML that since CloudStack now supports
aggregate commands for VR, this behaviour is unnecessary.
The VR in 4.5+ supports aggregated execution of commands on VRs to allows us
to achieve eventual consistency of VR state without actually rebooting it, see
this for details: https://issues.apache.org/jira/browse/CLOUDSTACK-6047
Please share your comments on whether if we should revert the fix or not, and
the best way to do it. Thanks.
Regards,
Rohit Yadav
Software Architect, ShapeBlue
M. [phone] |
Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design & Build<http://secure-web.cisco.com/1dxwCI-
iCtF6Z_nBrHijVPBNYijQKrOiFTNAyoWGJy4rj0SYtw0-E0Y0pAkH3D-
NB0b01e1nt_XkrPm26uaNt3xXaVo4ZbAU7coVeFNM7kzqwOki5MLTTck7Y8OTbP
1c-Zv9PMiy-M43LI_uAvDx-mT_q0vQcDZCt4fKB5W9A3-
8Bdv5rzE0m03ik1kkWENKj/http%3A%2F%2Fshapeblue.com%2Fiaas-cloud-
design-and-build%2F%2F>
CSForge – rapid IaaS deployment framework<http://secure-
web.cisco.com/1KBMX-
P7Uz8rk1CFZiKEJ6XOBtwCL6e4cxHjHhKOIeqCjjNcPHHrb0GoqH0CpXLUo42d8Ms
SS0X8jUVxeUQUuCEywQ8cHO7-
KakvmeWrimKoqc3jlsmu_OgNQqdMRNwrO_uw71z7aOol5O-
nshuUYfVgArT2kp2roAFkOUfEGQ2uyy18JLB5k2WhCMiyVyNs1/http%3A%2F%2F
shapeblue.com%2Fcsforge%2F>
CloudStack Consulting<http://secure-
web.cisco.com/18B_NmraREMmC8vTWKXxDVFNi6uVPHolLmQtOC5pxOCqqVKF
HBRYmXzR0e97TssbyoSyR90ruMa9xhxhyhPzd1nHvmYZXJWuiWx5eG0625EWlv3
ZmrPpKJQcMrxeXCP4RXYkxVrnqKj-
V0pH1AMKLwP5WRSrohlcx2oht8DsGOcyHmpFGdATnwmsBwW15ubts/http%3A
%2F%2Fshapeblue.com%2Fcloudstack-consultancy%2F>
CloudStack Software Engineering<http://secure-
web.cisco.com/1YTx4foSk2LL4qOkJ6hp8arX_tkHSgGLRhcrIh4tiJcMv4yOrG9LDzy
23u-
Of4to5Vw7pzPe7Z2BtX6UOObpoMHTu1Ygga8bXXaPX063jdDnTXVIeAzV86Qiya0
QaVMLi8Yr7h619NJTW99-b2MVlgfrhYr2y0-
xhqyX8x_qwxJsJ7ImNvcIeqtzluHsyEMwW/http%3A%2F%2Fshapeblue.com%2Fcl
oudstack-software-engineering%2F>
CloudStack Infrastructure Support<http://secure-
web.cisco.com/1mWtUTi0OKF8weMnbfj_o1Xj1zQYn6dXlewUgkdu2L6UuSQi2hZ
dVPstsZ4PKm8seKRW9wKYnvF70hbKmhFdfk6XVguYW3ZXOBYvkNdiBK-yy31X-
1Y4f2ESLclUvykFEl7X-hNmgBZkXpgP-
uBL1ohAu36e0HtgdNE1zuX04JWN1YQzfKcYUHCb7GeKszF3x/http%3A%2F%2Fsh
apeblue.com%2Fcloudstack-infrastructure-support%2F>
CloudStack Bootcamp Training Courses<http://secure-
web.cisco.com/16ybXVrdBe17I4mnCuuKp4rP7MHtjPD8iKc-
dybhmY02WlxPCR05bGrGVQcpjdJyLH0Iw_X1oC5orlEkNVEpIQLjDWOdf3rsjI2WO
QTECMHhA89ZuuE_96g0HQ-
eLw_ylw4XsSOUq6KPyLjYryreJO38qidzmHjY6bxYb1MpUr9E7sCwzq_XBVDBdZFH
5gdWF/http%3A%2F%2Fshapeblue.com%2Fcloudstack-training%2F>
This email and any attachments to it may be confidential and are intended solely
for the use of the individual to whom it is addressed. Any views or opinions
expressed are solely those of the author and do not necessarily represent those
of Shape Blue Ltd or related companies. If you are not the intended recipient of
this email, you must neither take any action based upon its contents, nor copy or
show it to anyone. Please contact the sender if you believe you have received
this email in error. Shape Blue Ltd is a company incorporated in England &
Wales. ShapeBlue Services India LLP is a company incorporated in India and is
operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda
is a company incorporated in Brasil and is operated under license from Shape
Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South
Africa and is traded under license from Shape Blue Ltd. Sh

[Daan Hoogland]

agreed, but we do controlled patching on xs using out of bound migration of
VRs to have no downtime during patching. The present fix actually worsens
the downtime. Let's go for the aggregate command tactics

Op wo 3 jun. 2015 om 15:18 schreef Sateesh Chodapuneedi <

Post by Remi Bergsma

Post by Koushik Das
-----Original Message-----
Sent: Wednesday, June 3, 2015 6:08 PM
Subject: RE: [DISCUSS] Out of Band VR migration, should we reboot VR or

not?

Post by Koushik Das
I think as a design principle we shouldn't introduce HV specific checks

in the

Post by Koushik Das
orchestration/API layers.
I am not sure if the problem is specific to Vmware. Any out of band VR
movement can lead to this issue. For now it is seen in Vmware but what

about

Post by Koushik Das
other HVs where CS relies on native HA provided by HV.

[Sateesh] Same is applicable to Hyper-v if HA is enabled
Also in case of VMware, even enabling DRS causes out of band movement of
VMs from host to host.
I think it'd be better to address this irrespective of underlying hypervisor.
Regards,
Sateesh

Post by Koushik Das
-----Original Message-----
Sent: Wednesday, 3 June 2015 17:54
Subject: Re: [DISCUSS] Out of Band VR migration, should we reboot VR or

not?

Post by Koushik Das
This is a VMware specific problem, right?
We could also limit the fix to this hypervisor only.
Regards,
Remi
On 03 Jun 2015, at 14:12, Koushik Das
In case the VR is moved out of band (say as part of Vmware DRS), all

network

Post by Koushik Das
rules are lost. Rebooting VR from CS re-applies all the rules. Either

the reboot is

Post by Koushik Das
done manually from UI/API or automatically as was done as part of
CLOUDSTACK-7994.
I haven't looked at the aggregate command. If it can be used to apply

rules on

Post by Koushik Das
VR without a reboot then that should work as well.
-Koushik
-----Original Message-----
Sent: Wednesday, 3 June 2015 17:18
To: dev
Subject: [DISCUSS] Out of Band VR migration, should we reboot VR or not?
Hi all,
Recently a behaviour was reported for ACS 4.5.1, where out of band VR
migration would cause rebooting of VR. It seems this is a desired

behaviour as

Post by Koushik Das
per this issue: https://issues.apache.org/jira/browse/CLOUDSTACK-7994
It was shared on the thread on users ML that since CloudStack now

supports

Post by Koushik Das
aggregate commands for VR, this behaviour is unnecessary.
The VR in 4.5+ supports aggregated execution of commands on VRs to

allows us

Post by Koushik Das
to achieve eventual consistency of VR state without actually rebooting

it, see

Post by Koushik Das
this for details: https://issues.apache.org/jira/browse/CLOUDSTACK-6047
Please share your comments on whether if we should revert the fix or

not, and

Post by Koushik Das
the best way to do it. Thanks.
Regards,
Rohit Yadav
Software Architect, ShapeBlue
M. [phone] |
Find out more about ShapeBlue and our range of CloudStack related

services

Post by Koushik Das
IaaS Cloud Design & Build<http://secure-web.cisco.com/1dxwCI-
iCtF6Z_nBrHijVPBNYijQKrOiFTNAyoWGJy4rj0SYtw0-E0Y0pAkH3D-
NB0b01e1nt_XkrPm26uaNt3xXaVo4ZbAU7coVeFNM7kzqwOki5MLTTck7Y8OTbP
1c-Zv9PMiy-M43LI_uAvDx-mT_q0vQcDZCt4fKB5W9A3-
8Bdv5rzE0m03ik1kkWENKj/http%3A%2F%2Fshapeblue.com%2Fiaas-cloud-
design-and-build%2F%2F>
CSForge – rapid IaaS deployment framework<http://secure-
web.cisco.com/1KBMX-
P7Uz8rk1CFZiKEJ6XOBtwCL6e4cxHjHhKOIeqCjjNcPHHrb0GoqH0CpXLUo42d8Ms
SS0X8jUVxeUQUuCEywQ8cHO7-
KakvmeWrimKoqc3jlsmu_OgNQqdMRNwrO_uw71z7aOol5O-
nshuUYfVgArT2kp2roAFkOUfEGQ2uyy18JLB5k2WhCMiyVyNs1/http%3A%2F%2F
shapeblue.com%2Fcsforge%2F>
CloudStack Consulting<http://secure-
web.cisco.com/18B_NmraREMmC8vTWKXxDVFNi6uVPHolLmQtOC5pxOCqqVKF
HBRYmXzR0e97TssbyoSyR90ruMa9xhxhyhPzd1nHvmYZXJWuiWx5eG0625EWlv3
ZmrPpKJQcMrxeXCP4RXYkxVrnqKj-
V0pH1AMKLwP5WRSrohlcx2oht8DsGOcyHmpFGdATnwmsBwW15ubts/http%3A
%2F%2Fshapeblue.com%2Fcloudstack-consultancy%2F>
CloudStack Software Engineering<http://secure-
web.cisco.com/1YTx4foSk2LL4qOkJ6hp8arX_tkHSgGLRhcrIh4tiJcMv4yOrG9LDzy
23u-
Of4to5Vw7pzPe7Z2BtX6UOObpoMHTu1Ygga8bXXaPX063jdDnTXVIeAzV86Qiya0
QaVMLi8Yr7h619NJTW99-b2MVlgfrhYr2y0-
xhqyX8x_qwxJsJ7ImNvcIeqtzluHsyEMwW/http%3A%2F%2Fshapeblue.com%2Fcl
oudstack-software-engineering%2F>
CloudStack Infrastructure Support<http://secure-
web.cisco.com/1mWtUTi0OKF8weMnbfj_o1Xj1zQYn6dXlewUgkdu2L6UuSQi2hZ
dVPstsZ4PKm8seKRW9wKYnvF70hbKmhFdfk6XVguYW3ZXOBYvkNdiBK-yy31X-
1Y4f2ESLclUvykFEl7X-hNmgBZkXpgP-
uBL1ohAu36e0HtgdNE1zuX04JWN1YQzfKcYUHCb7GeKszF3x/http%3A%2F%2Fsh
apeblue.com%2Fcloudstack-infrastructure-support%2F>
CloudStack Bootcamp Training Courses<http://secure-
web.cisco.com/16ybXVrdBe17I4mnCuuKp4rP7MHtjPD8iKc-
dybhmY02WlxPCR05bGrGVQcpjdJyLH0Iw_X1oC5orlEkNVEpIQLjDWOdf3rsjI2WO
QTECMHhA89ZuuE_96g0HQ-
eLw_ylw4XsSOUq6KPyLjYryreJO38qidzmHjY6bxYb1MpUr9E7sCwzq_XBVDBdZFH
5gdWF/http%3A%2F%2Fshapeblue.com%2Fcloudstack-training%2F>
This email and any attachments to it may be confidential and are

intended solely

Post by Koushik Das
for the use of the individual to whom it is addressed. Any views or

opinions

Post by Koushik Das
expressed are solely those of the author and do not necessarily

represent those

Post by Koushik Das
of Shape Blue Ltd or related companies. If you are not the intended

recipient of

Post by Koushik Das
this email, you must neither take any action based upon its contents,

nor copy or

Post by Koushik Das
show it to anyone. Please contact the sender if you believe you have

received

Post by Koushik Das
this email in error. Shape Blue Ltd is a company incorporated in England

&

Post by Koushik Das
Wales. ShapeBlue Services India LLP is a company incorporated in India

and is

Post by Koushik Das
operated under license from Shape Blue Ltd. Shape Blue Brasil

Consultoria Ltda

Post by Koushik Das
is a company incorporated in Brasil and is operated under license from

Shape

Post by Koushik Das
Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic

of South

Post by Koushik Das
Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a

registered

Post by Koushik Das
trademark.

[Rene Moser]

Sorry for not answering in the thread, I was not on the dev ML so I
could not reply

I reported this current behavior to be an issue on the user ML and
wanted to ask Koushik Das about his experiences.

I would not agree, in an Vmware environment live migrations, e.g. Vmware
DRS breaks IPtables normally. In my opinion, this would make DRS
senseless. And if it happens, it would be uncommon.

We do live migrations daily with VMs having IPtables rules and I didn't
see such a behaviour on any of these VMs.

Could you share more information about your experiences Koushik Das? In
what conditions this happend?

In any case I would love to test DRS live migraions on VR without this
current behavior. In any way, with this current behaviour, we would have
a lot of downtimes.

The other "solution" would be reapplying the rules without reboot, I am
not fully aware of the new behaviour af aggration but wouldn't this also
cause a network outage?

Yours
René

[Ian Southam]

If the machine crashes and/or rebooted during the oob migration by a party that is not the orchestrator, (read vCenter) then the rules will be lost.

The new persistent config code in the 4.6 branch will definitely prevent this behaviour as, the config will be preserved even if the machines is rebooted without receiving the “command line” parameters. The previous known state will be preserved.

—
Ian

Post by Rene Moser
Sorry for not answering in the thread, I was not on the dev ML so I
could not reply
I reported this current behavior to be an issue on the user ML and
wanted to ask Koushik Das about his experiences.
I would not agree, in an Vmware environment live migrations, e.g. Vmware
DRS breaks IPtables normally. In my opinion, this would make DRS
senseless. And if it happens, it would be uncommon.
We do live migrations daily with VMs having IPtables rules and I didn't
see such a behaviour on any of these VMs.
Could you share more information about your experiences Koushik Das? In
what conditions this happend?
In any case I would love to test DRS live migraions on VR without this
current behavior. In any way, with this current behaviour, we would have
a lot of downtimes.
The other "solution" would be reapplying the rules without reboot, I am
not fully aware of the new behaviour af aggration but wouldn't this also
cause a network outage?
Yours
René

[Rene Moser]

Hi

Post by Ian Southam
If the machine crashes and/or rebooted during the oob migration by a party that is not the orchestrator, (read vCenter) then the rules will be lost.

I fully agree, a reboot due a failing live migration, would cause a
reboot. So what? Then we blame VMware, the orchestration will reboot the
VR and everything is fine. This would cause seconds of outage.

But then the missing persistence of the iptables would be the problem,
not the live migration task, right?

We should fix the persistence of the rules during reboot and not try to
be more clever then the hypervisor cluster orchestration.

Just my 2 cents.

[Remi Bergsma]

Hi all,

I just had a look at this more closely and had a chat with Ian about it. The only way for the original problem to happen (losing iptables rules) is if the live migrate would fail and the hypervisor rebooted the vm. The cause is the non-persistance of the router configuration, which is fixed in 4.6 by the way. I would say failing live migrations does not happen often (I have never seen it happening).

Anyway, once this happens to the router, it is stuck in a state where it does not have the linklocal configuration any more. Would CloudStack be able to issue a aggregate command while it cannot reach it? Rebooting might be the only way out after all. It’s just that rebooting by default in case of out-of-band migrations I’d say is a little bit too much. CloudStack would detect the problem anyway, as it cannot reach the linklocal anymore.

The interesting situation is that we have releases out there that now reboot by default.

My proposal to solve it in 4.4 and 4.5:
- Implement a setting ‘reboot systemvm when out-of-band migration detected’.
The default should be false and release notes should mention a changed behaviour from 4.4.3 and 4.5.1. To get the old behaviour, switch to true. A small group of people should be interested in this.

I guess this is the best of both worlds. Do you guys agree?

The other option I see is to revert the commit, as I think that serves most people.

Who is willing to help implement it?

Regards, Remi


On 03 Jun 2015, at 17:42, Rene Moser <***@renemoser.net<mailto:***@renemoser.net>> wrote:

Hi

On 03.06.2015 17:06, Ian Southam wrote:
If the machine crashes and/or rebooted during the oob migration by a party that is not the orchestrator, (read vCenter) then the rules will be lost.

I fully agree, a reboot due a failing live migration, would cause a
reboot. So what? Then we blame VMware, the orchestration will reboot the
VR and everything is fine. This would cause seconds of outage.

But then the missing persistence of the iptables would be the problem,
not the live migration task, right?

We should fix the persistence of the rules during reboot and not try to
be more clever then the hypervisor cluster orchestration.

Just my 2 cents.

[Koushik Das]

Post by Rohit Yadav
Hi all,
I just had a look at this more closely and had a chat with Ian about it. The only way for the original problem to happen (losing iptables rules) is if the live migrate would fail and the hypervisor rebooted the vm. The cause is the non-persistance of the router configuration, which is fixed in 4.6 by the way. I would say failing live migrations does not happen often (I have never seen it happening).

What about native HA in Vmware and HyperV? Say the original host has failed, Vmware will bring up the VR on another host as part of native HA. In this case also the configuration is lost.

Post by Rohit Yadav
Anyway, once this happens to the router, it is stuck in a state where it does not have the linklocal configuration any more. Would CloudStack be able to issue a aggregate command while it cannot reach it? Rebooting might be the only way out after all. It’s just that rebooting by default in case of out-of-band migrations I’d say is a little bit too much. CloudStack would detect the problem anyway, as it cannot reach the linklocal anymore.
The interesting situation is that we have releases out there that now reboot by default.
- Implement a setting ‘reboot systemvm when out-of-band migration detected’.
The default should be false and release notes should mention a changed behaviour from 4.4.3 and 4.5.1. To get the old behaviour, switch to true. A small group of people should be interested in this.
I guess this is the best of both worlds. Do you guys agree?
The other option I see is to revert the commit, as I think that serves most people.
Who is willing to help implement it?
Regards, Remi
Hi
If the machine crashes and/or rebooted during the oob migration by a party that is not the orchestrator, (read vCenter) then the rules will be lost.
I fully agree, a reboot due a failing live migration, would cause a
reboot. So what? Then we blame VMware, the orchestration will reboot the
VR and everything is fine. This would cause seconds of outage.
But then the missing persistence of the iptables would be the problem,
not the live migration task, right?
We should fix the persistence of the rules during reboot and not try to
be more clever then the hypervisor cluster orchestration.
Just my 2 cents.

[Jayapal Reddy Uradi]

In VR configuration persistence (4.6) only iptables rules are persisted ?
There are other configuration (interface ips, routes etc in VR will be lost on reboot) are these taken care ?


Thanks,
Jayapal

Post by Koushik Das

Post by Rohit Yadav
Hi all,
I just had a look at this more closely and had a chat with Ian about it. The only way for the original problem to happen (losing iptables rules) is if the live migrate would fail and the hypervisor rebooted the vm. The cause is the non-persistance of the router configuration, which is fixed in 4.6 by the way. I would say failing live migrations does not happen often (I have never seen it happening).

What about native HA in Vmware and HyperV? Say the original host has failed, Vmware will bring up the VR on another host as part of native HA. In this case also the configuration is lost.

Post by Rohit Yadav
Anyway, once this happens to the router, it is stuck in a state where it does not have the linklocal configuration any more. Would CloudStack be able to issue a aggregate command while it cannot reach it? Rebooting might be the only way out after all. It’s just that rebooting by default in case of out-of-band migrations I’d say is a little bit too much. CloudStack would detect the problem anyway, as it cannot reach the linklocal anymore.
The interesting situation is that we have releases out there that now reboot by default.
- Implement a setting ‘reboot systemvm when out-of-band migration detected’.
The default should be false and release notes should mention a changed behaviour from 4.4.3 and 4.5.1. To get the old behaviour, switch to true. A small group of people should be interested in this.
I guess this is the best of both worlds. Do you guys agree?
The other option I see is to revert the commit, as I think that serves most people.
Who is willing to help implement it?
Regards, Remi
Hi
If the machine crashes and/or rebooted during the oob migration by a party that is not the orchestrator, (read vCenter) then the rules will be lost.
I fully agree, a reboot due a failing live migration, would cause a
reboot. So what? Then we blame VMware, the orchestration will reboot the
VR and everything is fine. This would cause seconds of outage.
But then the missing persistence of the iptables would be the problem,
not the live migration task, right?
We should fix the persistence of the rules during reboot and not try to
be more clever then the hypervisor cluster orchestration.
Just my 2 cents.

[Daan Hoogland]

On Thu, Jun 4, 2015 at 8:15 AM, Jayapal Reddy Uradi

Post by Jayapal Reddy Uradi
In VR configuration persistence (4.6) only iptables rules are persisted ?
There are other configuration (interface ips, routes etc in VR will be lost on reboot) are these taken care ?

By design all config is persisted but let's double check on that.

@Ian: sir, can you confirm?

[Wilder Rodrigues]

There was a bug in the persistent stuff that was fixed here: https://issues.apache.org/jira/browse/CLOUDSTACK-4605

User data, IP tables, guest networks and pretty much everything else will be persisted.

Cheers,
Wilder


On 04 Jun 2015, at 09:54, Daan Hoogland <***@gmail.com<mailto:***@gmail.com>> wrote:

On Thu, Jun 4, 2015 at 8:15 AM, Jayapal Reddy Uradi
<***@citrix.com<mailto:***@citrix.com>> wrote:
In VR configuration persistence (4.6) only iptables rules are persisted ?
There are other configuration (interface ips, routes etc in VR will be lost on reboot) are these taken care ?


By design all config is persisted but let's double check on that.

@Ian: sir, can you confirm?

--
Daan (being to lazy/busy to check the code)

[Jayapal Reddy Uradi]

One more thing on VR config persistence, the configuration is persisted only on reboot not in VR stop, start am I right ?
@Wilder Can please point me to the FS ?

-Jayapal

Post by Wilder Rodrigues
There was a bug in the persistent stuff that was fixed here: https://issues.apache.org/jira/browse/CLOUDSTACK-4605
User data, IP tables, guest networks and pretty much everything else will be persisted.
Cheers,
Wilder
On Thu, Jun 4, 2015 at 8:15 AM, Jayapal Reddy Uradi
In VR configuration persistence (4.6) only iptables rules are persisted ?
There are other configuration (interface ips, routes etc in VR will be lost on reboot) are these taken care ?
By design all config is persisted but let's double check on that.
@Ian: sir, can you confirm?
--
Daan (being to lazy/busy to check the code)

[Remi Bergsma]

To summarise:
#1. rebooting VR is needed for hypervisors that have their own DR (like VMware and Hyperv) as a restart outside of CloudStack makes it lose its config hence the VR is unavailable
#2. rebooting is NOT needed for successful live migrations on _any_ hypervisor (since there was no restart everything still works)
#3. CloudStack 4.6 has persistent config in VR, so rebooting is never needed

The current behaviour in 4.4.3, 4.5.1 and master:
- always rebooting VR when out-of-band detected ==> Works great for #1, but makes case #2 not work

The previous behaviour:
- never rebooting VR when out-of-band detected ==> Works great for #2, but makes case #1 not work

We need something that works for both cases :-)

About #1 and #2:
Can we detect in another way that a VR became unreachable/non-functional and do an action based on that?

So, if a VR lives on a VMware hypervisor that happens to crash, VMware HA will start it on another available hypervisor but without config it will not be reachable on the control network. If we want to do it generic, I’d say that when a VR is not controllable any more we could reboot it. We could also make this a setting ‘systemvm auto reboot on control failure’ or whatever we call it.

This would then also be a useful feature in 4.6

About #3:
I’d suggest at least reverting the commit to master, as it makes no sense since the VR is persistent already.
https://github.com/apache/cloudstack/blob/master/server/src/com/cloud/network/router/VirtualNetworkApplianceManagerImpl.java#L2638
https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=commitdiff;h=c3515c9

Regards,
Remi



On 04 Jun 2015, at 10:03, Wilder Rodrigues <***@schubergphilis.com<mailto:***@schubergphilis.com>> wrote:

There was a bug in the persistent stuff that was fixed here: https://issues.apache.org/jira/browse/CLOUDSTACK-4605

User data, IP tables, guest networks and pretty much everything else will be persisted.

Cheers,
Wilder


On 04 Jun 2015, at 09:54, Daan Hoogland <***@gmail.com<mailto:***@gmail.com><mailto:***@gmail.com>> wrote:

On Thu, Jun 4, 2015 at 8:15 AM, Jayapal Reddy Uradi
<***@citrix.com<mailto:***@citrix.com><mailto:***@citrix.com>> wrote:
In VR configuration persistence (4.6) only iptables rules are persisted ?
There are other configuration (interface ips, routes etc in VR will be lost on reboot) are these taken care ?


By design all config is persisted but let's double check on that.

@Ian: sir, can you confirm?

[Rohit Yadav]

Hi,

Post by Remi Bergsma
#1. rebooting VR is needed for hypervisors that have their own DR (like VMware and Hyperv) as a restart outside of CloudStack makes it lose its config hence the VR is unavailable
#2. rebooting is NOT needed for successful live migrations on _any_ hypervisor (since there was no restart everything still works)
#3. CloudStack 4.6 has persistent config in VR, so rebooting is never needed
- always rebooting VR when out-of-band detected ==> Works great for #1, but makes case #2 not work
- never rebooting VR when out-of-band detected ==> Works great for #2, but makes case #1 not work
We need something that works for both cases :-)
Can we detect in another way that a VR became unreachable/non-functional and do an action based on that?
So, if a VR lives on a VMware hypervisor that happens to crash, VMware HA will start it on another available hypervisor but without config it will not be reachable on the control network. If we want to do it generic, I’d say that when a VR is not controllable any more we could reboot it. We could also make this a setting ‘systemvm auto reboot on control failure’ or whatever we call it.
This would then also be a useful feature in 4.6
I’d suggest at least reverting the commit to master, as it makes no sense since the VR is persistent already.
https://github.com/apache/cloudstack/blob/master/server/src/com/cloud/network/router/VirtualNetworkApplianceManagerImpl.java#L2638
https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=commitdiff;h=c3515c9

I agree on master, we can revert due to #3.

On 4.4 and 4.5 branches, if we revert it than it will break #1 and if we keep it - it breaks #2. We need to fix this in a hypervisor agnostic way.

So, if we can use aggregated cmd execution that Sudhashu and others have shared, in case of an out of band migration we can do this:

1. If we’re unable to reach VR, we can keep the RebootTask to do its job. In many cases I’ve seen VR disk corruption, so if after rebooting VR (which might kick in fsck) it fails CloudStack may eventually recreate a new VR makes sense. This will solve #1. This case IMO is highly unlikely.

2. If we’re able to reach the VR after VR is migrated out of band, we run an AgrregationExecutionTask sending all the rules for that VR. This will solve #2 without needing to reboot a VM. This case is more likely to happen, so if I’ve pick between this and the above case, I would try to solve for this case.

I’m not sure about the actual implementation, and will re-applying rules using a AgrregationExecutionTask cause any issue in the VR? Comments?

Regards,
Rohit Yadav
Software Architect, ShapeBlue
M. [phone] | ***@shapeblue.com
Blog: bhaisaab.org | Twitter: @_bhaisaab



Find out more about ShapeBlue and our range of CloudStack related services

IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from

[Rene Moser]

Hi

Post by Rohit Yadav
Hi,

Post by Remi Bergsma
#1. rebooting VR is needed for hypervisors that have their own DR (like VMware and Hyperv) as a restart outside of CloudStack makes it lose its config hence the VR is unavailable
#2. rebooting is NOT needed for successful live migrations on _any_ hypervisor (since there was no restart everything still works)
#3. CloudStack 4.6 has persistent config in VR, so rebooting is never needed

On 4.4 and 4.5 branches, if we revert it than it will break #1 and if we keep it - it breaks #2. We need to fix this in a hypervisor agnostic way.

#1 is something we are already aware of. So I vote for a revert.

We can currently live with it, because

1. Live Migration in VMware hardly fails for such a small VM without
much I/O and such low amount of RAM.

2. We are monitoring the router reboots (uptime) and cloudstack restart
router manually if this happens.

Just a silly question: why not implementing a detected OS out of bound
reboot looking for the

if uptime < last_uptime ? restartRouter

on the router in 4.4 and 4.5?

[Remi Bergsma]

+1

On 04 Jun 2015, at 14:29, Rohit Yadav <***@shapeblue.com<mailto:***@shapeblue.com>> wrote:

Hi,

On 04-Jun-2015, at 11:05 am, Remi Bergsma <***@schubergphilis.com<mailto:***@schubergphilis.com>> wrote:

To summarise:
#1. rebooting VR is needed for hypervisors that have their own DR (like VMware and Hyperv) as a restart outside of CloudStack makes it lose its config hence the VR is unavailable
#2. rebooting is NOT needed for successful live migrations on _any_ hypervisor (since there was no restart everything still works)
#3. CloudStack 4.6 has persistent config in VR, so rebooting is never needed

The current behaviour in 4.4.3, 4.5.1 and master:
- always rebooting VR when out-of-band detected ==> Works great for #1, but makes case #2 not work

The previous behaviour:
- never rebooting VR when out-of-band detected ==> Works great for #2, but makes case #1 not work

We need something that works for both cases :-)

About #1 and #2:
Can we detect in another way that a VR became unreachable/non-functional and do an action based on that?

So, if a VR lives on a VMware hypervisor that happens to crash, VMware HA will start it on another available hypervisor but without config it will not be reachable on the control network. If we want to do it generic, I’d say that when a VR is not controllable any more we could reboot it. We could also make this a setting ‘systemvm auto reboot on control failure’ or whatever we call it.

This would then also be a useful feature in 4.6

About #3:
I’d suggest at least reverting the commit to master, as it makes no sense since the VR is persistent already.
https://github.com/apache/cloudstack/blob/master/server/src/com/cloud/network/router/VirtualNetworkApplianceManagerImpl.java#L2638
https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=commitdiff;h=c3515c9

I agree on master, we can revert due to #3.

On 4.4 and 4.5 branches, if we revert it than it will break #1 and if we keep it - it breaks #2. We need to fix this in a hypervisor agnostic way.

So, if we can use aggregated cmd execution that Sudhashu and others have shared, in case of an out of band migration we can do this:

1. If we’re unable to reach VR, we can keep the RebootTask to do its job. In many cases I’ve seen VR disk corruption, so if after rebooting VR (which might kick in fsck) it fails CloudStack may eventually recreate a new VR makes sense. This will solve #1. This case IMO is highly unlikely.

2. If we’re able to reach the VR after VR is migrated out of band, we run an AgrregationExecutionTask sending all the rules for that VR. This will solve #2 without needing to reboot a VM. This case is more likely to happen, so if I’ve pick between this and the above case, I would try to solve for this case.

I’m not sure about the actual implementation, and will re-applying rules using a AgrregationExecutionTask cause any issue in the VR? Comments?

Regards,
Rohit Yadav
Software Architect, ShapeBlue
M. [phone] | ***@shapeblue.com<mailto:***@shapeblue.com>
Blog: bhaisaab.org<http://bhaisaab.org/> | Twitter: @_bhaisaab



Find out more about ShapeBlue and our range of CloudStack related services

IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.

[Koushik Das]

From the persistent config changes, it appears that aggregate command approach is not required in 4.6 and onwards. If someone wants to implement the aggregate approach only for 4.4 and 4.5 irrespective of the effort involved then that should be fine.

Otherwise we can just put some config to decide between #1 or #2. Another alternate could be to replace the reboot logic with an alert if there is an out of band VR migration. Based on it admin can reboot the VR from CCP if required. There will be a window during which the VR will be without ant config but that can be documented.

-----Original Message-----
From: Remi Bergsma [mailto:***@schubergphilis.com]
Sent: Thursday, 4 June 2015 19:56
To: ***@cloudstack.apache.org
Subject: Re: [DISCUSS] Out of Band VR migration, should we reboot VR or not?

+1

On 04 Jun 2015, at 14:29, Rohit Yadav <***@shapeblue.com<mailto:***@shapeblue.com>> wrote:

Hi,

On 04-Jun-2015, at 11:05 am, Remi Bergsma <***@schubergphilis.com<mailto:***@schubergphilis.com>> wrote:

To summarise:
#1. rebooting VR is needed for hypervisors that have their own DR (like VMware and Hyperv) as a restart outside of CloudStack makes it lose its config hence the VR is unavailable #2. rebooting is NOT needed for successful live migrations on _any_ hypervisor (since there was no restart everything still works) #3. CloudStack 4.6 has persistent config in VR, so rebooting is never needed

The current behaviour in 4.4.3, 4.5.1 and master:
- always rebooting VR when out-of-band detected ==> Works great for #1, but makes case #2 not work

The previous behaviour:
- never rebooting VR when out-of-band detected ==> Works great for #2, but makes case #1 not work

We need something that works for both cases :-)

About #1 and #2:
Can we detect in another way that a VR became unreachable/non-functional and do an action based on that?

So, if a VR lives on a VMware hypervisor that happens to crash, VMware HA will start it on another available hypervisor but without config it will not be reachable on the control network. If we want to do it generic, I’d say that when a VR is not controllable any more we could reboot it. We could also make this a setting ‘systemvm auto reboot on control failure’ or whatever we call it.

This would then also be a useful feature in 4.6

About #3:
I’d suggest at least reverting the commit to master, as it makes no sense since the VR is persistent already.
https://github.com/apache/cloudstack/blob/master/server/src/com/cloud/network/router/VirtualNetworkApplianceManagerImpl.java#L2638
https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=commitdiff;h=c3515c9

I agree on master, we can revert due to #3.

On 4.4 and 4.5 branches, if we revert it than it will break #1 and if we keep it - it breaks #2. We need to fix this in a hypervisor agnostic way.

So, if we can use aggregated cmd execution that Sudhashu and others have shared, in case of an out of band migration we can do this:

1. If we’re unable to reach VR, we can keep the RebootTask to do its job. In many cases I’ve seen VR disk corruption, so if after rebooting VR (which might kick in fsck) it fails CloudStack may eventually recreate a new VR makes sense. This will solve #1. This case IMO is highly unlikely.

2. If we’re able to reach the VR after VR is migrated out of band, we run an AgrregationExecutionTask sending all the rules for that VR. This will solve #2 without needing to reboot a VM. This case is more likely to happen, so if I’ve pick between this and the above case, I would try to solve for this case.

I’m not sure about the actual implementation, and will re-applying rules using a AgrregationExecutionTask cause any issue in the VR? Comments?

Regards,
Rohit Yadav
Software Architect, ShapeBlue
M. [phone] | ***@shapeblue.com<mailto:***@shapeblue.com>
Blog: bhaisaab.org<http://bhaisaab.org/> | Twitter: @_bhaisaab



Find out more about ShapeBlue and our range of CloudStack related services

IaaS Cloud Design & Build<http://secure-web.cisco.com/11NRz12My3B5Kzq2aGFKO015q6rxTtLhqlUFP7QMrC0-v_oGzAZzQRov3cOzFQSsoQWiiSpaXHXNHsmfncQDtL7aKrU44zCUAb42UC_t2RaRK1rZc7LdKyQRujP_oVBG8novKP65vr7lykJvafQToJSMcAFzzj2JUV9jZ6Lci-_snpKPxRE6vFwzL_Pi9BksK/http%3A%2F%2Fshapeblue.com%2Fiaas-cloud-design-and-build%2F%2F>
CSForge – rapid IaaS deployment framework<http://secure-web.cisco.com/1zDH16MEoBBXO2Aec7JxxfmsbN0-qdNbRkKTaQgb3gu3jOOXzW-4p6XKmZbRcvF5SE5njzz8vnF5PnPQIa_rsd5pUFtuDIx5ldguqoBt4Rk2nhHc6mwB2vtbzN4A9x-8vj7qn0ySYXnUC6gqnft-HDDQowNC3KHSq-CoU7bqHsNG3C5JRrNl1eCvNgVnudnbO/http%3A%2F%2Fshapeblue.com%2Fcsforge%2F>
CloudStack Consulting<http://secure-web.cisco.com/1-OtYLlvNuHbvbTUwsXuMO3DwRAlWKunbc3M6Q33mljUFXCuyzHK2fBYu0AFTyWepJ92k1NEHyCBTsbPRppXOC6QTmwwePInVQQCg0U6-WvvydOsE_gcyNG28N4NPoBdUZPVgJqZgKptvXAacDyAIHrv1QQxSxMUY36UNVXZAhSXsXock3Opz1pelR-9LBmm-/http%3A%2F%2Fshapeblue.com%2Fcloudstack-consultancy%2F>
CloudStack Software Engineering<http://secure-web.cisco.com/1eOOKbuGnQ5GwXa19eDF6ksRMZX5ph-7BozqOFbRiImEuer_HG6IBDJVlGyGcbGZdrCnHhvJFDp-Yl0L1eB-XbPf50nZMe2WOGKYSsx2mk-zuEuPNFwRiF0MEVRv4oOIznqqVe3tVXJwhdFfj9DYzq-7pBCplAeljApsG9e1yH0KfjpwzYN1pIX77I6NWMy1W/http%3A%2F%2Fshapeblue.com%2Fcloudstack-software-engineering%2F>
CloudStack Infrastructure Support<http://secure-web.cisco.com/1LjFiZT2WK_5OIqPsi82bcmnjV9p4bsGMz2bLnslp-tPxyh7oZ_rhhm3B0RkTE-7NVtbttV-kUxjPISzjpcwwsDcVpsdkC-CmrieRvQCvZj_8N_xmpmMcX32VE8-I2wSlfUACTy62VSNNtFDHiTY-g299bPGmj1DA1NxiAV7a2iDV31hfpZ84s_Q2GkQeeoeC/http%3A%2F%2Fshapeblue.com%2Fcloudstack-infrastructure-support%2F>
CloudStack Bootcamp Training Courses<http://secure-web.cisco.com/1GLYfy58uh9y-GuJdHLdpEruL8xIozImvUMGAxmjXtZvGSslV1DM4CdYTvSdbh2ci1Zor1wY5_Ibsq6Wb4RXrEKV9nuZhTWALVGYyDfuWtDBgjzFEgNx64DrghuaQa04hs2-VkbPFjnmBvQr2OsLG_YbLZHOsv7NLBqmwNq7YjEXEbIHvPvrLB4eMTyt4pfrg/http%3A%2F%2Fshapeblue.com%2Fcloudstack-training%2F>

This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. S

[Daan Hoogland]

nobody has been taking this up. In order to have a 4.4.4 I will go for
the quick hack (in 4.4 and 4.5)

Post by Koushik Das
From the persistent config changes, it appears that aggregate command approach is not required in 4.6 and onwards. If someone wants to implement the aggregate approach only for 4.4 and 4.5 irrespective of the effort involved then that should be fine.
Otherwise we can just put some config to decide between #1 or #2. Another alternate could be to replace the reboot logic with an alert if there is an out of band VR migration. Based on it admin can reboot the VR from CCP if required. There will be a window during which the VR will be without ant config but that can be documented.
-----Original Message-----
Sent: Thursday, 4 June 2015 19:56
Subject: Re: [DISCUSS] Out of Band VR migration, should we reboot VR or not?
+1
Hi,
#1. rebooting VR is needed for hypervisors that have their own DR (like VMware and Hyperv) as a restart outside of CloudStack makes it lose its config hence the VR is unavailable #2. rebooting is NOT needed for successful live migrations on _any_ hypervisor (since there was no restart everything still works) #3. CloudStack 4.6 has persistent config in VR, so rebooting is never needed
- always rebooting VR when out-of-band detected ==> Works great for #1, but makes case #2 not work
- never rebooting VR when out-of-band detected ==> Works great for #2, but makes case #1 not work
We need something that works for both cases :-)
Can we detect in another way that a VR became unreachable/non-functional and do an action based on that?
So, if a VR lives on a VMware hypervisor that happens to crash, VMware HA will start it on another available hypervisor but without config it will not be reachable on the control network. If we want to do it generic, I’d say that when a VR is not controllable any more we could reboot it. We could also make this a setting ‘systemvm auto reboot on control failure’ or whatever we call it.
This would then also be a useful feature in 4.6
I’d suggest at least reverting the commit to master, as it makes no sense since the VR is persistent already.
https://github.com/apache/cloudstack/blob/master/server/src/com/cloud/network/router/VirtualNetworkApplianceManagerImpl.java#L2638
https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=commitdiff;h=c3515c9
I agree on master, we can revert due to #3.
On 4.4 and 4.5 branches, if we revert it than it will break #1 and if we keep it - it breaks #2. We need to fix this in a hypervisor agnostic way.
1. If we’re unable to reach VR, we can keep the RebootTask to do its job. In many cases I’ve seen VR disk corruption, so if after rebooting VR (which might kick in fsck) it fails CloudStack may eventually recreate a new VR makes sense. This will solve #1. This case IMO is highly unlikely.
2. If we’re able to reach the VR after VR is migrated out of band, we run an AgrregationExecutionTask sending all the rules for that VR. This will solve #2 without needing to reboot a VM. This case is more likely to happen, so if I’ve pick between this and the above case, I would try to solve for this case.
I’m not sure about the actual implementation, and will re-applying rules using a AgrregationExecutionTask cause any issue in the VR? Comments?
Regards,
Rohit Yadav
Software Architect, ShapeBlue
Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design & Build<http://secure-web.cisco.com/11NRz12My3B5Kzq2aGFKO015q6rxTtLhqlUFP7QMrC0-v_oGzAZzQRov3cOzFQSsoQWiiSpaXHXNHsmfncQDtL7aKrU44zCUAb42UC_t2RaRK1rZc7LdKyQRujP_oVBG8novKP65vr7lykJvafQToJSMcAFzzj2JUV9jZ6Lci-_snpKPxRE6vFwzL_Pi9BksK/http%3A%2F%2Fshapeblue.com%2Fiaas-cloud-design-and-build%2F%2F>
CSForge – rapid IaaS deployment framework<http://secure-web.cisco.com/1zDH16MEoBBXO2Aec7JxxfmsbN0-qdNbRkKTaQgb3gu3jOOXzW-4p6XKmZbRcvF5SE5njzz8vnF5PnPQIa_rsd5pUFtuDIx5ldguqoBt4Rk2nhHc6mwB2vtbzN4A9x-8vj7qn0ySYXnUC6gqnft-HDDQowNC3KHSq-CoU7bqHsNG3C5JRrNl1eCvNgVnudnbO/http%3A%2F%2Fshapeblue.com%2Fcsforge%2F>
CloudStack Consulting<http://secure-web.cisco.com/1-OtYLlvNuHbvbTUwsXuMO3DwRAlWKunbc3M6Q33mljUFXCuyzHK2fBYu0AFTyWepJ92k1NEHyCBTsbPRppXOC6QTmwwePInVQQCg0U6-WvvydOsE_gcyNG28N4NPoBdUZPVgJqZgKptvXAacDyAIHrv1QQxSxMUY36UNVXZAhSXsXock3Opz1pelR-9LBmm-/http%3A%2F%2Fshapeblue.com%2Fcloudstack-consultancy%2F>
CloudStack Software Engineering<http://secure-web.cisco.com/1eOOKbuGnQ5GwXa19eDF6ksRMZX5ph-7BozqOFbRiImEuer_HG6IBDJVlGyGcbGZdrCnHhvJFDp-Yl0L1eB-XbPf50nZMe2WOGKYSsx2mk-zuEuPNFwRiF0MEVRv4oOIznqqVe3tVXJwhdFfj9DYzq-7pBCplAeljApsG9e1yH0KfjpwzYN1pIX77I6NWMy1W/http%3A%2F%2Fshapeblue.com%2Fcloudstack-software-engineering%2F>
CloudStack Infrastructure Support<http://secure-web.cisco.com/1LjFiZT2WK_5OIqPsi82bcmnjV9p4bsGMz2bLnslp-tPxyh7oZ_rhhm3B0RkTE-7NVtbttV-kUxjPISzjpcwwsDcVpsdkC-CmrieRvQCvZj_8N_xmpmMcX32VE8-I2wSlfUACTy62VSNNtFDHiTY-g299bPGmj1DA1NxiAV7a2iDV31hfpZ84s_Q2GkQeeoeC/http%3A%2F%2Fshapeblue.com%2Fcloudstack-infrastructure-support%2F>
CloudStack Bootcamp Training Courses<http://secure-web.cisco.com/1GLYfy58uh9y-GuJdHLdpEruL8xIozImvUMGAxmjXtZvGSslV1DM4CdYTvSdbh2ci1Zor1wY5_Ibsq6Wb4RXrEKV9nuZhTWALVGYyDfuWtDBgjzFEgNx64DrghuaQa04hs2-VkbPFjnmBvQr2OsLG_YbLZHOsv7NLBqmwNq7YjEXEbIHvPvrLB4eMTyt4pfrg/http%3A%2F%2Fshapeblue.com%2Fcloudstack-training%2F>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.

[Daan Hoogland]

I have a PR for this if someone can please verify...?

Post by Daan Hoogland
nobody has been taking this up. In order to have a 4.4.4 I will go for
the quick hack (in 4.4 and 4.5)

Post by Koushik Das
From the persistent config changes, it appears that aggregate command approach is not required in 4.6 and onwards. If someone wants to implement the aggregate approach only for 4.4 and 4.5 irrespective of the effort involved then that should be fine.
Otherwise we can just put some config to decide between #1 or #2. Another alternate could be to replace the reboot logic with an alert if there is an out of band VR migration. Based on it admin can reboot the VR from CCP if required. There will be a window during which the VR will be without ant config but that can be documented.
-----Original Message-----
Sent: Thursday, 4 June 2015 19:56
Subject: Re: [DISCUSS] Out of Band VR migration, should we reboot VR or not?
+1
Hi,
#1. rebooting VR is needed for hypervisors that have their own DR (like VMware and Hyperv) as a restart outside of CloudStack makes it lose its config hence the VR is unavailable #2. rebooting is NOT needed for successful live migrations on _any_ hypervisor (since there was no restart everything still works) #3. CloudStack 4.6 has persistent config in VR, so rebooting is never needed
- always rebooting VR when out-of-band detected ==> Works great for #1, but makes case #2 not work
- never rebooting VR when out-of-band detected ==> Works great for #2, but makes case #1 not work
We need something that works for both cases :-)
Can we detect in another way that a VR became unreachable/non-functional and do an action based on that?
So, if a VR lives on a VMware hypervisor that happens to crash, VMware HA will start it on another available hypervisor but without config it will not be reachable on the control network. If we want to do it generic, I’d say that when a VR is not controllable any more we could reboot it. We could also make this a setting ‘systemvm auto reboot on control failure’ or whatever we call it.
This would then also be a useful feature in 4.6
I’d suggest at least reverting the commit to master, as it makes no sense since the VR is persistent already.
https://github.com/apache/cloudstack/blob/master/server/src/com/cloud/network/router/VirtualNetworkApplianceManagerImpl.java#L2638
https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=commitdiff;h=c3515c9
I agree on master, we can revert due to #3.
On 4.4 and 4.5 branches, if we revert it than it will break #1 and if we keep it - it breaks #2. We need to fix this in a hypervisor agnostic way.
1. If we’re unable to reach VR, we can keep the RebootTask to do its job. In many cases I’ve seen VR disk corruption, so if after rebooting VR (which might kick in fsck) it fails CloudStack may eventually recreate a new VR makes sense. This will solve #1. This case IMO is highly unlikely.
2. If we’re able to reach the VR after VR is migrated out of band, we run an AgrregationExecutionTask sending all the rules for that VR. This will solve #2 without needing to reboot a VM. This case is more likely to happen, so if I’ve pick between this and the above case, I would try to solve for this case.
I’m not sure about the actual implementation, and will re-applying rules using a AgrregationExecutionTask cause any issue in the VR? Comments?
Regards,
Rohit Yadav
Software Architect, ShapeBlue
Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design & Build<http://secure-web.cisco.com/11NRz12My3B5Kzq2aGFKO015q6rxTtLhqlUFP7QMrC0-v_oGzAZzQRov3cOzFQSsoQWiiSpaXHXNHsmfncQDtL7aKrU44zCUAb42UC_t2RaRK1rZc7LdKyQRujP_oVBG8novKP65vr7lykJvafQToJSMcAFzzj2JUV9jZ6Lci-_snpKPxRE6vFwzL_Pi9BksK/http%3A%2F%2Fshapeblue.com%2Fiaas-cloud-design-and-build%2F%2F>
CSForge – rapid IaaS deployment framework<http://secure-web.cisco.com/1zDH16MEoBBXO2Aec7JxxfmsbN0-qdNbRkKTaQgb3gu3jOOXzW-4p6XKmZbRcvF5SE5njzz8vnF5PnPQIa_rsd5pUFtuDIx5ldguqoBt4Rk2nhHc6mwB2vtbzN4A9x-8vj7qn0ySYXnUC6gqnft-HDDQowNC3KHSq-CoU7bqHsNG3C5JRrNl1eCvNgVnudnbO/http%3A%2F%2Fshapeblue.com%2Fcsforge%2F>
CloudStack Consulting<http://secure-web.cisco.com/1-OtYLlvNuHbvbTUwsXuMO3DwRAlWKunbc3M6Q33mljUFXCuyzHK2fBYu0AFTyWepJ92k1NEHyCBTsbPRppXOC6QTmwwePInVQQCg0U6-WvvydOsE_gcyNG28N4NPoBdUZPVgJqZgKptvXAacDyAIHrv1QQxSxMUY36UNVXZAhSXsXock3Opz1pelR-9LBmm-/http%3A%2F%2Fshapeblue.com%2Fcloudstack-consultancy%2F>
CloudStack Software Engineering<http://secure-web.cisco.com/1eOOKbuGnQ5GwXa19eDF6ksRMZX5ph-7BozqOFbRiImEuer_HG6IBDJVlGyGcbGZdrCnHhvJFDp-Yl0L1eB-XbPf50nZMe2WOGKYSsx2mk-zuEuPNFwRiF0MEVRv4oOIznqqVe3tVXJwhdFfj9DYzq-7pBCplAeljApsG9e1yH0KfjpwzYN1pIX77I6NWMy1W/http%3A%2F%2Fshapeblue.com%2Fcloudstack-software-engineering%2F>
CloudStack Infrastructure Support<http://secure-web.cisco.com/1LjFiZT2WK_5OIqPsi82bcmnjV9p4bsGMz2bLnslp-tPxyh7oZ_rhhm3B0RkTE-7NVtbttV-kUxjPISzjpcwwsDcVpsdkC-CmrieRvQCvZj_8N_xmpmMcX32VE8-I2wSlfUACTy62VSNNtFDHiTY-g299bPGmj1DA1NxiAV7a2iDV31hfpZ84s_Q2GkQeeoeC/http%3A%2F%2Fshapeblue.com%2Fcloudstack-infrastructure-support%2F>
CloudStack Bootcamp Training Courses<http://secure-web.cisco.com/1GLYfy58uh9y-GuJdHLdpEruL8xIozImvUMGAxmjXtZvGSslV1DM4CdYTvSdbh2ci1Zor1wY5_Ibsq6Wb4RXrEKV9nuZhTWALVGYyDfuWtDBgjzFEgNx64DrghuaQa04hs2-VkbPFjnmBvQr2OsLG_YbLZHOsv7NLBqmwNq7YjEXEbIHvPvrLB4eMTyt4pfrg/http%3A%2F%2Fshapeblue.com%2Fcloudstack-training%2F>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.

--
Daan

[Anshul Gangwar]

In case of #1 timing of reboot will also have impact and may result to undesired behaviour as both CloudStack and native HA trying to act on same VM.

Regards,
Anshul

On 04-Jun-2015, at 2:35 pm, Remi Bergsma <***@schubergphilis.com<mailto:***@schubergphilis.com>> wrote:

To summarise:
#1. rebooting VR is needed for hypervisors that have their own DR (like VMware and Hyperv) as a restart outside of CloudStack makes it lose its config hence the VR is unavailable
#2. rebooting is NOT needed for successful live migrations on _any_ hypervisor (since there was no restart everything still works)
#3. CloudStack 4.6 has persistent config in VR, so rebooting is never needed

The current behaviour in 4.4.3, 4.5.1 and master:
- always rebooting VR when out-of-band detected ==> Works great for #1, but makes case #2 not work

The previous behaviour:
- never rebooting VR when out-of-band detected ==> Works great for #2, but makes case #1 not work

We need something that works for both cases :-)

About #1 and #2:
Can we detect in another way that a VR became unreachable/non-functional and do an action based on that?

So, if a VR lives on a VMware hypervisor that happens to crash, VMware HA will start it on another available hypervisor but without config it will not be reachable on the control network. If we want to do it generic, I’d say that when a VR is not controllable any more we could reboot it. We could also make this a setting ‘systemvm auto reboot on control failure’ or whatever we call it.

This would then also be a useful feature in 4.6

About #3:
I’d suggest at least reverting the commit to master, as it makes no sense since the VR is persistent already.
https://github.com/apache/cloudstack/blob/master/server/src/com/cloud/network/router/VirtualNetworkApplianceManagerImpl.java#L2638
https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=commitdiff;h=c3515c9

Regards,
Remi



On 04 Jun 2015, at 10:03, Wilder Rodrigues <***@schubergphilis.com<mailto:***@schubergphilis.com>> wrote:

There was a bug in the persistent stuff that was fixed here: https://issues.apache.org/jira/browse/CLOUDSTACK-4605

User data, IP tables, guest networks and pretty much everything else will be persisted.

Cheers,
Wilder


On 04 Jun 2015, at 09:54, Daan Hoogland <***@gmail.com<mailto:***@gmail.com><mailto:***@gmail.com>> wrote:

On Thu, Jun 4, 2015 at 8:15 AM, Jayapal Reddy Uradi
<***@citrix.com<mailto:***@citrix.com><mailto:***@citrix.com>> wrote:
In VR configuration persistence (4.6) only iptables rules are persisted ?
There are other configuration (interface ips, routes etc in VR will be lost on reboot) are these taken care ?


By design all config is persisted but let's double check on that.

@Ian: sir, can you confirm?

[Ian Southam]

The idea is to resist everything.

If stuff does not persist then it can be viewed as a bug ;).

—
Ian

On 04 Jun 2015, at 09:54, Daan Hoogland <***@gmail.com<mailto:***@gmail.com>> wrote:

By design all config is persisted but let's double check on that.

@Ian: sir, can you confirm?

[Ian Southam]

resist should be persist. Do not answer mails on iphones ;)

Post by Ian Southam
The idea is to resist everything.
If stuff does not persist then it can be viewed as a bug ;).
—
Ian
By design all config is persisted but let's double check on that.

[Koushik Das]

Ian,
With the persistent config change, the VR has become 'stateful'.
Is there a possibility for VR config and DB config to go out of sync? If so how is the config in the VR and DB kept in sync?

-----Original Message-----
From: Ian Southam [mailto:***@schubergphilis.com]
Sent: Thursday, 4 June 2015 15:41
To: ***@cloudstack.apache.org
Cc: Daan Hoogland
Subject: Re: [DISCUSS] Out of Band VR migration, should we reboot VR or not?

resist should be persist. Do not answer mails on iphones ;)

Post by Ian Southam
The idea is to resist everything.
If stuff does not persist then it can be viewed as a bug ;).
—
Ian
By design all config is persisted but let's double check on that.
@

[Daan Hoogland]

Post by Koushik Das
Is there a possibility for VR config and DB config to go out of sync? If so how is the config in the VR and DB kept in sync?

Koushik, the db is leading and conversion is applied to the state of
the router. On differences, newly sent config takes precedence.

[Koushik Das]

Thanks for the clarification Daan. So as I understand - after VR reboot outside of CS if there is any config differences (between persisted ones and DB) then MS will immediately push the config from DB to VR. Is that correct?
Is there any document/write-up available for the persistence changes?

Post by Daan Hoogland

Post by Koushik Das
Is there a possibility for VR config and DB config to go out of sync? If so how is the config in the VR and DB kept in sync?

Koushik, the db is leading and conversion is applied to the state of
the router. On differences, newly sent config takes precedence.
--
Daan

[Ian Southam]

On 04 Jun 2015, at 16:50, Koushik Das <***@citrix.com<mailto:***@citrix.com>> wrote:

Thanks for the clarification Daan. So as I understand - after VR reboot outside of CS if there is any config differences (between persisted ones and DB) then MS will immediately push the config from DB to VR. Is that correct?
Is there any document/write-up available for the persistence changes?

Hi,

I cannot find the docs we wrote up but will send the link when I do!

Essentially the shell commands are replaced with json which is passed to VPCr or VR. The son object is then merged into a son configuration on the router. There is one config per type (firewallacls, lips, forwarding_rules etc.). These are stored in /etc/cloudstack.

(See ./core/src/com/cloud/agent/resource/virtualnetwork/facade/ and ./core/src/com/cloud/agent/resource/virtualnetwork/model/)

Each time the config is merged, some python runs which compares the configured state to the expected state from the json files. It will then provision (or deprovision) anything that has changed.

(See systemvm/patches/debian/config/opt/cloud/bin/configure.py)

In addition on reboot (planned or otherwise), the configurator is also called setting the provisioned state back to the last known good state.

For VPCs we also implemented a redundant VPCr (based upon the same technology as the normal VR) and, calls to allow an upgrade from single VPCr to redundant VPCr.

Beyond this, cloudstack will work the same as it always did. If it loses touch with a router, it will takes steps to recreate it etc.

So yes, the router is no longer truly stateless but, cloudstack remains in charge. The intention is not to break the orchestration paradagm.

—
Grts!
Ian

[Daan Hoogland]

Post by Koushik Das
What about native HA in Vmware and HyperV? Say the original host has failed, Vmware will bring up the VR on another host as part of native HA. In this case also the configuration is lost.

Yes, in this case it is needed, no discussion there. (untill 4.6)

[Remi Bergsma]

Hi all,

I just had a look at this more closely and had a chat with Ian about it. The only way for the original problem to happen (losing iptables rules) is if the live migrate would fail and the hypervisor rebooted the vm. The cause is the non-persistance of the router configuration, which is fixed in 4.6 by the way. I would say failing live migrations does not happen often (I have never seen it happening).

Anyway, once this happens to the router, it is stuck in a state where it does not have the linklocal configuration any more. Would CloudStack be able to issue a aggregate command while it cannot reach it? Rebooting might be the only way out after all. It’s just that rebooting by default in case of out-of-band migrations I’d say is a little bit too much. CloudStack would detect the problem anyway, as it cannot reach the linklocal anymore.

The interesting situation is that we have releases out there that now reboot by default.

My proposal to solve it in 4.4 and 4.5:
- Implement a setting ‘reboot systemvm when out-of-band migration detected’.
The default should be false and release notes should mention a changed behaviour from 4.4.3 and 4.5.1. To get the old behaviour, switch to true. A small group of people should be interested in this.

I guess this is the best of both worlds. Do you guys agree?

The other option I see is to revert the commit, as I think that serves most people.

Who is willing to help implement it?

Regards, Remi


On 03 Jun 2015, at 17:42, Rene Moser <***@renemoser.net<mailto:***@renemoser.net>> wrote:

Hi

On 03.06.2015 17:06, Ian Southam wrote:
If the machine crashes and/or rebooted during the oob migration by a party that is not the orchestrator, (read vCenter) then the rules will be lost.

I fully agree, a reboot due a failing live migration, would cause a
reboot. So what? Then we blame VMware, the orchestration will reboot the
VR and everything is fine. This would cause seconds of outage.

But then the missing persistence of the iptables would be the problem,
not the live migration task, right?

We should fix the persistence of the rules during reboot and not try to
be more clever then the hypervisor cluster orchestration.

Just my 2 cents.

[Koushik Das]

The VR configurations are not persisted and so these are not present when out of band movement happens outside of CS. When a VR is recreated/rebooted from CS all the configurations are read from DB and pushed to the VR.

The VR was originally designed as a stateless entity, all state/configuration is pushed to it by on start by MS.

-----Original Message-----
From: Rene Moser [mailto:***@renemoser.net]
Sent: Wednesday, 3 June 2015 20:28
To: ***@cloudstack.apache.org
Subject: RE: [DISCUSS] Out of Band VR migration, should we reboot VR or not?

Sorry for not answering in the thread, I was not on the dev ML so I could not reply

I reported this current behavior to be an issue on the user ML and wanted to ask Koushik Das about his experiences.

I would not agree, in an Vmware environment live migrations, e.g. Vmware DRS breaks IPtables normally. In my opinion, this would make DRS senseless. And if it happens, it would be uncommon.

We do live migrations daily with VMs having IPtables rules and I didn't see such a behaviour on any of these VMs.

Could you share more information about your experiences Koushik Das? In what conditions this happend?

In any case I would love to test DRS live migraions on VR without this current behavior. In any way, with this current behaviour, we would have a lot of downtimes.

The other "solution" would be reapplying the rules without reboot, I am not fully aware of the new behaviour af aggration but wouldn't this also cause a network outage?

Yours
René

[Daan Hoogland]

Post by Koushik Das
The VR configurations are not persisted and so these are not present when out of band movement happens outside of CS. When a VR is recreated/rebooted from CS all the configurations are read from DB and pushed to the VR.

This is not needed if it was a live migration. Only when the vm was
actually recreated.