Using the new Deployment History Cleanup feature in UrbanCode Deploy 6.2.1
- Published:
- categories: ibm
- tags: ucd, deploy, administration
Overview
With UCD 6.2.1 we have added the ability to perform a Deployment Process History Cleanup, I mentioned this in my presentation at InterConnect but didn’t get too many bites from people in the room wanting to try this out. I have had a chance to take it for a spin and help some clients implement this in production now, so I think it is fair to say that this is ready for you to play around with, this feature is now available with 6.2.1.1 after having seen some clients and some refinement.
So what does the Process History Cleanup mean, well today there is no cleanup of Process Requests i.e. they stick around forever. So this feature is part of an enterprise hardening and improvement effort to make this solution scale further, as never deleting history leads to very large filesystem growth of the process logs directory and the database workflow history.
Turning on the feature
If you are on 6.2.1.1 you have this enabled, if you are only 6.2.1.0 then you need to manually turn it on.
As a preview feature, it required adding a custom flag to the UrbanCode Server installed.properties file and a system restart to pick it up.
com.urbancode.ds.ServerConstants.historyCleanupVisible=true
Add or confirm that this setting is present
sgwilbur@ucdandr:/opt/ibm-ucd/server$ grep 'com.urbancode.ds.ServerConstants.historyCleanupVisible=true' /opt/ibm-ucd/server/conf/server/installed.properties
com.urbancode.ds.ServerConstants.historyCleanupVisible=true
Once this has been added to the installed.properties file correctly, well you won’t see the screen below in System Settings, so fix this first and restart.
Next per app environment you need to enable cleanup, the default behavior for the feature preview is no cleanup will happen by default, so while testing we can configure a small number of environments manually to review the behavior.
App > Environment
Triggering the cleanup
By default the cleanup will be happening once a day, based on your provided System Settings values. That is all well in good for your day to day operations but is a pain for testing. The good news here is you have another way to kick off a cleanup via an exposed REST endpoint.
So fire up your favorite programming language, curl, or browser REST client and we have a fairly simple job.
Authenticate and POST to this uri /rest/historyCleanup/runCleanupNow, here is a simple example against my local server
sgwilbur@ucdandr:/opt/IBM/UCRelease$ curl -i --insecure --user admin:admin -X POST https://localhost:9443/rest/historyCleanup/runCleanupNow
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
Cache-Control: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: No-cache
Set-Cookie: JSESSIONID_9080=6E5218F27EFFD796951D3CA85344E89A; Path=/; Secure; HttpOnly
WWW-Authenticate: Basic realm="IBM UrbanCode Deploy"
Set-Cookie: UCD_SESSION_KEY=088c558c-6914-458b-a6a1-388424a4fdfc; Expires=Wed, 31-May-2084 19:13:25 GMT; Path=/; Secure
Content-Length: 0
Date: Fri, 13 May 2016 15:59:18 GMT
Server: SERVER
And you should immediately see a cleanup triggered in the logs:
2016-05-13 15:59:18,014 TRACE HistoryCleanup com.urbancode.ds.cleanup.HistoryCleanup - Started Process Cleanup, cleanup will end at Fri May 13 16:59:18 UTC 2016
This will use your System Setting for duration, but just triggers the job to run, also it is smart enough to not start it a bunch of times if it is already running.
Verifying and Enabling via DB Queries
You can check the settings in mass or per app via SQL as well once we start getting things going, the System settings
mysql> select name, value from ps_prop_value where prop_sheet_id = ( select id from ps_prop_sheet where name = 'System Settings') and name like '%history%';
+------------------------------+-------+
| name | value |
+------------------------------+-------+
| history.cleanup.days.to.keep | 10 |
| history.cleanup.hour.of.day | 0 |
| history.cleanup.enabled | true |
| history.cleanup.duration | 1 |
+------------------------------+-------+
4 rows in set (0.00 sec)
Then at the per Application level, you will likely want to do this via SQL here is an example that checks one for one specific application by name, here I am showing both the Component Version and Process History Cleanup settings.
select t1.name as env_name, t2.name as app_name, t1.cleanup_days_to_keep, t1.cleanup_count_to_keep, t1.history_days_to_keep
from ds_environment t1
inner join ds_application t2 on t1.application_id = t2.id
where t2.name like 'APP00001'
;
Running this SQL on a sample system I see:
+----------+-------------------------------+----------------------+-----------------------+----------------------+
| env_name | app_name | cleanup_days_to_keep | cleanup_count_to_keep | history_days_to_keep |
+----------+-------------------------------+----------------------+-----------------------+----------------------+
| DEV | APP00001 | 0 | 0 | -1 |
| PROD | APP00001 | 0 | 0 | -1 |
| UAT | APP00001 | 0 | 0 | -1 |
+----------+-------------------------------+----------------------+-----------------------+----------------------+
3 rows in set (0.00 sec)
After updating the values to some defaults I want via the Console:``
+----------+-------------------------------+----------------------+-----------------------+----------------------+
| env_name | app_name | cleanup_days_to_keep | cleanup_count_to_keep | history_days_to_keep |
+----------+-------------------------------+----------------------+-----------------------+----------------------+
| DEV | APP00001 | 0 | 0 | 7 |
| PROD | APP00001 | 0 | 0 | 120 |
| UAT | APP00001 | 0 | 0 | 30 |
+----------+-------------------------------+----------------------+-----------------------+----------------------+
3 rows in set (0.00 sec)
So in your actual environment you will probably have history so you won’t need to fudge it, but for me to test or for you to do any simple validations the first thing we need to do is request some application processes. The second part is time travel, either at regular speed or just be fudging with the system clock. So first I just generate a whole bunch of simple deployments against a specific app. see API example in python. Now that I have some history I want an easy way to see what is being cleaned up, so a simple query against history, again by Application:
Before:
select count(t3.name), t2.name as app_name, t3.name as env_name
from rt_app_process_request t1
inner join ds_application t2 on t1.application_id = t2.id
inner join ds_environment t3 on t1.environment_id = t3.id
where t2.name like 'APP00001'
group by t3.name
;
+----------------+----------+----------+
| count(t3.name) | app_name | env_name |
+----------------+----------+----------+
| 18 | APP00001 | DEV |
| 18 | APP00001 | PROD |
| 50 | APP00001 | UAT |
+----------------+----------+----------+
3 rows in set (0.00 sec)
So now I fake my date forward 11 days, and as cleanup is triggered I expect to see my DEV process requests get cleaned up, sudo date XXX and ask the cleanup to run again via curl, we just rinse and repeat below to fully drive the example home.
+----------------+----------+----------+
| count(t3.name) | app_name | env_name |
+----------------+----------+----------+
| 1 | APP00001 | DEV |
| 18 | APP00001 | PROD |
| 50 | APP00001 | UAT |
+----------------+----------+----------+
3 rows in set (0.00 sec)
Now let’s check UAT, bump a month into the future sudo date XXX
+----------------+----------+----------+
| count(t3.name) | app_name | env_name |
+----------------+----------+----------+
| 1 | APP00001 | DEV |
| 18 | APP00001 | PROD |
| 1 | APP00001 | UAT |
+----------------+----------+----------+
3 rows in set (0.00 sec)
Now another 2 months, sudo date XXXX:
+----------------+----------+----------+
| count(t3.name) | app_name | env_name |
+----------------+----------+----------+
| 1 | APP00001 | DEV |
| 1 | APP00001 | PROD |
| 1 | APP00001 | UAT |
+----------------+----------+----------+
3 rows in set (0.00 sec)
Kinda goofy, but successful in illustrating how this works and something you can pretty easily script if you need to, here is a starter bash friendly example:
sudo date `date --date='now + day + week' '+%m%d%H%M'`
curl -i --insecure --user admin:admin -X POST https://localhost:9443/rest/historyCleanup/runCleanupNow
sleep 5
sudo date `date --date='now + month' '+%m%d%H%M'`
curl -i --insecure --user admin:admin -X POST https://localhost:9443/rest/historyCleanup/runCleanupNow
sleep 5
sudo date `date --date='now + month + month' '+%m%d%H%M'`
curl -i --insecure --user admin:admin -X POST https://localhost:9443/rest/historyCleanup/runCleanupNow
sleep 5
New Logging entries
Here is an example run, the system is already clean in this case.
2016-09-30 21:00:10,371 DEBUG AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Found 17 processes to delete.
2016-09-30 21:00:10,404 DEBUG AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Deleting deployment id: 0bee91a5-b56a-4c27-91f9-a2882270c3ef
2016-09-30 21:00:10,409 INFO AsynEventService-5 com.urbancode.ds.cleanup.SourceConfigExecutionRecordCleanupScheduledEvent - Version Import Record Cleanup is complete.
2016-09-30 21:00:10,412 INFO WorkflowTraceCleanup com.urbancode.ds.cleanup.WorkflowTraceCleanup - Can't start workflow trace cleanup, already running
2016-09-30 21:00:10,740 TRACE AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Successfully deleted deployment with id 0bee91a5-b56a-4c27-91f9-a2882270c3ef
2016-09-30 21:00:10,741 DEBUG AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Deleting deployment id: 0fae7893-5cd0-4925-852a-7471b59d4fd6
2016-09-30 21:00:10,929 TRACE AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Successfully deleted deployment with id 0fae7893-5cd0-4925-852a-7471b59d4fd6
2016-09-30 21:00:10,930 DEBUG AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Deleting deployment id: 2e061a21-3a3b-4d79-bf40-4dceb50ac5e7
2016-09-30 21:00:11,080 TRACE AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Successfully deleted deployment with id 2e061a21-3a3b-4d79-bf40-4dceb50ac5e7
2016-09-30 21:00:11,081 DEBUG AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Deleting deployment id: 56de453e-637b-462b-a5e5-8bb04cafbfba
2016-09-30 21:00:11,283 TRACE AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Successfully deleted deployment with id 56de453e-637b-462b-a5e5-8bb04cafbfba
2016-09-30 21:00:11,291 DEBUG AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Deleting deployment id: 5985f098-3a13-4a40-ab13-609df6b79aaa
2016-09-30 21:00:11,449 TRACE AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Successfully deleted deployment with id 5985f098-3a13-4a40-ab13-609df6b79aaa
2016-09-30 21:00:11,449 DEBUG AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Deleting deployment id: 791404ad-cc79-461a-9fd5-8e6bba8dd05d
2016-09-30 21:00:11,593 TRACE AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Successfully deleted deployment with id 791404ad-cc79-461a-9fd5-8e6bba8dd05d
2016-09-30 21:00:11,594 DEBUG AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Deleting deployment id: 7fa32602-9efa-4271-89dc-2357fa160ded
2016-09-30 21:00:11,691 TRACE AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Successfully deleted deployment with id 7fa32602-9efa-4271-89dc-2357fa160ded
2016-09-30 21:00:11,691 DEBUG AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Deleting deployment id: 8341e132-e055-49de-8c13-1aee4d3472b3
2016-09-30 21:00:11,800 TRACE AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Successfully deleted deployment with id 8341e132-e055-49de-8c13-1aee4d3472b3
2016-09-30 21:00:11,800 DEBUG AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Deleting deployment id: 8f3a696c-314a-47d1-9339-ce45bbc75ead
2016-09-30 21:00:11,897 TRACE AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Successfully deleted deployment with id 8f3a696c-314a-47d1-9339-ce45bbc75ead
2016-09-30 21:00:11,898 DEBUG AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Deleting deployment id: 9c9ea070-c8ac-4fd9-a01e-6a9744cc10a9
2016-09-30 21:00:12,032 TRACE AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Successfully deleted deployment with id 9c9ea070-c8ac-4fd9-a01e-6a9744cc10a9
2016-09-30 21:00:12,033 DEBUG AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Deleting deployment id: b8ae1d44-cdc8-4dbf-b4c6-261468a1f679
2016-09-30 21:00:12,185 TRACE AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Successfully deleted deployment with id b8ae1d44-cdc8-4dbf-b4c6-261468a1f679
2016-09-30 21:00:12,186 DEBUG AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Deleting deployment id: b9c0c646-1746-489a-93d5-ac437a3a1d0b
2016-09-30 21:00:12,301 TRACE AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Successfully deleted deployment with id b9c0c646-1746-489a-93d5-ac437a3a1d0b
2016-09-30 21:00:12,302 DEBUG AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Deleting deployment id: bcf2bf21-77cd-407d-b703-d2bd41b2a597
2016-09-30 21:00:12,476 TRACE AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Successfully deleted deployment with id bcf2bf21-77cd-407d-b703-d2bd41b2a597
2016-09-30 21:00:12,477 DEBUG AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Deleting deployment id: c7310aa9-8a1e-45c2-ae59-7b248317d50e
2016-09-30 21:00:12,603 TRACE AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Successfully deleted deployment with id c7310aa9-8a1e-45c2-ae59-7b248317d50e
2016-09-30 21:00:12,605 DEBUG AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Deleting deployment id: d71cefb0-4bac-45ae-b5d6-6d662c7a2cf8
2016-09-30 21:00:12,775 TRACE AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Successfully deleted deployment with id d71cefb0-4bac-45ae-b5d6-6d662c7a2cf8
2016-09-30 21:00:12,775 DEBUG AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Deleting deployment id: e4cdce14-781b-41ed-af73-f6360b33a4e2
2016-09-30 21:00:12,905 TRACE AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Successfully deleted deployment with id e4cdce14-781b-41ed-af73-f6360b33a4e2
2016-09-30 21:00:12,905 DEBUG AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Deleting deployment id: efe5d5f0-1a73-4a64-99da-74da5259180b
2016-09-30 21:00:13,016 TRACE AsynEventService-3 com.urbancode.ds.cleanup.HistoryCleanup - Successfully deleted deployment with id efe5d5f0-1a73-4a64-99da-74da5259180b
Conclusion
While this feature is new to your org I would be a bit more conservative with how aggressive you get with production type deployments, when you are first getting started. Knowing that this is a one way process means that once it is remove the only way to restore is to perform a database restoration. This is not that surprising as this is the primary goal is to cleanup space in the database and filesystem, so this is really to be expected. Give it a try and see how well it works for you to cleanup the horde of CI build/deploys you may have in yoru DEV/SIT environments.