hi there!
Is there a way to import plans, builds and distributions from Bamboo?
Thanks,
Jacques.
Hi Jacques,
To import data from Bamboo you can check out our documentation about how to import from REST API. How I can see in the Bamboo documentation there are API endpoints available for plans and builds https://developer.atlassian.com/bamboodev/rest-apis/bamboo-rest-resources
For authentication probably use basic HTTP authentification described there https://developer.atlassian.com/bamboodev/rest-apis/using-the-bamboo-rest-apis#UsingtheBambooRESTAPIs-authenticateRESTAuthentication
Regards,
Gatis.
Hi Jacques,
I did a bit deeper investigation and come up with a solution for data import from Bamboo.
As Bamboo projects, plans and other information aren’t tightly coupled with JIRA Projects and versions, the best would be to import it into a separate eazyBI account.
I divided import into two source applications - one for Plan and Build import, other for Deployment imports.
The following import definitions are tested on my test Bamboo instance with quite few build projects, builds and deployments. It could be that some information could be missing due to missing pagination options in REST API calls.
Additional information from REST API could be added by changing custom JavaScript code.
Plan and Build information import
Import definition
{
"application_type": "rest_api",
"application_params": {
"source_params": {
"url": "BAMBOO_HOME_URL/rest/api/latest/project?expand=projects.project.plans.plan",
"skip_ssl_verification": "0",
"pagination": "none",
"authentication_type": "basic",
"username": "admin",
"password": null,
"content_type": "json",
"json_data_path": "$.projects.project",
"custom_javascript_code": "return _.flatten(_.map(doc.plans.plan, function(plan) {\n offset = 0;\n builds = [];\n while(true){\n d = getDocument(\"/rest/api/latest/result/\" + plan.key + \"?expand=results.result,results.result.jiraIssues&start-index=\" + offset);\n builds = builds.concat(d.results.result);\n if (d.results.size == 25) {\n offset = offset + 25;\n } else {\n break;\n }\n };\n return _.map(builds, function(build) {\n keys = '';\n _.each(build.jiraIssues.issue, function(issue) {\n keys = keys + issue.key + ','\n });\n return {\n project_name: doc.name,\n project_key: doc.key,\n plan_name: plan.name,\n plan_key: plan.key,\n build_key: build.key,\n build_state: build.state,\n build_number: build.number,\n build_duration: build.buildDurationInSeconds,\n build_time: build.buildStartedTime,\n build_started: build.buildStartedTime,\n build_completed: build.buildCompletedTime,\n build_issue_keys: keys,\n finished_build_count: build.finished ? 1 : 0,\n in_progress_build_count: build.finished ? 0 : 1\n }\n });\n}));"
},
"columns_options": {
"default_names": false
},
"extra_options": {
"regular_import_frequency": 0,
"regular_import_at": "",
"time_zone": ""
}
},
"source_cube_name": "Builds",
"columns": [
{
"name": "project_name",
"data_type": "string",
"dimension": "Project",
"name_column": true,
"dimension_level": "Project"
},
{
"name": "project_key",
"data_type": "string",
"dimension": "Project",
"key_column": true,
"dimension_level": "Project"
},
{
"name": "plan_name",
"data_type": "string",
"dimension": "Project",
"dimension_level": "Plan",
"name_column": true
},
{
"name": "plan_key",
"data_type": "string",
"dimension": "Project",
"dimension_level": "Plan",
"key_column": true
},
{
"name": "build_key",
"data_type": "string",
"dimension": "Build",
"key_column": true
},
{
"name": "build_state",
"data_type": "string",
"dimension": "Build status"
},
{
"name": "build_number",
"data_type": "integer",
"dimension": "Build",
"property": "Build number"
},
{
"name": "build_duration",
"data_type": "integer",
"dimension": "Measures",
"dimension_member": "Build duration"
},
{
"name": "build_time",
"data_type": "datetime",
"dimension": "Time"
},
{
"name": "build_started",
"data_type": "datetime",
"dimension": "Build",
"property": "Build start time",
"date_count_measure": "Builds started",
"date_count_dimension": "Time"
},
{
"name": "build_completed",
"data_type": "datetime",
"dimension": "Build",
"property": "Build completed time",
"date_count_measure": "Builds completed",
"date_count_dimension": "Time"
},
{
"name": "build_issue_keys",
"data_type": "string",
"dimension": "Build",
"property": "Issues"
},
{
"name": "finished_build_count",
"data_type": "integer",
"dimension": "Measures",
"dimension_member": "Finished build count"
},
{
"name": "in_progress_build_count",
"data_type": "integer",
"dimension": "Measures",
"dimension_member": "In progress build count"
}
]
}
Import uses the following API end points:
- https://developer.atlassian.com/bamboodev/rest-apis/bamboo-rest-resources#BambooRESTResources-ProjectService
- https://developer.atlassian.com/bamboodev/rest-apis/bamboo-rest-resources#BambooRESTResources-BuildService—SpecificPlan
It will create the following dimensions:
- Project with two levels - Project and Plan
- Build
- Build status
Deployment information import
Import definition
{
"application_type": "rest_api",
"application_params": {
"source_params": {
"url": "http://localhost:8085/rest/api/latest/deploy/project/all",
"skip_ssl_verification": "0",
"pagination": "none",
"authentication_type": "basic",
"username": "admin",
"password": null,
"content_type": "json",
"custom_javascript_code": "function timestampToDate(ts) {\n var d = new Date(ts);\n return d.toISOString();\n};\n\nreturn _.flatten(_.map(doc.environments, function(environment) {\n offset = 0;\n results = [];\n while(true){\n d = getDocument(\"/rest/api/latest/deploy/environment/\" + environment.id + \"/results?max-results=25&start-index=\" + offset);\n results = results.concat(d.results);\n if (d.size >= offset + 25) {\n offset = offset + 25;\n } else {\n break;\n }\n };\n return _.map(results, function(result) {\n ret = {\n deploy_project_name: doc.name,\n project_id: doc.id,\n environment_project_name: doc.name,\n environment_id: environment.id,\n environment_name: environment.name,\n version_project_name: doc.name,\n deploy_id: result.id,\n deploy_start_time: timestampToDate(result.startedDate),\n deploy_end_time: timestampToDate(result.finishedDate),\n deployment_state: result.deploymentState,\n lifecycle_state: result.lifeCycleState,\n deploy_time: timestampToDate(result.startedDate),\n };\n \n if (doc.planKey) {\n ret.plan_key = doc.planKey.key;\n } else {\n ret.plan_key = '(none)';\n }\n version = result.deploymentVersion;\n if (version) {\n ret.version_id = version.id;\n ret.version_name = version.name;\n ret.version_date = timestampToDate(version.creationDate);\n ret.version_creator_key = version.creatorUserName;\n ret.version_creator_name = version.creatorDisplayName;\n ret.version_branch = version.planBranchName;\n } else {\n ret.version_name = '(none)';\n ret.version_id = -1;\n }\n return ret;\n });\n}));"
},
"columns_options": {
"default_names": false
},
"extra_options": {
"regular_import_frequency": 0,
"regular_import_at": "",
"time_zone": "Helsinki"
}
},
"source_cube_name": "Builds",
"columns": [
{
"name": "deploy_project_name",
"data_type": "string",
"dimension": "Deploy Project",
"dimension_level": "Deploy Project"
},
{
"name": "project_id",
"data_type": "integer",
"dimension": "Deploy Project",
"id_column": true,
"dimension_level": "Deploy Project"
},
{
"name": "environment_project_name",
"data_type": "string",
"dimension": "Environment",
"dimension_level": "Deploy Project",
"name_column": true
},
{
"name": "environment_id",
"data_type": "integer",
"dimension": "Environment",
"id_column": true,
"dimension_level": "Environment"
},
{
"name": "environment_name",
"data_type": "string",
"dimension": "Environment",
"dimension_level": "Environment",
"name_column": true
},
{
"name": "version_project_name",
"data_type": "string",
"dimension": "Version",
"dimension_level": "Deploy Project",
"name_column": true
},
{
"name": "deploy_id",
"data_type": "integer",
"dimension": "Deployment",
"key_column": true
},
{
"name": "deploy_start_time",
"data_type": "datetime",
"dimension": "Deployment",
"property": "Start time",
"date_count_measure": "Deployments started",
"date_count_dimension": "Time"
},
{
"name": "deploy_end_time",
"data_type": "datetime",
"dimension": "Deployment",
"property": "End time",
"date_count_measure": "Deployments ended",
"date_count_dimension": "Time"
},
{
"name": "deployment_state",
"data_type": "string",
"dimension": "Deployment Status"
},
{
"name": "lifecycle_state",
"data_type": "string",
"dimension": "Lifecycle State"
},
{
"name": "deploy_time",
"data_type": "datetime",
"dimension": "Time"
},
{
"name": "plan_key",
"data_type": "string",
"dimension": "Project",
"dimension_level": "Plan",
"key_column": true
},
{
"name": "version_id",
"data_type": "integer",
"dimension": "Version",
"id_column": true,
"dimension_level": "Version"
},
{
"name": "version_name",
"data_type": "string",
"dimension": "Version",
"dimension_level": "Version",
"name_column": true
},
{
"name": "version_date",
"data_type": "datetime",
"dimension": "Version",
"dimension_level": "Version",
"property": "Creation date"
},
{
"name": "version_creator_key",
"data_type": "string",
"dimension": "Version Creator",
"dimension_level": "Creator",
"key_column": true
},
{
"name": "version_creator_name",
"data_type": "string",
"dimension": "Version Creator",
"dimension_level": "Creator",
"name_column": true
},
{
"name": "version_branch",
"data_type": "string",
"dimension": "Branch"
}
]
}
Import uses the following API end points:
- https://docs.atlassian.com/bamboo/REST/6.0.3/#d2e632
- https://docs.atlassian.com/bamboo/REST/6.0.3/#d2e1541
The following dimensions will be created:
- Deploy Project
- Environment. Two levels - Deploy Project and Environment
- Version. Two levels - Deploy Project and Version
- Version Creator
- Branch
- Deployment Status
- Lifecycle State
In the eazyBI version 4.4.0, we improved custom JavaScript processing, and now the import should also work for larger datasets.
Plan and Build information import
{
"application_type": "rest_api",
"application_params": {
"source_params": {
"url": "BAMBOO_HOME_URL/rest/api/latest/project?expand=projects.project.plans.plan",
"skip_ssl_verification": "0",
"pagination": "none",
"authentication_type": "basic",
"username": "admin",
"password": null,
"content_type": "json",
"json_data_path": "$.projects.project",
"custom_javascript_code": "var results = [];\n\nfunction processBuilds(doc, plan, builds) {\n _.each(builds, function(build) {\n keys = '';\n _.each(build.jiraIssues.issue, function(issue) {\n keys = keys + issue.key + ','\n });\n results.push({\n project_name: doc.name,\n project_key: doc.key,\n plan_name: plan.name,\n plan_key: plan.key,\n build_key: build.key,\n build_state: build.state,\n build_number: build.number,\n build_duration: build.buildDurationInSeconds,\n build_time: build.buildStartedTime,\n build_started: build.buildStartedTime,\n build_completed: build.buildCompletedTime,\n build_issue_keys: keys,\n finished_build_count: build.finished ? 1 : 0,\n in_progress_build_count: build.finished ? 0 : 1\n })\n });\n}\n\nfunction getResult(doc, plan, offset) {\n getDocument(\"/rest/api/latest/result/\" + plan.key + \"?expand=results.result,results.result.jiraIssues&start-index=\" + offset, \n function(res) {\n processBuilds(doc, plan, res.results.result);\n if (res.results.size == 25) {\n getResult(doc, plan, offset + 25);\n }\n }\n );\n}\n_.map(doc.plans.plan, function(plan) {\n getResult(doc, plan, 0);\n});\n\nreturn results;"
},
"columns_options": {
"default_names": false
},
"extra_options": {
"regular_import_frequency": 0,
"regular_import_at": "",
"time_zone": "Helsinki"
}
},
"source_cube_name": "Builds",
"columns": [
{
"name": "project_name",
"data_type": "string",
"dimension": "Project",
"name_column": true,
"dimension_level": "Project"
},
{
"name": "project_key",
"data_type": "string",
"dimension": "Project",
"key_column": true,
"dimension_level": "Project"
},
{
"name": "plan_name",
"data_type": "string",
"dimension": "Project",
"dimension_level": "Plan",
"name_column": true
},
{
"name": "plan_key",
"data_type": "string",
"dimension": "Project",
"dimension_level": "Plan",
"key_column": true
},
{
"name": "build_key",
"data_type": "string",
"dimension": "Build",
"key_column": true
},
{
"name": "build_state",
"data_type": "string",
"dimension": "Build status"
},
{
"name": "build_number",
"data_type": "integer",
"dimension": "Build",
"property": "Build number"
},
{
"name": "build_duration",
"data_type": "integer",
"dimension": "Measures",
"dimension_member": "Build duration"
},
{
"name": "build_time",
"data_type": "datetime",
"dimension": "Time"
},
{
"name": "build_started",
"data_type": "datetime",
"dimension": "Build",
"property": "Build start time",
"date_count_measure": "Builds started",
"date_count_dimension": "Time"
},
{
"name": "build_completed",
"data_type": "datetime",
"dimension": "Build",
"property": "Build completed time",
"date_count_measure": "Builds completed",
"date_count_dimension": "Time"
},
{
"name": "build_issue_keys",
"data_type": "string",
"dimension": "Build",
"property": "Issues"
},
{
"name": "finished_build_count",
"data_type": "integer",
"dimension": "Measures",
"dimension_member": "Finished build count"
},
{
"name": "in_progress_build_count",
"data_type": "integer",
"dimension": "Measures",
"dimension_member": "In progress build count"
}
]
}
Deployment information import
{
"application_type": "rest_api",
"application_params": {
"source_params": {
"url": "BAMBOO_HOME_URL/rest/api/latest/deploy/project/all",
"skip_ssl_verification": "0",
"pagination": "none",
"authentication_type": "basic",
"username": "admin",
"password": null,
"content_type": "json",
"custom_javascript_code": "var results = [];\n\nfunction timestampToDate(ts) {\n var d = new Date(ts);\n return d.toISOString();\n};\n\nfunction processDeployments(doc, environment, deployments) {\n _.each(deployments, function(deploy) {\n result = {\n deploy_project_name: doc.name,\n project_id: doc.id,\n environment_project_name: doc.name,\n environment_id: environment.id,\n environment_name: environment.name,\n version_project_name: doc.name,\n deploy_id: deploy.id,\n deploy_start_time: timestampToDate(deploy.startedDate),\n deploy_end_time: timestampToDate(deploy.finishedDate),\n deployment_state: deploy.deploymentState,\n lifecycle_state: deploy.lifeCycleState,\n deploy_time: timestampToDate(deploy.startedDate),\n };\n \n if (doc.planKey) {\n result.plan_key = doc.planKey.key;\n } else {\n result.plan_key = '(none)';\n }\n version = deploy.deploymentVersion;\n if (version) {\n result.version_id = version.id;\n result.version_name = version.name;\n result.version_date = timestampToDate(version.creationDate);\n result.version_creator_key = version.creatorUserName;\n result.version_creator_name = version.creatorDisplayName;\n result.version_branch = version.planBranchName;\n } else {\n result.version_name = '(none)';\n result.version_id = -1;\n }\n results.push(result)\n });\n}\n\nfunction getResult(doc, environment, offset) {\n getDocument(\"/rest/api/latest/deploy/environment/\" + environment.id + \"/results?max-results=25&start-index=\" + offset, \n function(res) {\n processDeployments(doc, environment, res.results);\n if (res.results.size == 25) {\n getResult(doc, environment, offset + 25);\n }\n }\n );\n}\n\n_.map(doc.environments, function(environment) {\n getResult(doc, environment, 0);\n});\n\nreturn results;\n"
},
"columns_options": {
"default_names": false
},
"extra_options": {
"regular_import_frequency": 0,
"regular_import_at": "",
"time_zone": "Helsinki"
}
},
"source_cube_name": "Builds",
"columns": [
{
"name": "deploy_project_name",
"data_type": "string",
"dimension": "Deploy Project",
"dimension_level": "Deploy Project"
},
{
"name": "project_id",
"data_type": "integer",
"dimension": "Deploy Project",
"id_column": true,
"dimension_level": "Deploy Project"
},
{
"name": "environment_project_name",
"data_type": "string",
"dimension": "Environment",
"dimension_level": "Deploy Project",
"name_column": true
},
{
"name": "environment_id",
"data_type": "integer",
"dimension": "Environment",
"id_column": true,
"dimension_level": "Environment"
},
{
"name": "environment_name",
"data_type": "string",
"dimension": "Environment",
"dimension_level": "Environment",
"name_column": true
},
{
"name": "version_project_name",
"data_type": "string",
"dimension": "Version",
"dimension_level": "Deploy Project",
"name_column": true
},
{
"name": "deploy_id",
"data_type": "integer",
"dimension": "Deployment",
"key_column": true
},
{
"name": "deploy_start_time",
"data_type": "datetime",
"dimension": "Deployment",
"property": "Start time",
"date_count_measure": "Deployments started",
"date_count_dimension": "Time"
},
{
"name": "deploy_end_time",
"data_type": "datetime",
"dimension": "Deployment",
"property": "End time",
"date_count_measure": "Deployments ended",
"date_count_dimension": "Time"
},
{
"name": "deployment_state",
"data_type": "string",
"dimension": "Deployment Status"
},
{
"name": "lifecycle_state",
"data_type": "string",
"dimension": "Lifecycle State"
},
{
"name": "deploy_time",
"data_type": "datetime",
"dimension": "Time"
},
{
"name": "plan_key",
"data_type": "string",
"dimension": "Project",
"dimension_level": "Plan",
"key_column": true
},
{
"name": "version_id",
"data_type": "integer",
"dimension": "Version",
"id_column": true,
"dimension_level": "Version"
},
{
"name": "version_name",
"data_type": "string",
"dimension": "Version",
"dimension_level": "Version",
"name_column": true
},
{
"name": "version_date",
"data_type": "datetime",
"dimension": "Version",
"dimension_level": "Version",
"property": "Creation date"
},
{
"name": "version_creator_key",
"data_type": "string",
"dimension": "Version Creator",
"dimension_level": "Creator",
"key_column": true
},
{
"name": "version_creator_name",
"data_type": "string",
"dimension": "Version Creator",
"dimension_level": "Creator",
"name_column": true
},
{
"name": "version_branch",
"data_type": "string",
"dimension": "Branch"
}
]
}
1 Like
Thanks Janis!! I’ll try that in a couple of days!!
Hello,
I am trying to import build information from Bamboo, but I have over 7700+ records to import and eventually it times out. I am unclear on the pagination or offset snytax to use to limit the number of records sourced or how to speed up the sourcing.
What syntax do I use to for pagination to limit the amount of records sourced?
Hi, all!
I was trying it and this is the error that I get:
“There was error when performing your request.
Please retry or go to other page.”
It’s kinda impossible to debug.
Is there a way for EazyBI to report some details about errors it encounters?
Hi, Timur!
This seems to be some error inside eazyBI code. Please send eazyBI log files ( https://docs.eazybi.com/display/EAZYBIJIRA/Troubleshooting#Troubleshooting-eazyBIlogfiles ) to support@eazybi.com for investigation.
In case of problem with data definition or custom JavaScript code, there should be a different error displayed.
Best regards,
Jānis
1 Like
Thank you, Janis!
I was able to debug it and got it working!
Janis, hi!
Can you please explain how you get results array populated in your JS script Bamboo example?
getDocument function with a callback should be async, so you launch a bunch of async requests and then you return results array.
At that point it should be empty.
How do you wait for the results array to be fully populated?
Hi Timur,
You are right - all getDocument function calls with a callback are asynchronous. We have defined a thread pool for asynchronous operations where all callbacks are collected and will be executed in separate threads.
As the returned array is defined at the beginning of the code, we receive object reference when JS code is executed.
Then we will wait till all asynchronous operations in the thread pool are executed and made needed changes to the array.
As we received object reference, not value, we will see all changes made after JS code synchronous execution was completed. And the result is returned only when all operations in thread poll are completed.
Regards,
Janis
1 Like
Janis, hi!
Is there any way to write something to the EazyBI logs from our custom JavaScript?
Sometimes our imports with multiple getDocuments invocations start to hang up and those imports cannot even be canceled (cancellations hang up too), we have to restart the hole plugin.
Seems like that is happening because some async operations are not getting completed in the thread pool.
If we could write something to logs we might be able to understand which ones.
Thank you!
Hi Timur,
You can use console.log method to log information to eazyBI log files.
If the custom JavaScript code will be executed from UI, for example, import option screen, then it will be written to the eazybi-web.log file. If from background job, then in eazybi-queues.log file.
Best regards,
Janis
1 Like
Thank you, Janis!
Another important question is about getDocument options.
I saw this usage in one of the threads here:
getDocument(url, {ignoreErrors: [404]}, function(data) {})
What are the possible options to pass except ignoreErrors in the second options object?
Can we set a timeout for a response?
Thank you!
Hi Tumur,
This is the only option you can pass to the getDocument call. There you can provide HTTP error codes you wish to be ignored.
Currently, the default timeout is 60 seconds and it can’t be changed.
Regards,
Janis
Thank you!
Is there a way to ignore all the http error codes?
Yes, you can achieve it by adding {ignoreErrors: true}.
1 Like
Janis, hi!
Is there a way to wait for a bunch of getDocument async calls to finish executing explicitly in our code?
Does getDocument with callback return any kind of Promise object we can wait on in our custom JS?
Thank you!
Hi Timur,
There is no way to wait for async getDocument calls. All callbacks are processed and waited in the background.
Regards,
Janis
Hey there!
I know it’s been a while since the last reply to this thread but I would really appreciate if you could help me with the question bellow.
It seems that the Build Import Definition is only grabbing information from the Build Projects on the first page. Second Page and beyond are not being shown. )=