Skip to main content

​
More Info:

Amazon Elasticsearch (ES) clusters should not appear to be idle. Such idle clusters should be removed from your account to help lower the cost of your monthly AWS bill.

​
Risk Level

Informational

​
Address

Cost Optimisation

​
Compliance Standards

CBP

​
Triage and Remediation

  • Remediation

​
Remediation

Using Console

To remediate the Elasticsearch cluster idle misconfiguration in AWS using the AWS console, follow the below steps:
  1. Go to the AWS Elasticsearch console.
  2. Click on the name of the Elasticsearch cluster that you want to remediate.
  3. Click on the “Actions” button and select “Modify Cluster Settings”.
  4. Scroll down to the “Elasticsearch cluster settings” section and locate the “Instance Count” option.
  5. Increase the “Instance Count” to a minimum of two instances. This will ensure that there is always at least one active instance in the cluster.
  6. Click on the “Apply” button to save the changes.
Once you have followed these steps, your Elasticsearch cluster will no longer be idle and will have at least two active instances running at all times. This will help ensure that your data is always available and that your cluster is performing optimally.

​

The following are the step-by-step instructions to remediate the “Elasticsearch Clusters Should Not Be Idle” misconfiguration for AWS using AWS CLI:
  1. Open the AWS CLI on your local machine.
  2. Run the following command to list all the Elasticsearch domains in your AWS account: 
    aws es list-domain-names
  3. Choose the Elasticsearch domain that you want to remediate.
  4. Run the following command to check the status of the Elasticsearch domain: 
    aws es describe-elasticsearch-domain --domain-name <your-domain-name> --output json
    Replace <your-domain-name> with the name of your Elasticsearch domain.
  5. Check the ElasticsearchClusterConfig.InstanceCount parameter in the output of the previous command. If the value is 0, it means that the Elasticsearch cluster is idle.
  6. Run the following command to update the ElasticsearchClusterConfig.InstanceCount parameter to a non-zero value: 
    aws es update-elasticsearch-domain-config --domain-name <your-domain-name> --elasticsearch-cluster-config InstanceCount=<number-of-instances> --output json
    Replace <your-domain-name> with the name of your Elasticsearch domain and <number-of-instances> with the desired number of instances.
  7. Wait for a few minutes for the changes to take effect.
  8. Run the following command to check the status of the Elasticsearch domain again: 
    aws es describe-elasticsearch-domain --domain-name <your-domain-name> --output json
    Ensure that the ElasticsearchClusterConfig.InstanceCount parameter has been updated to the desired value.
  9. Verify that the Elasticsearch cluster is no longer idle.
By following these steps, you can remediate the “Elasticsearch Clusters Should Not Be Idle” misconfiguration for AWS using AWS CLI.
To remediate the Elasticsearch cluster idle misconfiguration in AWS using Python, you can follow these steps:
  1. Install the AWS SDK for Python (Boto3) using pip. You can use the following command to install it: 
    pip install boto3
  2. Create a Python script and import the necessary modules: 
    import boto3
import json
from datetime import datetime, timedelta
  3. Set the AWS region where your Elasticsearch cluster is located: 
    region_name = 'your_region_name'
  4. Create a Boto3 Elasticsearch client: 
    es_client = boto3.client('es', region_name=region_name)
  5. Get a list of all Elasticsearch domains in the region: 
    domains = es_client.list_domain_names()['DomainNames']
  6. For each domain, check if it is idle: 
    for domain in domains:
    domain_name = domain['DomainName']
    domain_status = es_client.describe_elasticsearch_domain(DomainName=domain_name)['DomainStatus']
    last_updated = datetime.strptime(domain_status['LastUpdated'], '%Y-%m-%dT%H:%M:%SZ')
    idle_time = datetime.utcnow() - last_updated
    if idle_time > timedelta(hours=1):
        # Take remediation action here
        print(f'{domain_name} is idle for {idle_time.total_seconds()/3600} hours')
    else:
        print(f'{domain_name} is active')
  7. Replace the # Take remediation action here comment with the code to remediate the idle Elasticsearch cluster. For example, you can start a new instance or increase the size of an existing instance. 
    es_client.update_elasticsearch_domain_config(
    DomainName=domain_name,
    ElasticsearchClusterConfig={
        'InstanceType': 'm5.large.elasticsearch',
        'InstanceCount': 2,
        'DedicatedMasterEnabled': False,
        'ZoneAwarenessEnabled': False
    }
)
    This code updates the Elasticsearch cluster configuration to use two m5.large instances instead of the current configuration. You can adjust the instance type and count based on your requirements.
  8. Run the Python script periodically to check for idle Elasticsearch clusters and remediate them. You can use a scheduling tool like cron to run the script at regular intervals. 
    # Run the script every hour
0 * * * * python /path/to/script.py
With these steps, you can remediate the Elasticsearch cluster idle misconfiguration in AWS using Python.

​
Additional Reading:

⌘I