Amazon Athena HBase connector

To use an AWS Glue table for supplemental metadata

1. When you edit the table and database in the AWS Glue console, add the following table properties:

   • hbase-metadata-flag – This property indicates to the HBase connector that the connector can use the table for supplemental metadata. You can provide any value for hbase-metadata-flag as long as the hbase-metadata-flag property is present in the list of table properties.

   • hbase-native-storage-flag – Use this flag to toggle the two value serialization modes supported by the connector. By default, when this field is not present, the connector assumes all values are stored in HBase as strings. As such it will attempt to parse data types such as INT , BIGINT , and DOUBLE from HBase as strings. If this field is set with any value on the table in AWS Glue, the connector switches to "native" storage mode and attempts to read INT , BIGINT , BIT , and DOUBLE as bytes by using the following functions:

     ByteBuffer.wrap(value).getInt() 
     ByteBuffer.wrap(value).getLong() 
     ByteBuffer.wrap(value).get() 
     ByteBuffer.wrap(value).getDouble()

     Make sure that you use the data types appropriate for AWS Glue as listed in this document.

     Modeling column families

     The Athena HBase connector supports two ways to model HBase column families: fully qualified (flattened) naming like family:column , or using STRUCT objects.

     In the STRUCT model, the name of the STRUCT field should match the column family, and children of the STRUCT should match the names of the columns of the family. However, because predicate push down and columnar reads are not yet fully supported for complex types like STRUCT , using STRUCT is currently not advised.

     The following image shows a table configured in AWS Glue that uses a combination of the two approaches.

     Data type support

     The connector retrieves all HBase values as the basic byte type. Then, based on how you defined your tables in AWS Glue Data Catalog, it maps the values into one of the Apache Arrow data types in the following table.

     AWS Glue data type Apache Arrow data type bigint BIGINT double FLOAT8 float FLOAT4 boolean binary VARBINARY string VARCHAR

     Note

     If you do not use AWS Glue to supplement your metadata, the connector's schema inferencing uses only the data types BIGINT , FLOAT8 , and VARCHAR .

     Required Permissions

     For full details on the IAM policies that this connector requires, review the Policies section of the athena-hbase.yaml file. The following list summarizes the required permissions.

     Amazon S3 write access – The connector requires write access to a location in Amazon S3 in order to spill results from large queries.

     Athena GetQueryExecution – The connector uses this permission to fast-fail when the upstream Athena query has terminated.

     AWS Glue Data Catalog – The HBase connector requires read only access to the AWS Glue Data Catalog to obtain schema information.

     CloudWatch Logs – The connector requires access to CloudWatch Logs for storing logs.

     AWS Secrets Manager read access – If you choose to store HBase endpoint details in Secrets Manager, you must grant the connector access to those secrets.

     VPC access – The connector requires the ability to attach and detach interfaces to your VPC so that it can connect to it and communicate with your HBase instances.

     Performance

     The Athena HBase connector attempts to parallelize queries against your HBase instance by reading each region server in parallel. The Athena HBase connector performs predicate pushdown to decrease the data scanned by the query.

     The Lambda function also performs projection pushdown to decrease the data scanned by the query. However, selecting a subset of columns sometimes results in a longer query execution runtime. LIMIT clauses reduce the amount of data scanned, but if you do not provide a predicate, you should expect SELECT queries with a LIMIT clause to scan at least 16 MB of data.

     HBase is prone to query failures and variable query execution times. You might have to retry your queries multiple times for them to succeed. The HBase connector is resilient to throttling due to concurrency.

     Passthrough queries

     The HBase connector supports passthrough queries and is NoSQL based. For information about querying Apache HBase using filtering, see Filter language in the Apache documentation.

     To use passthrough queries with HBase, use the following syntax:

     SELECT * FROM TABLE(
             system.query(
                 database => 'database_name',
                 collection => 'collection_name',
                 filter => '{query_syntax}'
             
     The following example HBase passthrough query filters for employees aged 24 or 30
                 within the employee collection of the default database.
             
     SELECT * FROM TABLE(
             system.query(
                 DATABASE => 'default',
                 COLLECTION => 'employee',
                 FILTER => 'SingleColumnValueFilter(''personaldata'', ''age'', =, ''binary:30'')' ||
                            ' OR SingleColumnValueFilter(''personaldata'', ''age'', =, ''binary:24'')'