Skip to content

Commit

Permalink
Use a paginator when iterating over glue tables
Browse files Browse the repository at this point in the history
Without this you only get the first 100 tables in the glue database
  • Loading branch information
robotblake authored and jezdez committed Feb 28, 2018
1 parent 570187f commit dceb58c
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion redash/query_runner/athena.py
Original file line number Diff line number Diff line change
Expand Up @@ -125,9 +125,11 @@ def __get_schema_from_glue(self):
region_name=self.configuration['region']
)
schema = {}
paginator = client.get_paginator('get_tables')

for database in client.get_databases()['DatabaseList']:
for table in client.get_tables(DatabaseName=database['Name'])['TableList']:
iterator = paginator.paginate(DatabaseName=database['Name'])
for table in iterator.search('TableList[]'):
table_name = '%s.%s' % (database['Name'], table['Name'])
if table_name not in schema:
column = [columns['Name'] for columns in table['StorageDescriptor']['Columns']]
Expand Down

0 comments on commit dceb58c

Please sign in to comment.