utilities

Download and management utilities for syncing time and auxiliary files

  • Can list a directory on a ftp host

  • Can download a file from a ftp or http host

  • Can download a file from CDDIS via https when NASA Earthdata credentials are supplied

  • Checks MD5 or sha1 hashes between local and remote files

Source code

General Methods

pyTMD.utilities.get_data_path(relpath: list | str | pathlib.Path)[source]

Get the absolute path within a package from a relative path

Parameters
relpath: list, str or pathlib.Path

relative path

pyTMD.utilities.import_dependency(name: str, extra: str = '', raise_exception: bool = False)[source]

Import an optional dependency

Adapted from pandas.compat._optional::import_optional_dependency

Parameters
name: str

Module name

extra: str, default “”

Additional text to include in the ImportError message

raise_exception: bool, default False

Raise an ImportError if the module is not found

Returns
module: obj

Imported module

class pyTMD.utilities.reify(wrapped)[source]

Class decorator that puts the result of the method it decorates into the instance

pyTMD.utilities.file_opener(filename: str | pathlib.Path)[source]

Platform independent file opener

Parameters
filename: str or pathlib.Path

path to file

pyTMD.utilities.get_hash(local: str | io.IOBase | pathlib.Path, algorithm: str = 'MD5')[source]

Get the hash value from a local file or BytesIO object

Parameters
local: obj, str or pathlib.Path

BytesIO object or path to file

algorithm: str, default ‘MD5’

hashing algorithm for checksum validation

  • 'MD5': Message Digest

  • 'sha1': Secure Hash Algorithm

pyTMD.utilities.get_git_revision_hash(refname: str = 'HEAD', short: bool = False)[source]

Get the git hash value for a particular reference

Parameters
refname: str, default HEAD

Symbolic reference name

short: bool, default False

Return the shorted hash value

pyTMD.utilities.get_git_status()[source]

Get the status of a git repository as a boolean value

pyTMD.utilities.url_split(s: str)[source]

Recursively split a url path into a list

Parameters
s: str

url string

pyTMD.utilities.roman_to_int(roman: str)[source]

Converts a string from Roman numerals into an integer (Arabic)

Parameters
roman: str

Roman numeral string

pyTMD.utilities.get_unix_time(time_string: str, format: str = '%Y-%m-%d %H:%M:%S')[source]

Get the Unix timestamp value for a formatted date string

Parameters
time_string: str

formatted time string to parse

format: str, default ‘%Y-%m-%d %H:%M:%S’

format for input time string

pyTMD.utilities.isoformat(time_string: str)[source]

Reformat a date string to ISO formatting

Parameters
time_string: str

formatted time string to parse

pyTMD.utilities.even(value: float)[source]

Rounds a number to an even number less than or equal to original

Parameters
value: float

number to be rounded

pyTMD.utilities.ceil(value: float)[source]

Rounds a number upward to its nearest integer

Parameters
value: float

number to be rounded upward

pyTMD.utilities.copy(source: str | pathlib.Path, destination: str | pathlib.Path, move: bool = False, **kwargs)[source]

Copy or move a file with all system information

Parameters
source: str

source file

destination: str

copied destination file

move: bool, default False

remove the source file

pyTMD.utilities.check_ftp_connection(HOST: str, username: str | None = None, password: str | None = None)[source]

Check internet connection with ftp host

Parameters
HOST: str

remote ftp host

username: str or NoneType

ftp username

password: str or NoneType

ftp password

pyTMD.utilities.ftp_list(HOST: str | list, username: str | None = None, password: str | None = None, timeout: int | None = None, basename: bool = False, pattern: str | None = None, sort: bool = False)[source]

List a directory on a ftp host

Parameters
HOST: str or list

remote ftp host path split as list

username: str or NoneType

ftp username

password: str or NoneType

ftp password

timeout: int or NoneType, default None

timeout in seconds for blocking operations

basename: bool, default False

return the file or directory basename instead of the full path

pattern: str or NoneType, default None

regular expression pattern for reducing list

sort: bool, default False

sort output list

Returns
output: list

items in a directory

mtimes: list

last modification times for items in the directory

pyTMD.utilities.from_ftp(HOST: str | list, username: str | None = None, password: str | None = None, timeout: int | None = None, local: str | pathlib.Path | None = None, hash: str = '', chunk: int = 8192, verbose: bool = False, fid=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, mode: oct = 509)[source]

Download a file from a ftp host

Parameters
HOST: str or list

remote ftp host path

username: str or NoneType

ftp username

password: str or NoneType

ftp password

timeout: int or NoneType, default None

timeout in seconds for blocking operations

local: str, pathlib.Path or NoneType, default None

path to local file

hash: str, default ‘’

MD5 hash of local file

chunk: int, default 8192

chunk size for transfer encoding

verbose: bool, default False

print file transfer information

fid: obj, default sys.stdout

open file object to print if verbose

mode: oct, default 0o775

permissions mode of output local file

Returns
remote_buffer: obj

BytesIO representation of file

pyTMD.utilities._create_default_ssl_context() → SSLContext[source]

Creates the default SSL context

pyTMD.utilities._create_ssl_context_no_verify() → SSLContext[source]

Creates an SSL context for unverified connections

pyTMD.utilities._set_ssl_context_options(context: SSLContext) → None[source]

Sets the default options for the SSL context

pyTMD.utilities.check_connection(HOST: str, context: ~ssl.SSLContext = <ssl.SSLContext object>)[source]

Check internet connection with http host

Parameters
HOST: str

remote http host

context: obj, default pyTMD.utilities._default_ssl_context

SSL context for urllib opener object

pyTMD.utilities.http_list(HOST: str | list, timeout: int | None = None, context: ~ssl.SSLContext = <ssl.SSLContext object>, parser=<lxml.etree.HTMLParser object>, format: str = '%Y-%m-%d %H:%M', pattern: str = '', sort: bool = False)[source]

List a directory on an Apache http Server

Parameters
HOST: str or list

remote http host path

timeout: int or NoneType, default None

timeout in seconds for blocking operations

context: obj, default pyTMD.utilities._default_ssl_context

SSL context for urllib opener object

parser: obj, default lxml.etree.HTMLParser()

HTML parser for lxml

format: str, default ‘%Y-%m-%d %H:%M’

format for input time string

pattern: str, default ‘’

regular expression pattern for reducing list

sort: bool, default False

sort output list

Returns
colnames: list

column names in a directory

collastmod: list

last modification times for items in the directory

pyTMD.utilities.from_http(HOST: str | list, timeout: int | None = None, context: ~ssl.SSLContext = <ssl.SSLContext object>, local: str | pathlib.Path | None = None, hash: str = '', chunk: int = 16384, verbose: bool = False, fid=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, mode: oct = 509)[source]

Download a file from a http host

Parameters
HOST: str or list

remote http host path split as list

timeout: int or NoneType, default None

timeout in seconds for blocking operations

context: obj, default pyTMD.utilities._default_ssl_context

SSL context for urllib opener object

local: str, pathlib.Path or NoneType, default None

path to local file

hash: str, default ‘’

MD5 hash of local file

chunk: int, default 16384

chunk size for transfer encoding

verbose: bool, default False

print file transfer information

fid: obj, default sys.stdout

open file object to print if verbose

mode: oct, default 0o775

permissions mode of output local file

Returns
remote_buffer: obj

BytesIO representation of file

pyTMD.utilities.attempt_login(urs: str, context=<ssl.SSLContext object>, password_manager: bool = True, get_ca_certs: bool = True, redirect: bool = True, authorization_header: bool = False, **kwargs)[source]

attempt to build a urllib opener for NASA Earthdata

Parameters
urs: str

Earthdata login URS 3 host

context: obj, default pyTMD.utilities._default_ssl_context

SSL context for urllib opener object

password_manager: bool, default True

Create password manager context using default realm

get_ca_certs: bool, default True

Get list of loaded “certification authority” certificates

redirect: bool, default True

Create redirect handler object

authorization_header: bool, default False

Add base64 encoded authorization header to opener

username: str, default from environmental variable

NASA Earthdata username

password: str, default from environmental variable

NASA Earthdata password

retries: int, default 5

number of retry attempts

netrc: str, default ~/.netrc

path to .netrc file for authentication

Returns
opener: obj

OpenerDirector instance

pyTMD.utilities.build_opener(username: str, password: str, context=<ssl.SSLContext object>, password_manager: bool = True, get_ca_certs: bool = True, redirect: bool = True, authorization_header: bool = False, urs: str = 'https://urs.earthdata.nasa.gov')[source]

Build urllib opener for NASA Earthdata with supplied credentials

Parameters
username: str or NoneType, default None

NASA Earthdata username

password: str or NoneType, default None

NASA Earthdata password

context: obj, default pyTMD.utilities._default_ssl_context

SSL context for urllib opener object

password_manager: bool, default True

Create password manager context using default realm

get_ca_certs: bool, default True

Get list of loaded “certification authority” certificates

redirect: bool, default True

Create redirect handler object

authorization_header: bool, default False

Add base64 encoded authorization header to opener

urs: str, default ‘https://urs.earthdata.nasa.gov’

Earthdata login URS 3 host

Returns
opener: obj

OpenerDirector instance

pyTMD.utilities.get_token(HOST: str = 'https://urs.earthdata.nasa.gov/api/users/token', username: str | None = None, password: str | None = None, build: bool = True, urs: str = 'urs.earthdata.nasa.gov')[source]

Generate a NASA Earthdata User Token

Parameters
HOST: str or list

NASA Earthdata token API host

username: str or NoneType, default None

NASA Earthdata username

password: str or NoneType, default None

NASA Earthdata password

build: bool, default True

Build opener and check WebDAV credentials

timeout: int or NoneType, default None

timeout in seconds for blocking operations

urs: str, default ‘urs.earthdata.nasa.gov’

NASA Earthdata URS 3 host

Returns
token: dict

JSON response with NASA Earthdata User Token

pyTMD.utilities.list_tokens(HOST: str = 'https://urs.earthdata.nasa.gov/api/users/tokens', username: str | None = None, password: str | None = None, build: bool = True, urs: str = 'urs.earthdata.nasa.gov')[source]

List the current associated NASA Earthdata User Tokens

Parameters
HOST: str

NASA Earthdata list token API host

username: str or NoneType, default None

NASA Earthdata username

password: str or NoneType, default None

NASA Earthdata password

build: bool, default True

Build opener and check WebDAV credentials

timeout: int or NoneType, default None

timeout in seconds for blocking operations

urs: str, default ‘urs.earthdata.nasa.gov’

NASA Earthdata URS 3 host

Returns
tokens: list

JSON response with NASA Earthdata User Tokens

pyTMD.utilities.revoke_token(token: str, HOST: str = 'https://urs.earthdata.nasa.gov/api/users/revoke_token', username: str | None = None, password: str | None = None, build: bool = True, urs: str = 'urs.earthdata.nasa.gov')[source]

Generate a NASA Earthdata User Token

Parameters
token: str

NASA Earthdata token to be revoked

HOST: str

NASA Earthdata revoke token API host

username: str or NoneType, default None

NASA Earthdata username

password: str or NoneType, default None

NASA Earthdata password

build: bool, default True

Build opener and check WebDAV credentials

timeout: int or NoneType, default None

timeout in seconds for blocking operations

urs: str, default ‘urs.earthdata.nasa.gov’

NASA Earthdata URS 3 host

pyTMD.utilities.check_credentials()[source]

Check that entered NASA Earthdata credentials are valid

pyTMD.utilities.cddis_list(HOST: str | list, username: str | None = None, password: str | None = None, build: bool = True, timeout: int | None = None, urs: str = 'urs.earthdata.nasa.gov', parser=<lxml.etree.HTMLParser object>, pattern: str = '', sort: bool = False)[source]

List a directory on GSFC CDDIS archive server

Parameters
HOST: str or list

remote https host

username: str or NoneType, default None

NASA Earthdata username

password: str or NoneType, default None

NASA Earthdata password

build: bool, default True

Build opener and check Earthdata credentials

timeout: int or NoneType, default None

timeout in seconds for blocking operations

urs: str, default ‘urs.earthdata.nasa.gov’

NASA Earthdata URS 3 host

parser: obj, default lxml.etree.HTMLParser()

HTML parser for lxml

pattern: str, default ‘’

regular expression pattern for reducing list

sort: bool, default False

sort output list

Returns
colnames: list

column names in a directory

collastmod: list

last modification times for items in the directory

pyTMD.utilities.from_cddis(HOST: str | list, username: str | None = None, password: str | None = None, build: bool = True, timeout: int | None = None, urs: str = 'urs.earthdata.nasa.gov', local: str | pathlib.Path | None = None, hash: str = '', chunk: int = 16384, verbose: bool = False, fid=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, mode: oct = 509)[source]

Download a file from GSFC CDDIS archive server

Parameters
HOST: str or list

remote https host

username: str or NoneType, default None

NASA Earthdata username

password: str or NoneType, default None

NASA Earthdata password

build: bool, default True

Build opener and check Earthdata credentials

timeout: int or NoneType, default None

timeout in seconds for blocking operations

urs: str, default ‘urs.earthdata.nasa.gov’

NASA Earthdata URS 3 host

local: str, pathlib.Path or NoneType, default None

path to local file

hash: str, default ‘’

MD5 hash of local file

chunk: int, default 16384

chunk size for transfer encoding

verbose: bool, default False

print file transfer information

fid: obj, default sys.stdout

open file object to print if verbose

mode: oct, default 0o775

permissions mode of output local file

Returns
remote_buffer: obj

BytesIO representation of file

pyTMD.utilities.iers_list(HOST: str | list, timeout: int | None = None, context: ~ssl.SSLContext = <ssl.SSLContext object>, parser=<lxml.etree.HTMLParser object>)[source]

List a directory on IERS Bulletin-A https server

Parameters
HOST: str or list

remote http host path

timeout: int or NoneType, default None

timeout in seconds for blocking operations

context: obj, default pyTMD.utilities._default_ssl_context

SSL context for urllib opener object

parser: obj, default lxml.etree.HTMLParser()

HTML parser for lxml

Returns
colnames: list

column names in a directory

collastmod: list

last modification times for items in the directory

pyTMD.utilities.from_jpl_ssd(kernel='de440s.bsp', timeout: int | None = None, context: ~ssl.SSLContext = <ssl.SSLContext object>, local: str | pathlib.Path | None = None, hash: str = '', chunk: int = 16384, verbose: bool = False, mode: oct = 509)[source]

Download planetary ephemeride kernels from the JPL Solar System Dynamics server

Parameters
kernel: str

JPL kernel file to download

timeout: int or NoneType, default None

timeout in seconds for blocking operations

context: obj, default pyTMD.utilities._default_ssl_context

SSL context for urllib opener object

hash: str, default ‘’

MD5 hash of local file

chunk: int, default 16384

chunk size for transfer encoding

verbose: bool, default False

print file transfer information

mode: oct, default 0o775

permissions mode of output local file