Source ETL API#

Base#

class gene.etl.base.Base(database, host, data_dir, src_data_dir, seqrepo_dir=PosixPath('/usr/local/share/seqrepo/latest'), *args, **kwargs)[source]#

The ETL base class.

__init__(database, host, data_dir, src_data_dir, seqrepo_dir=PosixPath('/usr/local/share/seqrepo/latest'), *args, **kwargs)[source]#

Instantiate Base class.

Parameters:

database (AbstractDatabase) – database instance
host (str) – Hostname of FTP site
data_dir (str) – Data directory of FTP site to look at
src_data_dir (Path) – Data directory for source
seqrepo_dir (Path) – Path to seqrepo directory

get_seqrepo(seqrepo_dir)[source]#

Return SeqRepo instance if seqrepo_dir exists.

Parameters:: seqrepo_dir (Path) – Path to seqrepo directory
Return type:: SeqRepo
Returns:: SeqRepo instance

abstract perform_etl()[source]#

Extract, Transform, and Load data into database.

Return type:: List[str]
Returns:: Concept IDs of concepts successfully loaded

NCBI#

class gene.etl.ncbi.NCBI(database, host='ftp.ncbi.nlm.nih.gov', data_dir='gene/DATA/', src_data_dir=PosixPath('/home/docs/checkouts/readthedocs.org/user_builds/gene-normalizer/envs/0.1.37/lib/python3.10/site-packages/gene/data/ncbi'))[source]#

Bases: Base

ETL class for NCBI source

__init__(database, host='ftp.ncbi.nlm.nih.gov', data_dir='gene/DATA/', src_data_dir=PosixPath('/home/docs/checkouts/readthedocs.org/user_builds/gene-normalizer/envs/0.1.37/lib/python3.10/site-packages/gene/data/ncbi'))[source]#

Construct the NCBI ETL instance.

Parameters:

database (AbstractDatabase) – gene database for adding new data
host (str) – FTP host name
data_dir (str) – FTP data directory to use
src_data_dir (Path) – Data directory for NCBI

perform_etl()[source]#

Perform ETL methods.

Return type:: List[str]
Returns:: Concept IDs of concepts successfully loaded

HGNC#

class gene.etl.hgnc.HGNC(database, host='ftp.ebi.ac.uk', data_dir='pub/databases/genenames/hgnc/json/', src_data_dir=PosixPath('/home/docs/checkouts/readthedocs.org/user_builds/gene-normalizer/envs/0.1.37/lib/python3.10/site-packages/gene/data/hgnc'), fn='hgnc_complete_set.json')[source]#

Bases: Base

ETL the HGNC source into the normalized database.

__init__(database, host='ftp.ebi.ac.uk', data_dir='pub/databases/genenames/hgnc/json/', src_data_dir=PosixPath('/home/docs/checkouts/readthedocs.org/user_builds/gene-normalizer/envs/0.1.37/lib/python3.10/site-packages/gene/data/hgnc'), fn='hgnc_complete_set.json')[source]#

Initialize HGNC ETL class.

Parameters:

database (AbstractDatabase) – DynamoDB database
host (str) – FTP host name
data_dir (str) – FTP data directory to use
src_data_dir (Path) – Data directory for HGNC
fn (str) – Data file to download

perform_etl(*args, **kwargs)[source]#

Extract, Transform, and Load data into DynamoDB database.

Return type:: List[str]
Returns:: Concept IDs of concepts successfully loaded

Ensembl#

class gene.etl.ensembl.Ensembl(database, host='ftp.ensembl.org', data_dir='pub/current_gff3/homo_sapiens/', src_data_dir=PosixPath('/home/docs/checkouts/readthedocs.org/user_builds/gene-normalizer/envs/0.1.37/lib/python3.10/site-packages/gene/data/ensembl'))[source]#

Bases: Base

ETL the Ensembl source into the normalized database.

__init__(database, host='ftp.ensembl.org', data_dir='pub/current_gff3/homo_sapiens/', src_data_dir=PosixPath('/home/docs/checkouts/readthedocs.org/user_builds/gene-normalizer/envs/0.1.37/lib/python3.10/site-packages/gene/data/ensembl'))[source]#

Initialize Ensembl ETL class.

Parameters:

database (AbstractDatabase) – DynamoDB database
host (str) – FTP host name
data_dir (str) – FTP data directory to use
src_data_dir (Path) – Data directory for Ensembl

perform_etl(*args, **kwargs)[source]#

Extract, Transform, and Load data into DynamoDB database.

Return type:: List[str]
Returns:: Concept IDs of concepts successfully loaded

Normalized Records#

class gene.etl.merge.Merge(database)[source]#

Bases: object

Handles record merging.

__init__(database)[source]#

Initialize Merge instance.

Parameters:: database (AbstractDatabase) – db instance to use for record retrieval and creation.

create_merged_concepts(record_ids)[source]#

Create concept groups, generate merged concept records, and update database.

Parameters:: record_ids (Set[str]) – concept identifiers from which groups should be generated. Should not include any records from excluded sources.
Return type:: None

Chromosome Location#

class gene.etl.vrs_locations.chromosome_location.ChromosomeLocation[source]#

The class for GA4GH Chromosome Location.

get_location(location, gene)[source]#

Transform a gene’s location into a Chromosome Location.

Parameters:

location (Dict) – A gene’s location.
gene (Dict) – A transformed gene record.

Return type:

Optional[Dict]

Returns:

If location is a valid VRS ChromosomeLocation, return a dictionary containing the ChromosomeLocation. Else, return None.

set_interval_range(loc, arm_ix, location)[source]#

Set the location interval range.

Parameters:

loc (str) – A gene location
arm_ix (int) – The index of the q or p arm for a given location
location (Dict) – A gene’s location

Return type:

None

Sequence Location#

class gene.etl.vrs_locations.sequence_location.SequenceLocation[source]#

The class for GA4GH Sequence Location.

add_location(seqid, gene, params, sr)[source]#

Get a gene’s Sequence Location.

Parameters:

seqid (str) – The sequence ID.
gene (Feature) – A gene from the source file.
params (Dict) – The transformed gene record.
sr (SeqRepo) – Access to the SeqRepo

Return type:

Dict

Returns:

A dictionary of a GA4GH VRS SequenceLocation.

get_aliases(sr, seqid)[source]#

Get aliases for a sequence id

Parameters:

sr (SeqRepo) – seqrepo instance
seqid (str) – Sequence ID accession

Return type:

List[str]

Returns:

List of aliases for seqid