Programmatic Access

File download from the 4DN data portal now requiresauthentication, even if the file is public. Accounts can becreated by anyone, including those not part of the 4DNNetwork; for more information on account creation, see the[Account Creation](/help/user-guide/account-creation) page.Files can be downloaded from the web portal after loggingin, as usual; for file download from the command line, theprocess is described below.

First you need to create a new access key, if you don't havean access key already. Note that access keys created forJupyterhub can't be used for the rest of the portal.

To create a new access key, first log in to the data portal,then click on your account in the upper right and click onProfile from the dropdown menu. There will be a button nearthe bottom of the page to add an access key. Save the keyand secret; typically this is done by creating a file inyour home directory called keypairs.json with thefollowing contents/format (replacing the x’s with theappropriate key and secret, of course):

{
  "default": {
      "key": "XXXXXXXX",
      "secret": "xxxxxxxxxxxxxxxx",
      "server": "https://data.4dnucleome.org"
  }
}

Once the access key is created and stored, the file can bedownloaded via curl with the following command:

curl -O -L --user <key>:<secret> <download-url>

For more information on file downloads see this page

The data portal has an implementation of the Data Repository Service v1.0 API.

This standard has been developed and adopted as part of the Global Alliance for Genomics and Health (GA4GH) and implementation has been encouraged by the Common Fund Data Ecosystem.

In brief from the specification, the Data Repository Service (DRS) API provides a generic interface to data repositories so data consumers, including workflow systems, can access data in a single, standard way regardless of where it’s stored and how it’s managed.

A DRS request uses 4DN file accessions as unique identifiers and can be made with the following uri formats:

drs://data.4dnucleome.org/files-processed/4DNFIIA7E3HL/@@drs

drs://data.4dnucleome.org/ga4gh/drs/v1/objects/4DNFIIA7E3HL

and will return a json that follows the API specification:

{
    "id": "/files-processed/4DNFIIA7E3HL/",
    "created_time": "2022-02-25T20:40:38.976850+00:00",
    "drs_id": "4DNFIIA7E3HL",
    "self_uri": "drs://data.4dnucleome.org/files-processed/4DNFIIA7E3HL/@@drs",
    "access_methods": [
      {
          "access_url": {
             "url": "https://data.4dnucleome.org/4DNFIIA7E3HL/@@download"
          },
          "type": "https"
      },
      {
          "access_url": {
             "url": "http://data.4dnucleome.org/4DNFIIA7E3HL/@@download"
          },
          "type": "http"
      }
   ],
   "description": "ATAC-seq signal fold change",
   "size": 2345462672,
   "aliases": [
       "4692b7c1-addf-47aa-b4cb-e30dc65a38f6"
   ],
   "checksums": [
      {
         "checksum": "73a2635fe8382a28339370a22b84de48",
         "type": "md5"
      }
   ],
   "version": "73a2635fe8382a28339370a22b84de48",
   "updated_time": "2022-05-03T21:45:53.756034+00:00"
}

A request that specifies the access method will return a uri that can be used for data download e.g.

https://data.4dnucleome.org/ga4gh/drs/v1/objects/4DNFIIA7E3HL/access/https

Returns:

{
    "url": "https://data.4dnucleome.org/4DNFIIA7E3HL/@@download"
}

Access to the data via the access_url respects the authentication and authorization for that data and therefore, attempts to download data to which you do not have access will fail.